Oct 30 16:17:51.592919 ip-10-0-131-136 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Oct 30 16:17:51.592932 ip-10-0-131-136 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Oct 30 16:17:51.592941 ip-10-0-131-136 systemd[1]: kubelet.service: Failed with result 'resources'. Oct 30 16:17:51.593273 ip-10-0-131-136 systemd[1]: Failed to start Kubernetes Kubelet. Oct 30 16:18:01.274572 ip-10-0-131-136 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Oct 30 16:18:01.274584 ip-10-0-131-136 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot e42287c773b04523bb5cd0439396016a -- Oct 30 16:19:47.419539 ip-10-0-131-136 systemd[1]: Starting Kubernetes Kubelet... Oct 30 16:19:47.766645 ip-10-0-131-136 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 16:19:47.766645 ip-10-0-131-136 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 16:19:47.766645 ip-10-0-131-136 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 16:19:47.766645 ip-10-0-131-136 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 16:19:47.771387 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.771303 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:19:47.771387 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.771386 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:19:47.771387 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.771390 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:19:47.771484 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.771394 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:19:47.771484 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.771396 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:19:47.771484 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.771399 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:19:47.771484 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.771403 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:19:47.771484 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.771406 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:19:47.771484 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.771408 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:19:47.771484 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.771411 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:19:47.771484 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.771414 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:19:47.771484 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.771417 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:19:47.771484 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.771419 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:19:47.771484 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.771426 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:19:47.771484 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.771429 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:19:47.771484 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.771432 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:19:47.771484 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.771434 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:19:47.771484 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.771437 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:19:47.771484 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.771440 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:19:47.771484 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.771442 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:19:47.771484 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.771445 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:19:47.771484 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.771448 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:19:47.771484 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.771450 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:19:47.771983 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.771452 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:19:47.771983 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.771455 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:19:47.771983 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.771457 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:19:47.771983 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.771460 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:19:47.771983 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.771463 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:19:47.771983 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.771465 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:19:47.771983 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.771468 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:19:47.771983 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.771470 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:19:47.771983 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.771473 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:19:47.771983 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.771475 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:19:47.771983 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.771478 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:19:47.771983 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.771480 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:19:47.771983 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.771482 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:19:47.771983 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.771486 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:19:47.771983 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.771489 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:19:47.771983 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.771492 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:19:47.771983 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.771494 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:19:47.771983 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.771497 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:19:47.771983 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.771500 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:19:47.772446 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.771502 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:19:47.772446 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.771505 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:19:47.772446 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.771508 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:19:47.772446 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.771511 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:19:47.772446 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.771514 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:19:47.772446 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.771517 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:19:47.772446 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.771519 2580 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:19:47.772446 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.771522 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:19:47.772446 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.771524 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:19:47.772446 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.771527 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:19:47.772446 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.771529 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:19:47.772446 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.771532 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:19:47.772446 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.771534 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:19:47.772446 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.771537 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:19:47.772446 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.771539 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:19:47.772446 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.771542 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:19:47.772446 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.771544 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:19:47.772446 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.771547 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:19:47.772446 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.771549 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:19:47.772446 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.771552 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:19:47.772910 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.771556 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:19:47.772910 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.771560 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:19:47.772910 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.771563 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:19:47.772910 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.771566 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:19:47.772910 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.771570 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:19:47.772910 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.771575 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:19:47.772910 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.771579 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:19:47.772910 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.771582 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:19:47.772910 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.771585 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:19:47.772910 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.771587 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:19:47.772910 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.771590 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:19:47.772910 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.771593 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:19:47.772910 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.771596 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:19:47.772910 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.771598 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:19:47.772910 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.771601 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:19:47.772910 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.771604 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:19:47.772910 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.771606 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:19:47.772910 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.771610 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:19:47.772910 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.771613 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:19:47.773476 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.771615 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:19:47.773476 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.771618 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:19:47.773476 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.771620 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:19:47.773476 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.771623 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:19:47.773476 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.771625 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:19:47.773476 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.771628 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:19:47.773476 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.771630 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:19:47.773476 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.771632 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:19:47.773476 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.771635 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:19:47.773476 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.771637 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:19:47.773476 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.771640 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:19:47.773476 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.771643 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:19:47.773476 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.771646 2580 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:19:47.773476 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.771649 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:19:47.773476 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.771652 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:19:47.773476 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.771655 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:19:47.773476 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.771657 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:19:47.773476 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.771660 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:19:47.773476 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.771663 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:19:47.773476 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.771666 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:19:47.774127 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.771668 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:19:47.774127 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.771671 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:19:47.774127 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.771673 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:19:47.774127 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.771677 2580 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:19:47.774127 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.771679 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:19:47.774905 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.774887 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:19:47.774934 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.774905 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:19:47.774934 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.774910 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:19:47.774934 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.774913 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:19:47.774934 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.774916 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:19:47.774934 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.774918 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:19:47.774934 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.774921 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:19:47.774934 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.774924 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:19:47.774934 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.774926 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:19:47.774934 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.774929 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:19:47.774934 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.774932 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:19:47.774934 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.774935 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:19:47.774934 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.774938 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:19:47.775292 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.774941 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:19:47.775292 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.774944 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:19:47.775292 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.774946 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:19:47.775292 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.774949 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:19:47.775292 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.774951 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:19:47.775292 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.774954 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:19:47.775292 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.774956 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:19:47.775292 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.774959 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:19:47.775292 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.774961 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:19:47.775292 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.774964 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:19:47.775292 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.774966 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:19:47.775292 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.774969 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:19:47.775292 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.774971 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:19:47.775292 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.774974 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:19:47.775292 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.774977 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:19:47.775292 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.774980 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:19:47.775292 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.774983 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:19:47.775292 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.774985 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:19:47.775292 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.774988 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:19:47.775745 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.774990 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:19:47.775745 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.774995 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:19:47.775745 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.774998 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:19:47.775745 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775002 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:19:47.775745 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775004 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:19:47.775745 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775007 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:19:47.775745 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775010 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:19:47.775745 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775012 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:19:47.775745 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775015 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:19:47.775745 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775017 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:19:47.775745 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775020 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:19:47.775745 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775022 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:19:47.775745 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775025 2580 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:19:47.775745 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775027 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:19:47.775745 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775031 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:19:47.775745 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775035 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:19:47.775745 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775037 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:19:47.775745 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775040 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:19:47.775745 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775043 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:19:47.776199 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775046 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:19:47.776199 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775053 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:19:47.776199 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775056 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:19:47.776199 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775059 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:19:47.776199 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775061 2580 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:19:47.776199 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775064 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:19:47.776199 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775067 2580 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:19:47.776199 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775070 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:19:47.776199 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775072 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:19:47.776199 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775075 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:19:47.776199 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775078 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:19:47.776199 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775080 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:19:47.776199 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775083 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:19:47.776199 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775086 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:19:47.776199 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775088 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:19:47.776199 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775091 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:19:47.776199 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775093 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:19:47.776199 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775096 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:19:47.776199 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775098 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:19:47.776199 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775101 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:19:47.776684 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775104 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:19:47.776684 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775106 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:19:47.776684 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775108 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:19:47.776684 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775111 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:19:47.776684 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775114 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:19:47.776684 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775116 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:19:47.776684 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775118 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:19:47.776684 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775122 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:19:47.776684 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775125 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:19:47.776684 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775127 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:19:47.776684 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775130 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:19:47.776684 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775132 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:19:47.776684 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775135 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:19:47.776684 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775138 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:19:47.776684 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775141 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:19:47.776684 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775144 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:19:47.776684 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775146 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:19:47.776684 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775149 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:19:47.776684 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775151 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:19:47.776684 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775154 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:19:47.777157 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775156 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:19:47.777157 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775159 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:19:47.777157 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775161 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:19:47.777157 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775164 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:19:47.777157 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775166 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:19:47.777157 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775171 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:19:47.777157 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775174 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:19:47.777157 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775177 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:19:47.777157 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775179 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:19:47.777157 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775182 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:19:47.777157 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775184 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:19:47.777157 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775187 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:19:47.777157 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775190 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:19:47.777157 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775192 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:19:47.777157 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775195 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:19:47.777157 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775740 2580 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:19:47.777157 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775747 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:19:47.777157 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775750 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:19:47.777157 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775753 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:19:47.777157 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775757 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:19:47.777646 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775759 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:19:47.777646 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775762 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:19:47.777646 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775765 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:19:47.777646 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775768 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:19:47.777646 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775771 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:19:47.777646 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775774 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:19:47.777646 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775777 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:19:47.777646 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775779 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:19:47.777646 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775782 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:19:47.777646 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775785 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:19:47.777646 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775787 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:19:47.777646 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775790 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:19:47.777646 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775792 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:19:47.777646 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775795 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:19:47.777646 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775799 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:19:47.777646 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775803 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:19:47.777646 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775806 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:19:47.777646 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775809 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:19:47.777646 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775812 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:19:47.778090 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775815 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:19:47.778090 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775817 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:19:47.778090 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775820 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:19:47.778090 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775823 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:19:47.778090 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775825 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:19:47.778090 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775828 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:19:47.778090 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775831 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:19:47.778090 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775833 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:19:47.778090 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775836 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:19:47.778090 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775839 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:19:47.778090 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775841 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:19:47.778090 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775844 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:19:47.778090 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775847 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:19:47.778090 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775849 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:19:47.778090 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775852 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:19:47.778090 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775855 2580 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:19:47.778090 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775857 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:19:47.778090 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775860 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:19:47.778090 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775863 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:19:47.778090 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775866 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:19:47.778563 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775869 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:19:47.778563 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775871 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:19:47.778563 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775874 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:19:47.778563 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775877 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:19:47.778563 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775879 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:19:47.778563 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775881 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:19:47.778563 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775884 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:19:47.778563 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775886 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:19:47.778563 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775889 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:19:47.778563 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775892 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:19:47.778563 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775895 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:19:47.778563 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775897 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:19:47.778563 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775901 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:19:47.778563 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775904 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:19:47.778563 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775908 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:19:47.778563 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775911 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:19:47.778563 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775913 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:19:47.778563 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775916 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:19:47.778563 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775919 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:19:47.778563 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775921 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:19:47.779027 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775924 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:19:47.779027 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775926 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:19:47.779027 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775929 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:19:47.779027 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775931 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:19:47.779027 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775934 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:19:47.779027 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775937 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:19:47.779027 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775939 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:19:47.779027 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775942 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:19:47.779027 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775944 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:19:47.779027 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775947 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:19:47.779027 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775949 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:19:47.779027 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775952 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:19:47.779027 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775955 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:19:47.779027 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775957 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:19:47.779027 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775960 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:19:47.779027 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775962 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:19:47.779027 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775965 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:19:47.779027 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775967 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:19:47.779027 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775970 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:19:47.779482 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775972 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:19:47.779482 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775975 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:19:47.779482 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775981 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:19:47.779482 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775984 2580 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:19:47.779482 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775986 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:19:47.779482 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775989 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:19:47.779482 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775992 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:19:47.779482 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775994 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:19:47.779482 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775997 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:19:47.779482 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.775999 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:19:47.779482 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.776002 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:19:47.779482 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.776004 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:19:47.779482 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.776007 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:19:47.779482 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.776009 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:19:47.779482 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.776012 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:19:47.779482 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.776014 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:19:47.779482 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.776017 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:19:47.779482 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.776019 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:19:47.779482 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.776023 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:19:47.779946 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.776025 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:19:47.779946 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.776028 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:19:47.779946 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.776031 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:19:47.779946 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.776033 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:19:47.779946 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.776862 2580 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 16:19:47.779946 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.776872 2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 16:19:47.779946 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.776903 2580 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 16:19:47.779946 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.776908 2580 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 16:19:47.779946 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.776914 2580 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 16:19:47.779946 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.776918 2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 16:19:47.779946 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.776926 2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 16:19:47.779946 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.776931 2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 16:19:47.779946 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.776934 2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 16:19:47.779946 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.776937 2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 16:19:47.779946 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.776940 2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 16:19:47.779946 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.776946 2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 16:19:47.779946 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.776949 2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 16:19:47.779946 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.776952 2580 flags.go:64] FLAG: --cgroup-root="" Oct 30 16:19:47.779946 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.776955 2580 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 16:19:47.779946 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.776958 2580 flags.go:64] FLAG: --client-ca-file="" Oct 30 16:19:47.779946 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.776961 2580 flags.go:64] FLAG: --cloud-provider="external" Oct 30 16:19:47.779946 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.776964 2580 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 16:19:47.779946 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.776968 2580 flags.go:64] FLAG: --cluster-domain="" Oct 30 16:19:47.780526 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.776971 2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 16:19:47.780526 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.776975 2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 16:19:47.780526 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.776978 2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 16:19:47.780526 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.776981 2580 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 16:19:47.780526 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.776986 2580 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 16:19:47.780526 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.776989 2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 16:19:47.780526 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.776992 2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 16:19:47.780526 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.776995 2580 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 16:19:47.780526 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.776999 2580 flags.go:64] FLAG: --contention-profiling="false" Oct 30 16:19:47.780526 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777001 2580 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 16:19:47.780526 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777005 2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 16:19:47.780526 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777008 2580 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 16:19:47.780526 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777011 2580 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 16:19:47.780526 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777015 2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 16:19:47.780526 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777020 2580 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 16:19:47.780526 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777023 2580 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 16:19:47.780526 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777027 2580 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 16:19:47.780526 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777030 2580 flags.go:64] FLAG: --enable-server="true" Oct 30 16:19:47.780526 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777033 2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 16:19:47.780526 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777038 2580 flags.go:64] FLAG: --event-burst="100" Oct 30 16:19:47.780526 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777041 2580 flags.go:64] FLAG: --event-qps="50" Oct 30 16:19:47.780526 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777044 2580 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 16:19:47.780526 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777048 2580 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 16:19:47.780526 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777051 2580 flags.go:64] FLAG: --eviction-hard="" Oct 30 16:19:47.781121 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777056 2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 16:19:47.781121 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777059 2580 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 16:19:47.781121 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777063 2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 16:19:47.781121 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777066 2580 flags.go:64] FLAG: --eviction-soft="" Oct 30 16:19:47.781121 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777069 2580 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 16:19:47.781121 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777072 2580 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 16:19:47.781121 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777075 2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 16:19:47.781121 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777078 2580 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 16:19:47.781121 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777081 2580 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 16:19:47.781121 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777084 2580 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 16:19:47.781121 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777087 2580 flags.go:64] FLAG: --feature-gates="" Oct 30 16:19:47.781121 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777091 2580 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 16:19:47.781121 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777094 2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 16:19:47.781121 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777097 2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 16:19:47.781121 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777100 2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 16:19:47.781121 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777103 2580 flags.go:64] FLAG: --healthz-port="10248" Oct 30 16:19:47.781121 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777107 2580 flags.go:64] FLAG: --help="false" Oct 30 16:19:47.781121 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777110 2580 flags.go:64] FLAG: --hostname-override="ip-10-0-131-136.ec2.internal" Oct 30 16:19:47.781121 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777113 2580 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 16:19:47.781121 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777116 2580 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 16:19:47.781121 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777120 2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 16:19:47.781121 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777123 2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 16:19:47.781121 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777128 2580 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 16:19:47.781683 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777131 2580 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 16:19:47.781683 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777134 2580 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 16:19:47.781683 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777138 2580 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 16:19:47.781683 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777141 2580 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 16:19:47.781683 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777144 2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 16:19:47.781683 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777148 2580 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 16:19:47.781683 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777151 2580 flags.go:64] FLAG: --kube-reserved="" Oct 30 16:19:47.781683 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777154 2580 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 16:19:47.781683 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777157 2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 16:19:47.781683 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777162 2580 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 16:19:47.781683 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777165 2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 16:19:47.781683 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777168 2580 flags.go:64] FLAG: --lock-file="" Oct 30 16:19:47.781683 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777172 2580 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 16:19:47.781683 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777175 2580 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 16:19:47.781683 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777178 2580 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 16:19:47.781683 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777183 2580 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 16:19:47.781683 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777187 2580 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 16:19:47.781683 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777190 2580 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 16:19:47.781683 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777193 2580 flags.go:64] FLAG: --logging-format="text" Oct 30 16:19:47.781683 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777196 2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 16:19:47.781683 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777199 2580 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 16:19:47.781683 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777202 2580 flags.go:64] FLAG: --manifest-url="" Oct 30 16:19:47.781683 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777205 2580 flags.go:64] FLAG: --manifest-url-header="" Oct 30 16:19:47.781683 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777210 2580 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 16:19:47.781683 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777213 2580 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 16:19:47.782270 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777217 2580 flags.go:64] FLAG: --max-pods="110" Oct 30 16:19:47.782270 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777220 2580 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 16:19:47.782270 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777224 2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 16:19:47.782270 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777227 2580 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 16:19:47.782270 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777230 2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 16:19:47.782270 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777233 2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 16:19:47.782270 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777237 2580 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 16:19:47.782270 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777240 2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 16:19:47.782270 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777248 2580 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 16:19:47.782270 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777251 2580 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 16:19:47.782270 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777255 2580 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 16:19:47.782270 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777268 2580 flags.go:64] FLAG: --pod-cidr="" Oct 30 16:19:47.782270 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777272 2580 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 16:19:47.782270 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777275 2580 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 16:19:47.782270 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777278 2580 flags.go:64] FLAG: --pods-per-core="0" Oct 30 16:19:47.782270 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777281 2580 flags.go:64] FLAG: --port="10250" Oct 30 16:19:47.782270 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777285 2580 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 16:19:47.782270 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777288 2580 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0237ca767909819be" Oct 30 16:19:47.782270 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777291 2580 flags.go:64] FLAG: --qos-reserved="" Oct 30 16:19:47.782270 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777295 2580 flags.go:64] FLAG: --read-only-port="10255" Oct 30 16:19:47.782270 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777298 2580 flags.go:64] FLAG: --register-node="true" Oct 30 16:19:47.782270 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777301 2580 flags.go:64] FLAG: --register-with-taints="" Oct 30 16:19:47.782270 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777304 2580 flags.go:64] FLAG: --registry-burst="10" Oct 30 16:19:47.782270 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777307 2580 flags.go:64] FLAG: --registry-qps="5" Oct 30 16:19:47.782270 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777310 2580 flags.go:64] FLAG: --reserved-cpus="" Oct 30 16:19:47.782270 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777313 2580 flags.go:64] FLAG: --reserved-memory="" Oct 30 16:19:47.782967 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777317 2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 16:19:47.782967 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777320 2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 16:19:47.782967 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777323 2580 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 16:19:47.782967 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777326 2580 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 16:19:47.782967 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777329 2580 flags.go:64] FLAG: --runonce="false" Oct 30 16:19:47.782967 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777332 2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 16:19:47.782967 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777335 2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 16:19:47.782967 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777338 2580 flags.go:64] FLAG: --seccomp-default="false" Oct 30 16:19:47.782967 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777341 2580 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 16:19:47.782967 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777344 2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 16:19:47.782967 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777347 2580 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 16:19:47.782967 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777350 2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 16:19:47.782967 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777355 2580 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 16:19:47.782967 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777358 2580 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 16:19:47.782967 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777361 2580 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 16:19:47.782967 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777364 2580 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 16:19:47.782967 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777367 2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 16:19:47.782967 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777371 2580 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 16:19:47.782967 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777374 2580 flags.go:64] FLAG: --system-cgroups="" Oct 30 16:19:47.782967 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777377 2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 16:19:47.782967 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777383 2580 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 16:19:47.782967 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777386 2580 flags.go:64] FLAG: --tls-cert-file="" Oct 30 16:19:47.782967 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777389 2580 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 16:19:47.782967 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777393 2580 flags.go:64] FLAG: --tls-min-version="" Oct 30 16:19:47.782967 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777396 2580 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 16:19:47.783568 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777399 2580 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 16:19:47.783568 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777402 2580 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 16:19:47.783568 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777405 2580 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 16:19:47.783568 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777408 2580 flags.go:64] FLAG: --v="2" Oct 30 16:19:47.783568 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777412 2580 flags.go:64] FLAG: --version="false" Oct 30 16:19:47.783568 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777416 2580 flags.go:64] FLAG: --vmodule="" Oct 30 16:19:47.783568 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777420 2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 16:19:47.783568 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777424 2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 16:19:47.783568 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777566 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:19:47.783568 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777571 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:19:47.783568 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777574 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:19:47.783568 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777577 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:19:47.783568 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777580 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:19:47.783568 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777582 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:19:47.783568 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777585 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:19:47.783568 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777587 2580 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:19:47.783568 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777590 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:19:47.783568 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777592 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:19:47.783568 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777595 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:19:47.783568 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777599 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:19:47.783568 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777602 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:19:47.783568 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777604 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:19:47.784104 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777607 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:19:47.784104 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777610 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:19:47.784104 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777613 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:19:47.784104 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777616 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:19:47.784104 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777618 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:19:47.784104 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777621 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:19:47.784104 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777623 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:19:47.784104 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777626 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:19:47.784104 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777629 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:19:47.784104 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777631 2580 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:19:47.784104 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777634 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:19:47.784104 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777636 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:19:47.784104 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777639 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:19:47.784104 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777641 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:19:47.784104 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777644 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:19:47.784104 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777647 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:19:47.784104 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777650 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:19:47.784104 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777652 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:19:47.784104 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777655 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:19:47.784573 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777658 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:19:47.784573 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777660 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:19:47.784573 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777663 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:19:47.784573 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777665 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:19:47.784573 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777668 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:19:47.784573 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777671 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:19:47.784573 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777673 2580 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:19:47.784573 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777676 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:19:47.784573 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777678 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:19:47.784573 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777681 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:19:47.784573 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777684 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:19:47.784573 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777688 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:19:47.784573 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777690 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:19:47.784573 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777693 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:19:47.784573 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777696 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:19:47.784573 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777698 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:19:47.784573 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777701 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:19:47.784573 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777704 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:19:47.784573 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777706 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:19:47.784573 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777709 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:19:47.785053 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777712 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:19:47.785053 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777714 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:19:47.785053 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777718 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:19:47.785053 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777722 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:19:47.785053 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777725 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:19:47.785053 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777728 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:19:47.785053 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777730 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:19:47.785053 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777733 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:19:47.785053 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777736 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:19:47.785053 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777738 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:19:47.785053 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777741 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:19:47.785053 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777743 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:19:47.785053 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777746 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:19:47.785053 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777750 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:19:47.785053 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777754 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:19:47.785053 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777758 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:19:47.785053 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777761 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:19:47.785053 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777763 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:19:47.785492 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777766 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:19:47.785492 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777769 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:19:47.785492 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777771 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:19:47.785492 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777774 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:19:47.785492 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777776 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:19:47.785492 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777779 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:19:47.785492 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777782 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:19:47.785492 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777784 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:19:47.785492 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777788 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:19:47.785492 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777791 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:19:47.785492 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777794 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:19:47.785492 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777796 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:19:47.785492 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777799 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:19:47.785492 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777802 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:19:47.785492 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777805 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:19:47.785492 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777807 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:19:47.785492 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777810 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:19:47.785492 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777812 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:19:47.785492 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777815 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:19:47.785492 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777818 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:19:47.785959 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777820 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:19:47.785959 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777823 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:19:47.785959 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777825 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:19:47.785959 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777828 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:19:47.785959 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777831 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:19:47.785959 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777833 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:19:47.785959 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777836 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:19:47.785959 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777838 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:19:47.785959 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777841 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:19:47.785959 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777844 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:19:47.785959 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777847 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:19:47.785959 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777849 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:19:47.785959 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777852 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:19:47.785959 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777870 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:19:47.785959 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777874 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:19:47.785959 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.777901 2580 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 16:19:47.786334 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777938 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:19:47.786334 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777942 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:19:47.786334 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777945 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:19:47.786334 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777948 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:19:47.786334 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777951 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:19:47.786334 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777955 2580 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:19:47.786334 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777957 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:19:47.786334 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777960 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:19:47.786334 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777962 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:19:47.786334 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777965 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:19:47.786334 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777968 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:19:47.786334 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777970 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:19:47.786334 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777973 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:19:47.786334 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777976 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:19:47.786334 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777978 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:19:47.786334 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777981 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:19:47.786334 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777983 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:19:47.786334 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777986 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:19:47.786334 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777989 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:19:47.786778 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777992 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:19:47.786778 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777994 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:19:47.786778 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777997 2580 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:19:47.786778 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.777999 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:19:47.786778 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.778002 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:19:47.786778 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.778004 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:19:47.786778 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.778007 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:19:47.786778 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.778010 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:19:47.786778 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.778013 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:19:47.786778 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.778016 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:19:47.786778 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.778018 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:19:47.786778 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.778021 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:19:47.786778 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.778023 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:19:47.786778 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.778026 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:19:47.786778 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.778028 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:19:47.786778 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.778031 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:19:47.786778 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.778033 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:19:47.786778 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.778036 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:19:47.786778 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.778041 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:19:47.787227 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.778045 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:19:47.787227 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.778048 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:19:47.787227 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.778051 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:19:47.787227 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.778053 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:19:47.787227 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.778056 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:19:47.787227 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.778059 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:19:47.787227 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.778062 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:19:47.787227 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.778064 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:19:47.787227 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.778067 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:19:47.787227 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.778070 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:19:47.787227 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.778076 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:19:47.787227 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.778080 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:19:47.787227 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.778084 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:19:47.787227 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.778087 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:19:47.787227 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.778090 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:19:47.787227 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.778094 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:19:47.787227 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.778097 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:19:47.787227 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.778100 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:19:47.787227 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.778102 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:19:47.787685 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.778105 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:19:47.787685 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.778108 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:19:47.787685 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.778110 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:19:47.787685 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.778113 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:19:47.787685 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.778116 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:19:47.787685 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.778118 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:19:47.787685 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.778121 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:19:47.787685 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.778124 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:19:47.787685 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.778126 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:19:47.787685 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.778129 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:19:47.787685 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.778132 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:19:47.787685 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.778134 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:19:47.787685 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.778138 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:19:47.787685 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.778141 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:19:47.787685 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.778143 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:19:47.787685 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.778146 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:19:47.787685 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.778149 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:19:47.787685 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.778151 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:19:47.787685 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.778154 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:19:47.788126 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.778157 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:19:47.788126 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.778159 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:19:47.788126 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.778162 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:19:47.788126 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.778164 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:19:47.788126 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.778168 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:19:47.788126 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.778171 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:19:47.788126 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.778174 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:19:47.788126 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.778177 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:19:47.788126 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.778179 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:19:47.788126 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.778182 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:19:47.788126 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.778185 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:19:47.788126 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.778188 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:19:47.788126 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.778190 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:19:47.788126 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.778193 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:19:47.788126 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.778195 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:19:47.788126 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.778198 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:19:47.788126 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.778201 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:19:47.788126 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.778203 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:19:47.788126 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.778206 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:19:47.788126 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.778208 2580 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:19:47.788690 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.778211 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:19:47.788690 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.778214 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:19:47.788690 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.778216 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:19:47.788690 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.778219 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:19:47.788690 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.778221 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:19:47.788690 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.778225 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:19:47.788690 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.778227 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:19:47.788690 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.778230 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:19:47.788690 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.778234 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:19:47.788690 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.778237 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:19:47.788690 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.785017 2580 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 16:19:47.788690 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.785033 2580 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 16:19:47.788690 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.785049 2580 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 16:19:47.788690 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.785054 2580 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 16:19:47.788690 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785111 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:19:47.788690 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785116 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:19:47.788690 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785120 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:19:47.788690 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785123 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:19:47.788690 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785126 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:19:47.789174 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785128 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:19:47.789174 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785131 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:19:47.789174 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785134 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:19:47.789174 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785137 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:19:47.789174 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785139 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:19:47.789174 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785142 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:19:47.789174 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785145 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:19:47.789174 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785147 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:19:47.789174 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785150 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:19:47.789174 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785153 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:19:47.789174 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785155 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:19:47.789174 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785158 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:19:47.789174 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785161 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:19:47.789174 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785163 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:19:47.789174 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785166 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:19:47.789174 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785169 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:19:47.789174 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785171 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:19:47.789174 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785174 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:19:47.789174 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785177 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:19:47.789174 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785179 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:19:47.789639 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785181 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:19:47.789639 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785184 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:19:47.789639 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785187 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:19:47.789639 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785191 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:19:47.789639 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785196 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:19:47.789639 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785200 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:19:47.789639 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785204 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:19:47.789639 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785207 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:19:47.789639 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785210 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:19:47.789639 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785213 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:19:47.789639 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785215 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:19:47.789639 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785218 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:19:47.789639 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785221 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:19:47.789639 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785224 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:19:47.789639 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785227 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:19:47.789639 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785230 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:19:47.789639 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785233 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:19:47.789639 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785235 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:19:47.789639 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785238 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:19:47.790080 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785241 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:19:47.790080 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785243 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:19:47.790080 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785246 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:19:47.790080 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785248 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:19:47.790080 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785251 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:19:47.790080 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785254 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:19:47.790080 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785275 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:19:47.790080 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785280 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:19:47.790080 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785282 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:19:47.790080 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785286 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:19:47.790080 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785288 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:19:47.790080 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785291 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:19:47.790080 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785294 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:19:47.790080 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785297 2580 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:19:47.790080 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785299 2580 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:19:47.790080 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785302 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:19:47.790080 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785305 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:19:47.790080 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785308 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:19:47.790080 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785310 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:19:47.790080 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785314 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:19:47.790540 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785317 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:19:47.790540 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785320 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:19:47.790540 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785322 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:19:47.790540 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785326 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:19:47.790540 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785330 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:19:47.790540 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785333 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:19:47.790540 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785336 2580 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:19:47.790540 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785339 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:19:47.790540 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785341 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:19:47.790540 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785344 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:19:47.790540 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785347 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:19:47.790540 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785350 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:19:47.790540 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785352 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:19:47.790540 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785355 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:19:47.790540 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785357 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:19:47.790540 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785360 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:19:47.790540 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785363 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:19:47.790540 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785365 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:19:47.790540 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785368 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:19:47.790540 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785370 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:19:47.791027 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785373 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:19:47.791027 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785376 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:19:47.791027 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785378 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:19:47.791027 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785381 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:19:47.791027 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785383 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:19:47.791027 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785386 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:19:47.791027 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785388 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:19:47.791027 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785391 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:19:47.791027 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785394 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:19:47.791027 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785397 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:19:47.791027 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785400 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:19:47.791027 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785403 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:19:47.791027 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785406 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:19:47.791027 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785409 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:19:47.791027 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785411 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:19:47.791027 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785414 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:19:47.791027 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785417 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:19:47.791027 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785420 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:19:47.791027 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785422 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:19:47.791510 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785425 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:19:47.791510 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785427 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:19:47.791510 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785430 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:19:47.791510 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.785459 2580 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 16:19:47.791510 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785603 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:19:47.791510 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785608 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:19:47.791510 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785612 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:19:47.791510 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785615 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:19:47.791510 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785618 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:19:47.791510 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785621 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:19:47.791510 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785624 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:19:47.791510 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785627 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:19:47.791510 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785631 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:19:47.791510 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785634 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:19:47.791510 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785637 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:19:47.791891 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785640 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:19:47.791891 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785642 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:19:47.791891 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785645 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:19:47.791891 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785647 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:19:47.791891 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785650 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:19:47.791891 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785652 2580 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:19:47.791891 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785655 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:19:47.791891 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785658 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:19:47.791891 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785661 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:19:47.791891 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785663 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:19:47.791891 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785666 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:19:47.791891 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785675 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:19:47.791891 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785679 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:19:47.791891 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785682 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:19:47.791891 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785684 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:19:47.791891 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785701 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:19:47.791891 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785705 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:19:47.791891 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785709 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:19:47.791891 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785712 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:19:47.791891 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785714 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:19:47.792402 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785717 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:19:47.792402 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785719 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:19:47.792402 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785722 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:19:47.792402 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785724 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:19:47.792402 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785727 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:19:47.792402 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785730 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:19:47.792402 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785732 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:19:47.792402 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785735 2580 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:19:47.792402 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785737 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:19:47.792402 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785740 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:19:47.792402 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785743 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:19:47.792402 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785745 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:19:47.792402 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785748 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:19:47.792402 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785750 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:19:47.792402 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785753 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:19:47.792402 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785755 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:19:47.792402 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785758 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:19:47.792402 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785760 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:19:47.792402 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785763 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:19:47.792402 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785766 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:19:47.792869 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785768 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:19:47.792869 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785771 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:19:47.792869 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785773 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:19:47.792869 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785775 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:19:47.792869 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785783 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:19:47.792869 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785786 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:19:47.792869 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785788 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:19:47.792869 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785791 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:19:47.792869 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785793 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:19:47.792869 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785796 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:19:47.792869 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785798 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:19:47.792869 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785801 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:19:47.792869 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785803 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:19:47.792869 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785806 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:19:47.792869 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785808 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:19:47.792869 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785811 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:19:47.792869 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785813 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:19:47.792869 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785816 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:19:47.792869 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785818 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:19:47.792869 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785821 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:19:47.793462 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785823 2580 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:19:47.793462 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785826 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:19:47.793462 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785829 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:19:47.793462 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785831 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:19:47.793462 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785834 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:19:47.793462 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785836 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:19:47.793462 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785839 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:19:47.793462 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785842 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:19:47.793462 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785844 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:19:47.793462 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785847 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:19:47.793462 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785849 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:19:47.793462 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785852 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:19:47.793462 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785855 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:19:47.793462 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785857 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:19:47.793462 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785860 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:19:47.793462 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785862 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:19:47.793462 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785865 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:19:47.793462 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785868 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:19:47.793462 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785871 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:19:47.793990 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785873 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:19:47.793990 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785877 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:19:47.793990 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785880 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:19:47.793990 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785883 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:19:47.793990 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785886 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:19:47.793990 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785889 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:19:47.793990 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785891 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:19:47.793990 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785894 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:19:47.793990 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785896 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:19:47.793990 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785899 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:19:47.793990 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785902 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:19:47.793990 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785904 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:19:47.793990 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785907 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:19:47.793990 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785910 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:19:47.793990 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785912 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:19:47.793990 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785915 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:19:47.794388 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.785937 2580 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 16:19:47.794388 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785970 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:19:47.794388 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785974 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:19:47.794388 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785977 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:19:47.794388 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785980 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:19:47.794388 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785983 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:19:47.794388 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785986 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:19:47.794388 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785988 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:19:47.794388 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785991 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:19:47.794388 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785993 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:19:47.794388 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.785997 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:19:47.794388 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.786001 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:19:47.794388 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.786004 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:19:47.794388 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.786006 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:19:47.794388 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.786009 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:19:47.794751 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.786012 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:19:47.794751 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.786015 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:19:47.794751 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.786017 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:19:47.794751 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.786020 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:19:47.794751 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.786022 2580 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:19:47.794751 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.786024 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:19:47.794751 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.786027 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:19:47.794751 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.786029 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:19:47.794751 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.786032 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:19:47.794751 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.786034 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:19:47.794751 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.786037 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:19:47.794751 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.786040 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:19:47.794751 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.786052 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:19:47.794751 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.786055 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:19:47.794751 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.786057 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:19:47.794751 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.786060 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:19:47.794751 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.786065 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:19:47.794751 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.786068 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:19:47.794751 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.786070 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:19:47.794751 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.786073 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:19:47.795212 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.786075 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:19:47.795212 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.786078 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:19:47.795212 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.786080 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:19:47.795212 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.786083 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:19:47.795212 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.786085 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:19:47.795212 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.786088 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:19:47.795212 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.786090 2580 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:19:47.795212 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.786093 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:19:47.795212 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.786095 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:19:47.795212 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.786098 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:19:47.795212 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.786100 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:19:47.795212 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.786103 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:19:47.795212 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.786106 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:19:47.795212 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.786109 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:19:47.795212 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.786112 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:19:47.795212 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.786114 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:19:47.795212 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.786117 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:19:47.795212 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.786120 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:19:47.795212 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.786122 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:19:47.795212 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.786125 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:19:47.795738 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.786127 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:19:47.795738 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.786130 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:19:47.795738 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.786132 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:19:47.795738 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.786137 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:19:47.795738 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.786140 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:19:47.795738 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.786142 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:19:47.795738 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.786145 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:19:47.795738 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.786148 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:19:47.795738 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.786152 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:19:47.795738 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.786154 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:19:47.795738 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.786157 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:19:47.795738 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.786159 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:19:47.795738 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.786162 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:19:47.795738 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.786164 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:19:47.795738 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.786167 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:19:47.795738 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.786169 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:19:47.795738 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.786172 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:19:47.795738 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.786176 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:19:47.795738 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.786178 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:19:47.795738 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.786181 2580 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:19:47.796228 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.786184 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:19:47.796228 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.786186 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:19:47.796228 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.786188 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:19:47.796228 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.786191 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:19:47.796228 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.786194 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:19:47.796228 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.786196 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:19:47.796228 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.786199 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:19:47.796228 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.786202 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:19:47.796228 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.786205 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:19:47.796228 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.786207 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:19:47.796228 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.786210 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:19:47.796228 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.786213 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:19:47.796228 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.786215 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:19:47.796228 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.786218 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:19:47.796228 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.786220 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:19:47.796228 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.786224 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:19:47.796228 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.786227 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:19:47.796228 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.786229 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:19:47.796228 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.786232 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:19:47.796759 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.786236 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:19:47.796759 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.786241 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:19:47.796759 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.786244 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:19:47.796759 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.786247 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:19:47.796759 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.786252 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:19:47.796759 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.786254 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:19:47.796759 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.786270 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:19:47.796759 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.786274 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:19:47.796759 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.786276 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:19:47.796759 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.786279 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:19:47.796759 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.786282 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:19:47.796759 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.786285 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:19:47.796759 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:47.786287 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:19:47.796759 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.786341 2580 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 16:19:47.796759 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.788487 2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 16:19:47.796759 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.789316 2580 server.go:1014] "Starting client certificate rotation" Oct 30 16:19:47.796759 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.789410 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 16:19:47.796759 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.790018 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 16:19:47.797189 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.791067 2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 16:19:47.797189 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.793825 2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 16:19:47.803735 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.803716 2580 log.go:25] "Validated CRI v1 runtime API" Oct 30 16:19:47.807105 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.807092 2580 log.go:25] "Validated CRI v1 image API" Oct 30 16:19:47.807951 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.807924 2580 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 16:19:47.812375 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.812356 2580 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 966e0b7a-d5cb-468d-9551-a1038e613cba:/dev/nvme0n1p4 9d3cbe91-2e74-4aa6-ba44-7a5ec42b0a03:/dev/nvme0n1p3] Oct 30 16:19:47.812432 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.812374 2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 16:19:47.818188 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.818080 2580 manager.go:217] Machine: {Timestamp:2026-10-30 16:19:47.816826439 +0000 UTC m=+0.294434763 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3097318 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2cb508e032bccac7092b234caaa274 SystemUUID:ec2cb508-e032-bcca-c709-2b234caaa274 BootID:e42287c7-73b0-4523-bb5c-d0439396016a Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:c9:8e:d1:67 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:c9:8e:d1:67 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:22:c9:b7:15:fe:4c Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 16:19:47.818188 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.818186 2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 16:19:47.818324 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.818312 2580 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 16:19:47.820630 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.820607 2580 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 16:19:47.820767 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.820631 2580 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-136.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 16:19:47.820818 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.820772 2580 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 16:19:47.820818 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.820780 2580 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 16:19:47.820818 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.820813 2580 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 16:19:47.820899 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.820827 2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 16:19:47.821970 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.821959 2580 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 16:19:47.825966 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.825954 2580 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 16:19:47.826089 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.826079 2580 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 16:19:47.826754 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.826744 2580 kubelet.go:493] "Attempting to sync node with API server" Oct 30 16:19:47.826790 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.826763 2580 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 16:19:47.826790 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.826780 2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 16:19:47.826851 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.826793 2580 kubelet.go:405] "Adding apiserver pod source" Oct 30 16:19:47.826851 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.826804 2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 16:19:47.827228 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.827215 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 16:19:47.829131 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.829112 2580 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 16:19:47.834394 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.834379 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 16:19:47.835238 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.835227 2580 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 16:19:47.835290 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.835271 2580 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 16:19:47.835426 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.835416 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 16:19:47.835467 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.835432 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 16:19:47.835467 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.835439 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 16:19:47.835467 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.835445 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 16:19:47.835467 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.835451 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 16:19:47.835467 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.835456 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 16:19:47.835989 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.835981 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 16:19:47.836017 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.835990 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 16:19:47.836017 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.835997 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 16:19:47.836017 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.836005 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 16:19:47.836017 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.836014 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 16:19:47.836131 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.836022 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 16:19:47.836637 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.836629 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 16:19:47.836667 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.836638 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 16:19:47.840377 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.840363 2580 server.go:1263] "Started kubelet" Oct 30 16:19:47.840444 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.840420 2580 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 16:19:47.841254 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.841109 2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 16:19:47.841361 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.841325 2580 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 16:19:47.841253 ip-10-0-131-136 systemd[1]: Started Kubernetes Kubelet. Oct 30 16:19:47.844404 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.844389 2580 server.go:317] "Adding debug handlers to kubelet server" Oct 30 16:19:47.844629 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.844607 2580 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 16:19:47.848804 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:19:47.848778 2580 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 16:19:47.849698 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.849090 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 16:19:47.849698 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.849653 2580 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 16:19:47.849799 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:19:47.849754 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-136.ec2.internal\" not found" Oct 30 16:19:47.850327 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.850306 2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 16:19:47.850596 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.850582 2580 factory.go:55] Registering systemd factory Oct 30 16:19:47.850637 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.850602 2580 factory.go:223] Registration of the systemd container factory successfully Oct 30 16:19:47.850824 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.850810 2580 factory.go:153] Registering CRI-O factory Oct 30 16:19:47.850824 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.850825 2580 factory.go:223] Registration of the crio container factory successfully Oct 30 16:19:47.850903 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.850874 2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 16:19:47.850903 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.850895 2580 factory.go:103] Registering Raw factory Oct 30 16:19:47.850972 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.850912 2580 manager.go:1199] Started watching for new ooms in manager Oct 30 16:19:47.851819 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.851803 2580 manager.go:319] Starting recovery of all containers Oct 30 16:19:47.851908 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.851892 2580 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 16:19:47.851971 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.851912 2580 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 16:19:47.852044 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.852032 2580 reconstruct.go:97] "Volume reconstruction finished" Oct 30 16:19:47.852086 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.852045 2580 reconciler.go:29] "Reconciler: start to sync state" Oct 30 16:19:47.861599 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.861578 2580 manager.go:324] Recovery completed Oct 30 16:19:47.867079 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.867049 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 16:19:47.868925 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.868905 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-136.ec2.internal" event="NodeHasSufficientMemory" Oct 30 16:19:47.869009 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.868943 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-136.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 16:19:47.869009 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.868958 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-136.ec2.internal" event="NodeHasSufficientPID" Oct 30 16:19:47.869559 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.869543 2580 cpu_manager.go:226] "Starting" policy="none" Oct 30 16:19:47.869559 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.869552 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 16:19:47.869559 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.869556 2580 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 16:19:47.869701 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.869574 2580 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 16:19:47.870286 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.870245 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-136.ec2.internal" event="NodeHasSufficientMemory" Oct 30 16:19:47.870343 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.870288 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-136.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 16:19:47.870343 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.870301 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-136.ec2.internal" event="NodeHasSufficientPID" Oct 30 16:19:47.871769 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.871757 2580 policy_none.go:50] "Start" Oct 30 16:19:47.871812 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.871773 2580 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 16:19:47.871812 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.871783 2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 16:19:47.873781 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.873770 2580 policy_none.go:44] "Start" Oct 30 16:19:47.887245 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.887213 2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-136.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 16:19:47.887355 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:19:47.887214 2580 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-136.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 16:19:47.888152 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:19:47.887245 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-136.ec2.internal.18e35b0cf5831d22 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-136.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-136.ec2.internal,},FirstTimestamp:2026-10-30 16:19:47.840335138 +0000 UTC m=+0.317943461,LastTimestamp:2026-10-30 16:19:47.840335138 +0000 UTC m=+0.317943461,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-136.ec2.internal,}" Oct 30 16:19:47.894430 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:19:47.894403 2580 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 30 16:19:47.904530 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.904499 2580 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 16:19:47.906975 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.906961 2580 manager.go:341] "Starting Device Plugin manager" Oct 30 16:19:47.907025 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:19:47.907005 2580 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 16:19:47.907025 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.907019 2580 server.go:87] "Starting device plugin registration server" Oct 30 16:19:47.907288 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.907271 2580 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 16:19:47.907369 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.907287 2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 16:19:47.907473 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.907452 2580 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 16:19:47.907758 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.907570 2580 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 16:19:47.907758 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.907580 2580 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 16:19:47.907949 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:19:47.907929 2580 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 16:19:47.908030 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:19:47.907976 2580 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-136.ec2.internal\" not found" Oct 30 16:19:47.919803 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:19:47.919780 2580 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-136.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 16:19:47.919910 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:19:47.919785 2580 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 16:19:47.919910 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:19:47.919802 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-136.ec2.internal.18e35b0cf7375945 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-136.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-136.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-136.ec2.internal,},FirstTimestamp:2026-10-30 16:19:47.868924229 +0000 UTC m=+0.346532553,LastTimestamp:2026-10-30 16:19:47.868924229 +0000 UTC m=+0.346532553,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-136.ec2.internal,}" Oct 30 16:19:47.955403 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:19:47.946522 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-136.ec2.internal.18e35b0cf737bce7 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-136.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-131-136.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-131-136.ec2.internal,},FirstTimestamp:2026-10-30 16:19:47.868949735 +0000 UTC m=+0.346558059,LastTimestamp:2026-10-30 16:19:47.868949735 +0000 UTC m=+0.346558059,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-136.ec2.internal,}" Oct 30 16:19:47.972009 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:19:47.971941 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-136.ec2.internal.18e35b0cf737f63d default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-136.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-131-136.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-131-136.ec2.internal,},FirstTimestamp:2026-10-30 16:19:47.868964413 +0000 UTC m=+0.346572736,LastTimestamp:2026-10-30 16:19:47.868964413 +0000 UTC m=+0.346572736,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-136.ec2.internal,}" Oct 30 16:19:47.979394 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.979371 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 16:19:47.980594 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.980576 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 16:19:47.980675 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.980598 2580 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 16:19:47.980675 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:47.980615 2580 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 16:19:47.980675 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:19:47.980645 2580 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 16:19:48.000942 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:19:48.000856 2580 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-131-136.ec2.internal.18e35b0cf7375945\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-136.ec2.internal.18e35b0cf7375945 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-136.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-136.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-136.ec2.internal,},FirstTimestamp:2026-10-30 16:19:47.868924229 +0000 UTC m=+0.346532553,LastTimestamp:2026-10-30 16:19:47.870271059 +0000 UTC m=+0.347879384,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-136.ec2.internal,}" Oct 30 16:19:48.007786 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:48.007772 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 16:19:48.010409 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:48.010388 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-136.ec2.internal" event="NodeHasSufficientMemory" Oct 30 16:19:48.010499 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:48.010425 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-136.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 16:19:48.010499 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:48.010439 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-136.ec2.internal" event="NodeHasSufficientPID" Oct 30 16:19:48.010499 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:48.010467 2580 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-131-136.ec2.internal" Oct 30 16:19:48.022525 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:19:48.022422 2580 reflector.go:204] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" Oct 30 16:19:48.022525 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:19:48.022403 2580 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-131-136.ec2.internal.18e35b0cf737bce7\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-136.ec2.internal.18e35b0cf737bce7 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-136.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-131-136.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-131-136.ec2.internal,},FirstTimestamp:2026-10-30 16:19:47.868949735 +0000 UTC m=+0.346558059,LastTimestamp:2026-10-30 16:19:47.870295037 +0000 UTC m=+0.347903360,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-136.ec2.internal,}" Oct 30 16:19:48.057050 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:19:48.056951 2580 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-131-136.ec2.internal.18e35b0cf737f63d\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-136.ec2.internal.18e35b0cf737f63d default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-136.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-131-136.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-131-136.ec2.internal,},FirstTimestamp:2026-10-30 16:19:47.868964413 +0000 UTC m=+0.346572736,LastTimestamp:2026-10-30 16:19:47.870304924 +0000 UTC m=+0.347913248,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-136.ec2.internal,}" Oct 30 16:19:48.077713 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:19:48.077680 2580 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-131-136.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-131-136.ec2.internal" Oct 30 16:19:48.077860 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:19:48.077741 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-136.ec2.internal.18e35b0cf97b2cbe default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-136.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated limits on system reserved cgroup /system.slice,Source:EventSource{Component:kubelet,Host:ip-10-0-131-136.ec2.internal,},FirstTimestamp:2026-10-30 16:19:47.90692371 +0000 UTC m=+0.384532036,LastTimestamp:2026-10-30 16:19:47.90692371 +0000 UTC m=+0.384532036,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-136.ec2.internal,}" Oct 30 16:19:48.081516 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:48.081500 2580 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-136.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-136.ec2.internal"] Oct 30 16:19:48.081587 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:48.081552 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 16:19:48.082451 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:48.082436 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-136.ec2.internal" event="NodeHasSufficientMemory" Oct 30 16:19:48.082540 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:48.082468 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-136.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 16:19:48.082540 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:48.082484 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-136.ec2.internal" event="NodeHasSufficientPID" Oct 30 16:19:48.084876 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:48.084861 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 16:19:48.085032 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:48.085017 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-136.ec2.internal" Oct 30 16:19:48.085078 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:48.085048 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 16:19:48.085612 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:48.085596 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-136.ec2.internal" event="NodeHasSufficientMemory" Oct 30 16:19:48.085692 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:48.085628 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-136.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 16:19:48.085692 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:48.085643 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-136.ec2.internal" event="NodeHasSufficientPID" Oct 30 16:19:48.085692 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:48.085600 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-136.ec2.internal" event="NodeHasSufficientMemory" Oct 30 16:19:48.085835 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:48.085697 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-136.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 16:19:48.085835 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:48.085706 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-136.ec2.internal" event="NodeHasSufficientPID" Oct 30 16:19:48.087854 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:48.087839 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-136.ec2.internal" Oct 30 16:19:48.087924 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:48.087871 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 16:19:48.088650 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:48.088634 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-136.ec2.internal" event="NodeHasSufficientMemory" Oct 30 16:19:48.088718 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:48.088666 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-136.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 16:19:48.088718 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:48.088682 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-136.ec2.internal" event="NodeHasSufficientPID" Oct 30 16:19:48.107615 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:19:48.107537 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-136.ec2.internal.18e35b0cfa7aa63c default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-136.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated Node Allocatable limit across pods,Source:EventSource{Component:kubelet,Host:ip-10-0-131-136.ec2.internal,},FirstTimestamp:2026-10-30 16:19:47.923666492 +0000 UTC m=+0.401274804,LastTimestamp:2026-10-30 16:19:47.923666492 +0000 UTC m=+0.401274804,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-136.ec2.internal,}" Oct 30 16:19:48.107615 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:19:48.107603 2580 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-136.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="400ms" Oct 30 16:19:48.118589 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:19:48.118520 2580 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-131-136.ec2.internal.18e35b0cf7375945\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-136.ec2.internal.18e35b0cf7375945 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-136.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-136.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-136.ec2.internal,},FirstTimestamp:2026-10-30 16:19:47.868924229 +0000 UTC m=+0.346532553,LastTimestamp:2026-10-30 16:19:48.010406663 +0000 UTC m=+0.488014987,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-136.ec2.internal,}" Oct 30 16:19:48.120927 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:19:48.120909 2580 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-136.ec2.internal\" not found" node="ip-10-0-131-136.ec2.internal" Oct 30 16:19:48.122967 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:19:48.122952 2580 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-136.ec2.internal\" not found" node="ip-10-0-131-136.ec2.internal" Oct 30 16:19:48.154000 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:48.153976 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9ce644f9016009147990b020805ff06d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-136.ec2.internal\" (UID: \"9ce644f9016009147990b020805ff06d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-136.ec2.internal" Oct 30 16:19:48.154093 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:48.154003 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9ce644f9016009147990b020805ff06d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-136.ec2.internal\" (UID: \"9ce644f9016009147990b020805ff06d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-136.ec2.internal" Oct 30 16:19:48.154093 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:48.154018 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3de5cf7923bfc1c5b7b74a0700d7b3fa-config\") pod \"kube-apiserver-proxy-ip-10-0-131-136.ec2.internal\" (UID: \"3de5cf7923bfc1c5b7b74a0700d7b3fa\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-136.ec2.internal" Oct 30 16:19:48.161874 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:19:48.161807 2580 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-131-136.ec2.internal.18e35b0cf737bce7\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-136.ec2.internal.18e35b0cf737bce7 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-136.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-131-136.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-131-136.ec2.internal,},FirstTimestamp:2026-10-30 16:19:47.868949735 +0000 UTC m=+0.346558059,LastTimestamp:2026-10-30 16:19:48.010431179 +0000 UTC m=+0.488039503,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-136.ec2.internal,}" Oct 30 16:19:48.199662 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:19:48.199579 2580 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-131-136.ec2.internal.18e35b0cf737f63d\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-136.ec2.internal.18e35b0cf737f63d default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-136.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-131-136.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-131-136.ec2.internal,},FirstTimestamp:2026-10-30 16:19:47.868964413 +0000 UTC m=+0.346572736,LastTimestamp:2026-10-30 16:19:48.010444777 +0000 UTC m=+0.488053101,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-136.ec2.internal,}" Oct 30 16:19:48.215624 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:19:48.215552 2580 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-131-136.ec2.internal.18e35b0cf7375945\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-136.ec2.internal.18e35b0cf7375945 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-136.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-136.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-136.ec2.internal,},FirstTimestamp:2026-10-30 16:19:47.868924229 +0000 UTC m=+0.346532553,LastTimestamp:2026-10-30 16:19:48.082450311 +0000 UTC m=+0.560058641,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-136.ec2.internal,}" Oct 30 16:19:48.251231 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:19:48.251167 2580 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-131-136.ec2.internal.18e35b0cf737bce7\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-136.ec2.internal.18e35b0cf737bce7 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-136.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-131-136.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-131-136.ec2.internal,},FirstTimestamp:2026-10-30 16:19:47.868949735 +0000 UTC m=+0.346558059,LastTimestamp:2026-10-30 16:19:48.082474899 +0000 UTC m=+0.560083224,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-136.ec2.internal,}" Oct 30 16:19:48.254925 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:48.254908 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9ce644f9016009147990b020805ff06d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-136.ec2.internal\" (UID: \"9ce644f9016009147990b020805ff06d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-136.ec2.internal" Oct 30 16:19:48.254975 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:48.254933 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9ce644f9016009147990b020805ff06d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-136.ec2.internal\" (UID: \"9ce644f9016009147990b020805ff06d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-136.ec2.internal" Oct 30 16:19:48.254975 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:48.254948 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3de5cf7923bfc1c5b7b74a0700d7b3fa-config\") pod \"kube-apiserver-proxy-ip-10-0-131-136.ec2.internal\" (UID: \"3de5cf7923bfc1c5b7b74a0700d7b3fa\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-136.ec2.internal" Oct 30 16:19:48.255050 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:48.255017 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3de5cf7923bfc1c5b7b74a0700d7b3fa-config\") pod \"kube-apiserver-proxy-ip-10-0-131-136.ec2.internal\" (UID: \"3de5cf7923bfc1c5b7b74a0700d7b3fa\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-136.ec2.internal" Oct 30 16:19:48.255050 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:48.255017 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9ce644f9016009147990b020805ff06d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-136.ec2.internal\" (UID: \"9ce644f9016009147990b020805ff06d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-136.ec2.internal" Oct 30 16:19:48.255050 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:48.255017 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9ce644f9016009147990b020805ff06d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-136.ec2.internal\" (UID: \"9ce644f9016009147990b020805ff06d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-136.ec2.internal" Oct 30 16:19:48.278048 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:48.278009 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 16:19:48.278862 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:48.278848 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-136.ec2.internal" event="NodeHasSufficientMemory" Oct 30 16:19:48.278919 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:48.278875 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-136.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 16:19:48.278919 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:48.278886 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-136.ec2.internal" event="NodeHasSufficientPID" Oct 30 16:19:48.278983 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:48.278920 2580 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-131-136.ec2.internal" Oct 30 16:19:48.303983 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:19:48.303907 2580 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-131-136.ec2.internal.18e35b0cf737f63d\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-136.ec2.internal.18e35b0cf737f63d default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-136.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-131-136.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-131-136.ec2.internal,},FirstTimestamp:2026-10-30 16:19:47.868964413 +0000 UTC m=+0.346572736,LastTimestamp:2026-10-30 16:19:48.082489404 +0000 UTC m=+0.560097729,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-136.ec2.internal,}" Oct 30 16:19:48.324402 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:19:48.324378 2580 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-131-136.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-131-136.ec2.internal" Oct 30 16:19:48.324459 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:19:48.324387 2580 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-131-136.ec2.internal.18e35b0cf7375945\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-136.ec2.internal.18e35b0cf7375945 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-136.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-136.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-136.ec2.internal,},FirstTimestamp:2026-10-30 16:19:47.868924229 +0000 UTC m=+0.346532553,LastTimestamp:2026-10-30 16:19:48.085616564 +0000 UTC m=+0.563224889,Count:5,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-136.ec2.internal,}" Oct 30 16:19:48.351656 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:19:48.351599 2580 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-131-136.ec2.internal.18e35b0cf737bce7\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-136.ec2.internal.18e35b0cf737bce7 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-136.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-131-136.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-131-136.ec2.internal,},FirstTimestamp:2026-10-30 16:19:47.868949735 +0000 UTC m=+0.346558059,LastTimestamp:2026-10-30 16:19:48.085634393 +0000 UTC m=+0.563242724,Count:5,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-136.ec2.internal,}" Oct 30 16:19:48.374124 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:19:48.374055 2580 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-131-136.ec2.internal.18e35b0cf737f63d\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-136.ec2.internal.18e35b0cf737f63d default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-136.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-131-136.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-131-136.ec2.internal,},FirstTimestamp:2026-10-30 16:19:47.868964413 +0000 UTC m=+0.346572736,LastTimestamp:2026-10-30 16:19:48.085649453 +0000 UTC m=+0.563257778,Count:5,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-136.ec2.internal,}" Oct 30 16:19:48.402846 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:19:48.400950 2580 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-131-136.ec2.internal.18e35b0cf7375945\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-136.ec2.internal.18e35b0cf7375945 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-136.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-136.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-136.ec2.internal,},FirstTimestamp:2026-10-30 16:19:47.868924229 +0000 UTC m=+0.346532553,LastTimestamp:2026-10-30 16:19:48.085688044 +0000 UTC m=+0.563296368,Count:6,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-136.ec2.internal,}" Oct 30 16:19:48.419185 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:19:48.419102 2580 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-131-136.ec2.internal.18e35b0cf737bce7\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-136.ec2.internal.18e35b0cf737bce7 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-136.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-131-136.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-131-136.ec2.internal,},FirstTimestamp:2026-10-30 16:19:47.868949735 +0000 UTC m=+0.346558059,LastTimestamp:2026-10-30 16:19:48.085701796 +0000 UTC m=+0.563310118,Count:6,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-136.ec2.internal,}" Oct 30 16:19:48.423898 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:48.423880 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-136.ec2.internal" Oct 30 16:19:48.427189 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:48.427174 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-136.ec2.internal" Oct 30 16:19:48.457613 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:19:48.457345 2580 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-131-136.ec2.internal.18e35b0cf737f63d\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-136.ec2.internal.18e35b0cf737f63d default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-136.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-131-136.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-131-136.ec2.internal,},FirstTimestamp:2026-10-30 16:19:47.868964413 +0000 UTC m=+0.346572736,LastTimestamp:2026-10-30 16:19:48.085710391 +0000 UTC m=+0.563318715,Count:6,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-136.ec2.internal,}" Oct 30 16:19:48.508074 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:19:48.507985 2580 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-131-136.ec2.internal.18e35b0cf7375945\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-136.ec2.internal.18e35b0cf7375945 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-136.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-136.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-136.ec2.internal,},FirstTimestamp:2026-10-30 16:19:47.868924229 +0000 UTC m=+0.346532553,LastTimestamp:2026-10-30 16:19:48.088650274 +0000 UTC m=+0.566258602,Count:7,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-136.ec2.internal,}" Oct 30 16:19:48.554147 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:19:48.554076 2580 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-136.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="800ms" Oct 30 16:19:48.554272 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:19:48.554110 2580 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-131-136.ec2.internal.18e35b0cf737bce7\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-136.ec2.internal.18e35b0cf737bce7 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-136.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-131-136.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-131-136.ec2.internal,},FirstTimestamp:2026-10-30 16:19:47.868949735 +0000 UTC m=+0.346558059,LastTimestamp:2026-10-30 16:19:48.088673699 +0000 UTC m=+0.566282028,Count:7,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-136.ec2.internal,}" Oct 30 16:19:48.605699 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:19:48.605624 2580 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-131-136.ec2.internal.18e35b0cf737f63d\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-136.ec2.internal.18e35b0cf737f63d default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-136.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-131-136.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-131-136.ec2.internal,},FirstTimestamp:2026-10-30 16:19:47.868964413 +0000 UTC m=+0.346572736,LastTimestamp:2026-10-30 16:19:48.088688475 +0000 UTC m=+0.566296802,Count:7,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-136.ec2.internal,}" Oct 30 16:19:48.651800 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:19:48.651733 2580 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-131-136.ec2.internal.18e35b0cf7375945\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-136.ec2.internal.18e35b0cf7375945 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-136.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-136.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-136.ec2.internal,},FirstTimestamp:2026-10-30 16:19:47.868924229 +0000 UTC m=+0.346532553,LastTimestamp:2026-10-30 16:19:48.278862239 +0000 UTC m=+0.756470562,Count:8,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-136.ec2.internal,}" Oct 30 16:19:48.724975 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:48.724954 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 16:19:48.725881 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:48.725867 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-136.ec2.internal" event="NodeHasSufficientMemory" Oct 30 16:19:48.725957 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:48.725893 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-136.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 16:19:48.725957 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:48.725904 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-136.ec2.internal" event="NodeHasSufficientPID" Oct 30 16:19:48.725957 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:48.725929 2580 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-131-136.ec2.internal" Oct 30 16:19:48.817140 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:19:48.817080 2580 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-131-136.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-131-136.ec2.internal" Oct 30 16:19:48.898182 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:48.898151 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ce644f9016009147990b020805ff06d.slice/crio-622b6489c48a29a9d5eae0856d7d353098c922549fe22891407212db1d417313 WatchSource:0}: Error finding container 622b6489c48a29a9d5eae0856d7d353098c922549fe22891407212db1d417313: Status 404 returned error can't find the container with id 622b6489c48a29a9d5eae0856d7d353098c922549fe22891407212db1d417313 Oct 30 16:19:48.902749 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:48.902734 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 16:19:48.927058 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:19:48.927037 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3de5cf7923bfc1c5b7b74a0700d7b3fa.slice/crio-cff2925bc3a36c2f6ed0ba95b40b16b4cf5bf4b6d270f52efbc14bef48eef600 WatchSource:0}: Error finding container cff2925bc3a36c2f6ed0ba95b40b16b4cf5bf4b6d270f52efbc14bef48eef600: Status 404 returned error can't find the container with id cff2925bc3a36c2f6ed0ba95b40b16b4cf5bf4b6d270f52efbc14bef48eef600 Oct 30 16:19:48.936739 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:48.936713 2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-136.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 16:19:48.936935 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:19:48.936862 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-131-136.ec2.internal.18e35b0d34e3c56a openshift-machine-config-operator 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-131-136.ec2.internal,UID:9ce644f9016009147990b020805ff06d,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Pulling,Message:Pulling image \"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d017b7d199b1d54e889bb1eb6e572d8e6d94ad92610894009ee0562c2a5c0f51\",Source:EventSource{Component:kubelet,Host:ip-10-0-131-136.ec2.internal,},FirstTimestamp:2026-10-30 16:19:48.903634282 +0000 UTC m=+1.381242592,LastTimestamp:2026-10-30 16:19:48.903634282 +0000 UTC m=+1.381242592,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-136.ec2.internal,}" Oct 30 16:19:50.161207 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:50.161154 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-136.ec2.internal" event={"ID":"3de5cf7923bfc1c5b7b74a0700d7b3fa","Type":"ContainerStarted","Data":"cff2925bc3a36c2f6ed0ba95b40b16b4cf5bf4b6d270f52efbc14bef48eef600"} Oct 30 16:19:50.162070 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:50.162043 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-136.ec2.internal" event={"ID":"9ce644f9016009147990b020805ff06d","Type":"ContainerStarted","Data":"622b6489c48a29a9d5eae0856d7d353098c922549fe22891407212db1d417313"} Oct 30 16:19:50.162203 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:19:50.162136 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"kube-system\"" event="&Event{ObjectMeta:{kube-apiserver-proxy-ip-10-0-131-136.ec2.internal.18e35b0d367388f2 kube-system 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:kube-system,Name:kube-apiserver-proxy-ip-10-0-131-136.ec2.internal,UID:3de5cf7923bfc1c5b7b74a0700d7b3fa,APIVersion:v1,ResourceVersion:,FieldPath:spec.containers{haproxy},},Reason:Pulling,Message:Pulling image \"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3a679d80eb36ca7db888d4fbd1fdf16b0997a7c7a107936396b7a57d8b199f4e\",Source:EventSource{Component:kubelet,Host:ip-10-0-131-136.ec2.internal,},FirstTimestamp:2026-10-30 16:19:48.929833202 +0000 UTC m=+1.407441525,LastTimestamp:2026-10-30 16:19:48.929833202 +0000 UTC m=+1.407441525,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-136.ec2.internal,}" Oct 30 16:19:50.201350 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:50.201330 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 16:19:50.202131 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:50.202106 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-136.ec2.internal" event="NodeHasSufficientMemory" Oct 30 16:19:50.202202 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:50.202143 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-136.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 16:19:50.202202 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:50.202155 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-136.ec2.internal" event="NodeHasSufficientPID" Oct 30 16:19:50.233846 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:19:50.233813 2580 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 16:19:50.233922 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:19:50.233883 2580 reflector.go:204] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" Oct 30 16:19:50.533525 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:19:50.533483 2580 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 30 16:19:50.584700 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:19:50.584668 2580 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-136.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="1.6s" Oct 30 16:19:50.633951 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:19:50.633915 2580 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-136.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 16:19:50.794708 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:50.794629 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 16:19:50.795740 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:50.795719 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-136.ec2.internal" event="NodeHasSufficientMemory" Oct 30 16:19:50.795873 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:50.795757 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-136.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 16:19:50.795873 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:50.795772 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-136.ec2.internal" event="NodeHasSufficientPID" Oct 30 16:19:50.795873 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:50.795810 2580 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-131-136.ec2.internal" Oct 30 16:19:50.894915 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:19:50.894878 2580 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-131-136.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-131-136.ec2.internal" Oct 30 16:19:51.379625 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:51.379596 2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-136.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 16:19:51.380123 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:19:51.379740 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-131-136.ec2.internal.18e35b0dc54c9260 openshift-machine-config-operator 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-131-136.ec2.internal,UID:9ce644f9016009147990b020805ff06d,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Pulled,Message:Successfully pulled image \"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d017b7d199b1d54e889bb1eb6e572d8e6d94ad92610894009ee0562c2a5c0f51\" in 1.245s (1.245s including waiting). Image size: 272339155 bytes.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-136.ec2.internal,},FirstTimestamp:2026-10-30 16:19:51.3264216 +0000 UTC m=+2.626709818,LastTimestamp:2026-10-30 16:19:51.3264216 +0000 UTC m=+2.626709818,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-136.ec2.internal,}" Oct 30 16:19:51.569800 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:19:51.569704 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"kube-system\"" event="&Event{ObjectMeta:{kube-apiserver-proxy-ip-10-0-131-136.ec2.internal.18e35b0dd0e77dde kube-system 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:kube-system,Name:kube-apiserver-proxy-ip-10-0-131-136.ec2.internal,UID:3de5cf7923bfc1c5b7b74a0700d7b3fa,APIVersion:v1,ResourceVersion:,FieldPath:spec.containers{haproxy},},Reason:Pulled,Message:Successfully pulled image \"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3a679d80eb36ca7db888d4fbd1fdf16b0997a7c7a107936396b7a57d8b199f4e\" in 1.413s (1.413s including waiting). Image size: 294084393 bytes.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-136.ec2.internal,},FirstTimestamp:2026-10-30 16:19:51.521123806 +0000 UTC m=+2.821412014,LastTimestamp:2026-10-30 16:19:51.521123806 +0000 UTC m=+2.821412014,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-136.ec2.internal,}" Oct 30 16:19:51.616302 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:19:51.616198 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"kube-system\"" event="&Event{ObjectMeta:{kube-apiserver-proxy-ip-10-0-131-136.ec2.internal.18e35b0dd3c5c074 kube-system 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:kube-system,Name:kube-apiserver-proxy-ip-10-0-131-136.ec2.internal,UID:3de5cf7923bfc1c5b7b74a0700d7b3fa,APIVersion:v1,ResourceVersion:,FieldPath:spec.containers{haproxy},},Reason:Created,Message:Container created,Source:EventSource{Component:kubelet,Host:ip-10-0-131-136.ec2.internal,},FirstTimestamp:2026-10-30 16:19:51.569244276 +0000 UTC m=+2.869532477,LastTimestamp:2026-10-30 16:19:51.569244276 +0000 UTC m=+2.869532477,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-136.ec2.internal,}" Oct 30 16:19:51.670724 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:19:51.670658 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-131-136.ec2.internal.18e35b0dd3cfb180 openshift-machine-config-operator 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-131-136.ec2.internal,UID:9ce644f9016009147990b020805ff06d,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Created,Message:Container created,Source:EventSource{Component:kubelet,Host:ip-10-0-131-136.ec2.internal,},FirstTimestamp:2026-10-30 16:19:51.569895808 +0000 UTC m=+2.870184003,LastTimestamp:2026-10-30 16:19:51.569895808 +0000 UTC m=+2.870184003,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-136.ec2.internal,}" Oct 30 16:19:51.720811 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:19:51.720744 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"kube-system\"" event="&Event{ObjectMeta:{kube-apiserver-proxy-ip-10-0-131-136.ec2.internal.18e35b0dd4313844 kube-system 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:kube-system,Name:kube-apiserver-proxy-ip-10-0-131-136.ec2.internal,UID:3de5cf7923bfc1c5b7b74a0700d7b3fa,APIVersion:v1,ResourceVersion:,FieldPath:spec.containers{haproxy},},Reason:Started,Message:Container started,Source:EventSource{Component:kubelet,Host:ip-10-0-131-136.ec2.internal,},FirstTimestamp:2026-10-30 16:19:51.5762873 +0000 UTC m=+2.876575519,LastTimestamp:2026-10-30 16:19:51.5762873 +0000 UTC m=+2.876575519,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-136.ec2.internal,}" Oct 30 16:19:51.771365 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:19:51.771297 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-131-136.ec2.internal.18e35b0dd44f84f6 openshift-machine-config-operator 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-131-136.ec2.internal,UID:9ce644f9016009147990b020805ff06d,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Started,Message:Container started,Source:EventSource{Component:kubelet,Host:ip-10-0-131-136.ec2.internal,},FirstTimestamp:2026-10-30 16:19:51.578273014 +0000 UTC m=+2.878561235,LastTimestamp:2026-10-30 16:19:51.578273014 +0000 UTC m=+2.878561235,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-136.ec2.internal,}" Oct 30 16:19:52.114287 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:52.114219 2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-136.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 16:19:52.168442 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:52.168409 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-136.ec2.internal" event={"ID":"9ce644f9016009147990b020805ff06d","Type":"ContainerStarted","Data":"9bcc072de12fbce6c0c507299ed0e1da4cec7250e7543c41308f4a85176f3179"} Oct 30 16:19:52.168571 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:52.168444 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 16:19:52.169492 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:52.169464 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-136.ec2.internal" event="NodeHasSufficientMemory" Oct 30 16:19:52.169568 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:52.169511 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-136.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 16:19:52.169568 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:52.169523 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-136.ec2.internal" event="NodeHasSufficientPID" Oct 30 16:19:52.169732 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:19:52.169718 2580 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-136.ec2.internal\" not found" node="ip-10-0-131-136.ec2.internal" Oct 30 16:19:52.169814 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:52.169793 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-136.ec2.internal" event={"ID":"3de5cf7923bfc1c5b7b74a0700d7b3fa","Type":"ContainerStarted","Data":"853edd892ee7f9eea56e6e455c769971da6f0c48189481b115f4b81e636a22da"} Oct 30 16:19:52.169864 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:52.169850 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 16:19:52.170836 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:52.170816 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-136.ec2.internal" event="NodeHasSufficientMemory" Oct 30 16:19:52.170918 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:52.170844 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-136.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 16:19:52.170918 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:52.170857 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-136.ec2.internal" event="NodeHasSufficientPID" Oct 30 16:19:52.171046 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:19:52.171021 2580 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-136.ec2.internal\" not found" node="ip-10-0-131-136.ec2.internal" Oct 30 16:19:52.240306 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:19:52.240285 2580 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-136.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="3.2s" Oct 30 16:19:52.452830 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:19:52.452770 2580 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 30 16:19:52.494980 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:52.494963 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 16:19:52.496746 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:52.496719 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-136.ec2.internal" event="NodeHasSufficientMemory" Oct 30 16:19:52.496833 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:52.496752 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-136.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 16:19:52.496833 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:52.496765 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-136.ec2.internal" event="NodeHasSufficientPID" Oct 30 16:19:52.496833 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:52.496791 2580 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-131-136.ec2.internal" Oct 30 16:19:52.584038 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:19:52.584010 2580 reflector.go:204] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" Oct 30 16:19:52.584038 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:19:52.584015 2580 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-131-136.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-131-136.ec2.internal" Oct 30 16:19:52.624232 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:52.624210 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 16:19:52.625070 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:52.625055 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-136.ec2.internal" event="NodeHasSufficientMemory" Oct 30 16:19:52.625134 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:52.625084 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-136.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 16:19:52.625134 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:52.625096 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-136.ec2.internal" event="NodeHasSufficientPID" Oct 30 16:19:53.063167 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:19:53.063138 2580 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 16:19:53.118316 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:53.118287 2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-136.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 16:19:53.171428 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:53.171407 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 16:19:53.171565 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:53.171409 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 16:19:53.172215 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:53.172196 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-136.ec2.internal" event="NodeHasSufficientMemory" Oct 30 16:19:53.172314 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:53.172228 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-136.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 16:19:53.172314 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:53.172241 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-136.ec2.internal" event="NodeHasSufficientPID" Oct 30 16:19:53.172314 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:53.172196 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-136.ec2.internal" event="NodeHasSufficientMemory" Oct 30 16:19:53.172314 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:53.172296 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-136.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 16:19:53.172314 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:53.172306 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-136.ec2.internal" event="NodeHasSufficientPID" Oct 30 16:19:53.172482 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:19:53.172443 2580 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-136.ec2.internal\" not found" node="ip-10-0-131-136.ec2.internal" Oct 30 16:19:53.172620 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:19:53.172602 2580 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-136.ec2.internal\" not found" node="ip-10-0-131-136.ec2.internal" Oct 30 16:19:53.337832 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:19:53.337758 2580 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-136.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 16:19:54.128224 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:54.128143 2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-136.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 16:19:55.116209 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:55.116175 2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-136.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 16:19:55.478630 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:19:55.478541 2580 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-136.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="6.4s" Oct 30 16:19:55.784869 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:55.784785 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 16:19:55.785874 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:55.785857 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-136.ec2.internal" event="NodeHasSufficientMemory" Oct 30 16:19:55.785940 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:55.785887 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-136.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 16:19:55.785940 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:55.785897 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-136.ec2.internal" event="NodeHasSufficientPID" Oct 30 16:19:55.785940 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:55.785928 2580 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-131-136.ec2.internal" Oct 30 16:19:55.890351 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:19:55.890328 2580 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-131-136.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-131-136.ec2.internal" Oct 30 16:19:56.124173 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:56.124095 2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-136.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 16:19:56.978485 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:19:56.978452 2580 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-136.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 16:19:57.109064 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:57.109033 2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-136.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 16:19:57.229717 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:57.229643 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 16:19:57.230503 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:57.230488 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-136.ec2.internal" event="NodeHasSufficientMemory" Oct 30 16:19:57.230566 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:57.230517 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-136.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 16:19:57.230566 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:57.230527 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-136.ec2.internal" event="NodeHasSufficientPID" Oct 30 16:19:57.353830 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:19:57.353803 2580 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 16:19:57.841921 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:19:57.841893 2580 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 30 16:19:57.981885 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:19:57.981853 2580 reflector.go:204] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" Oct 30 16:19:58.111123 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:58.111061 2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-136.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 16:19:59.060757 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:59.060723 2580 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 16:19:59.084664 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:59.084640 2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2wbj7" Oct 30 16:19:59.085701 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:19:59.085674 2580 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-136.ec2.internal\" not found" Oct 30 16:19:59.098083 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:59.098066 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2wbj7" Oct 30 16:19:59.102387 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:59.102370 2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-136.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 16:19:59.966397 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:19:59.966365 2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 16:20:00.065371 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:00.065342 2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: Get "https://ab8e4399e5ec647c7bd954992cc95294-ad1d325e65a4bf33.elb.us-east-2.amazonaws.com:6443/apis/storage.k8s.io/v1/csinodes/ip-10-0-131-136.ec2.internal?resourceVersion=0": http2: client conn could not be established Oct 30 16:20:00.103038 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:00.103005 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 16:14:59 +0000 UTC" deadline="2028-06-02 03:53:23.642450799 +0000 UTC" Oct 30 16:20:00.103038 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:00.103033 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13931h33m23.53942062s" Oct 30 16:20:01.126373 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:01.126341 2580 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-131-136.ec2.internal" not found Oct 30 16:20:01.151729 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:01.151707 2580 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-131-136.ec2.internal" not found Oct 30 16:20:01.223276 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:01.223250 2580 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-131-136.ec2.internal" not found Oct 30 16:20:01.491469 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:01.491406 2580 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-131-136.ec2.internal" not found Oct 30 16:20:01.491469 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:01.491432 2580 csi_plugin.go:399] Failed to initialize CSINode: error updating CSINode annotation: timed out waiting for the condition; caused by: nodes "ip-10-0-131-136.ec2.internal" not found Oct 30 16:20:01.539325 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:01.539302 2580 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-131-136.ec2.internal" not found Oct 30 16:20:01.564842 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:01.564822 2580 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-131-136.ec2.internal" not found Oct 30 16:20:01.631493 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:01.631470 2580 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-131-136.ec2.internal" not found Oct 30 16:20:01.903479 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:01.903413 2580 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-131-136.ec2.internal" not found Oct 30 16:20:01.903479 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:01.903442 2580 csi_plugin.go:399] Failed to initialize CSINode: error updating CSINode annotation: timed out waiting for the condition; caused by: nodes "ip-10-0-131-136.ec2.internal" not found Oct 30 16:20:01.907395 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:01.907378 2580 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-131-136.ec2.internal\" not found" node="ip-10-0-131-136.ec2.internal" Oct 30 16:20:02.175978 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:02.175911 2580 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-131-136.ec2.internal" not found Oct 30 16:20:02.202225 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:02.202202 2580 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-131-136.ec2.internal" not found Oct 30 16:20:02.268751 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:02.268729 2580 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-131-136.ec2.internal" not found Oct 30 16:20:02.290860 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:02.290839 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 16:20:02.291827 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:02.291809 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-136.ec2.internal" event="NodeHasSufficientMemory" Oct 30 16:20:02.291919 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:02.291839 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-136.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 16:20:02.291919 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:02.291848 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-136.ec2.internal" event="NodeHasSufficientPID" Oct 30 16:20:02.291919 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:02.291877 2580 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-131-136.ec2.internal" Oct 30 16:20:02.312736 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:02.312717 2580 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-131-136.ec2.internal" Oct 30 16:20:02.312789 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:02.312754 2580 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-136.ec2.internal\": node \"ip-10-0-131-136.ec2.internal\" not found" Oct 30 16:20:02.346940 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:02.346912 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-136.ec2.internal\" not found" Oct 30 16:20:02.447933 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:02.447859 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-136.ec2.internal\" not found" Oct 30 16:20:02.548791 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:02.548770 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-136.ec2.internal\" not found" Oct 30 16:20:02.649710 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:02.649683 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-136.ec2.internal\" not found" Oct 30 16:20:02.750207 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:02.750125 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-136.ec2.internal\" not found" Oct 30 16:20:02.851177 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:02.851146 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-136.ec2.internal\" not found" Oct 30 16:20:02.951947 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:02.951907 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-136.ec2.internal\" not found" Oct 30 16:20:03.053035 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:03.052960 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-136.ec2.internal\" not found" Oct 30 16:20:03.153511 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:03.153476 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-136.ec2.internal\" not found" Oct 30 16:20:03.186201 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:03.186179 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-131-136.ec2.internal_9ce644f9016009147990b020805ff06d/setup/0.log" Oct 30 16:20:03.186509 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:03.186215 2580 generic.go:356] "Generic (PLEG): container finished" podID="9ce644f9016009147990b020805ff06d" containerID="9bcc072de12fbce6c0c507299ed0e1da4cec7250e7543c41308f4a85176f3179" exitCode=1 Oct 30 16:20:03.186509 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:03.186242 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-136.ec2.internal" event={"ID":"9ce644f9016009147990b020805ff06d","Type":"ContainerDied","Data":"9bcc072de12fbce6c0c507299ed0e1da4cec7250e7543c41308f4a85176f3179"} Oct 30 16:20:03.186509 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:03.186379 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 16:20:03.191748 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:03.191731 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-136.ec2.internal" event="NodeHasSufficientMemory" Oct 30 16:20:03.191824 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:03.191759 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-136.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 16:20:03.191824 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:03.191769 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-136.ec2.internal" event="NodeHasSufficientPID" Oct 30 16:20:03.191948 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:03.191936 2580 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-136.ec2.internal\" not found" node="ip-10-0-131-136.ec2.internal" Oct 30 16:20:03.254253 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:03.254107 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-136.ec2.internal\" not found" Oct 30 16:20:03.354807 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:03.354704 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-136.ec2.internal\" not found" Oct 30 16:20:03.455201 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:03.455168 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-136.ec2.internal\" not found" Oct 30 16:20:03.555478 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:03.555446 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-136.ec2.internal\" not found" Oct 30 16:20:03.655939 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:03.655875 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-136.ec2.internal\" not found" Oct 30 16:20:03.756287 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:03.756271 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-136.ec2.internal\" not found" Oct 30 16:20:03.856842 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:03.856821 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-136.ec2.internal\" not found" Oct 30 16:20:03.957866 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:03.957828 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-136.ec2.internal\" not found" Oct 30 16:20:04.058866 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:04.058830 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-136.ec2.internal\" not found" Oct 30 16:20:04.159032 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:04.159002 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-136.ec2.internal\" not found" Oct 30 16:20:04.189548 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:04.189529 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-131-136.ec2.internal_9ce644f9016009147990b020805ff06d/setup/0.log" Oct 30 16:20:04.189894 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:04.189568 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-136.ec2.internal" event={"ID":"9ce644f9016009147990b020805ff06d","Type":"ContainerStarted","Data":"e62b2d0f673552954712774014cc22bed99f8d3f3e61d4488511b83f0aeffd54"} Oct 30 16:20:04.189894 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:04.189691 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 16:20:04.190553 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:04.190539 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-136.ec2.internal" event="NodeHasSufficientMemory" Oct 30 16:20:04.190626 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:04.190564 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-136.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 16:20:04.190626 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:04.190577 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-136.ec2.internal" event="NodeHasSufficientPID" Oct 30 16:20:04.190816 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:04.190802 2580 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-136.ec2.internal\" not found" node="ip-10-0-131-136.ec2.internal" Oct 30 16:20:04.259248 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:04.259203 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-136.ec2.internal\" not found" Oct 30 16:20:04.360163 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:04.360138 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-136.ec2.internal\" not found" Oct 30 16:20:04.460735 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:04.460708 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-136.ec2.internal\" not found" Oct 30 16:20:04.561379 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:04.561318 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-136.ec2.internal\" not found" Oct 30 16:20:04.662261 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:04.662214 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-136.ec2.internal\" not found" Oct 30 16:20:04.763167 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:04.763142 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-136.ec2.internal\" not found" Oct 30 16:20:04.863721 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:04.863670 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-136.ec2.internal\" not found" Oct 30 16:20:04.964103 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:04.964067 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-136.ec2.internal\" not found" Oct 30 16:20:05.065088 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:05.065055 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-136.ec2.internal\" not found" Oct 30 16:20:05.165416 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:05.165317 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-136.ec2.internal\" not found" Oct 30 16:20:05.266264 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:05.266216 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-136.ec2.internal\" not found" Oct 30 16:20:05.367207 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:05.367167 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-136.ec2.internal\" not found" Oct 30 16:20:05.467751 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:05.467680 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-136.ec2.internal\" not found" Oct 30 16:20:05.568412 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:05.568369 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-136.ec2.internal\" not found" Oct 30 16:20:05.668877 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:05.668834 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-136.ec2.internal\" not found" Oct 30 16:20:05.769303 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:05.769242 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-136.ec2.internal\" not found" Oct 30 16:20:05.869814 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:05.869777 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-136.ec2.internal\" not found" Oct 30 16:20:05.917510 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:05.917491 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 16:20:05.918615 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:05.918594 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-136.ec2.internal" event="NodeHasSufficientMemory" Oct 30 16:20:05.918708 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:05.918626 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-136.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 16:20:05.918708 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:05.918640 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-136.ec2.internal" event="NodeHasSufficientPID" Oct 30 16:20:05.969900 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:05.969881 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-136.ec2.internal\" not found" Oct 30 16:20:06.070255 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:06.070206 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-136.ec2.internal\" not found" Oct 30 16:20:06.109012 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:06.108992 2580 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 16:20:06.170507 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:06.170487 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-136.ec2.internal\" not found" Oct 30 16:20:06.270879 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:06.270845 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-136.ec2.internal\" not found" Oct 30 16:20:06.338781 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:06.338714 2580 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 16:20:06.427931 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:06.427884 2580 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-136.ec2.internal" Oct 30 16:20:06.454712 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:06.454693 2580 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 16:20:06.456517 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:06.456504 2580 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-136.ec2.internal" Oct 30 16:20:06.478386 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:06.478364 2580 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 16:20:06.646406 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:06.646332 2580 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 16:20:07.012001 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.011921 2580 apiserver.go:52] "Watching apiserver" Oct 30 16:20:07.033342 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.033323 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 16:20:07.052390 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.052368 2580 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 16:20:07.052681 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.052664 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-c8x4k","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-136.ec2.internal","openshift-multus/multus-7f2mt","openshift-multus/network-metrics-daemon-xj6cp","openshift-ovn-kubernetes/ovnkube-node-2krz6","kube-system/global-pull-secret-syncer-ncfzj","kube-system/kube-apiserver-proxy-ip-10-0-131-136.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g69ll","openshift-dns/node-resolver-lghnm","openshift-image-registry/node-ca-2l5ht","openshift-multus/multus-additional-cni-plugins-x9ddt","openshift-network-diagnostics/network-check-target-b4ljl","kube-system/konnectivity-agent-sg9tg","openshift-network-operator/iptables-alerter-d9kr8"] Oct 30 16:20:07.058246 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.058232 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2l5ht" Oct 30 16:20:07.062139 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.062120 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-sg9tg" Oct 30 16:20:07.065440 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.065421 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7f2mt" Oct 30 16:20:07.068499 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.068484 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xj6cp" Oct 30 16:20:07.068605 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:07.068538 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xj6cp" podUID="6f3d36d6-5098-43cc-8f8a-bd83d5dea5fc" Oct 30 16:20:07.072054 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.072037 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:07.073408 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.073388 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-fpzw7\"" Oct 30 16:20:07.073594 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.073568 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 16:20:07.073653 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.073604 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 16:20:07.073653 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.073590 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 16:20:07.075028 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.075004 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ncfzj" Oct 30 16:20:07.075117 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:07.075094 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ncfzj" podUID="029f4ecb-eb6b-4693-9e9e-ca61f554dd33" Oct 30 16:20:07.076648 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.076629 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 16:20:07.076648 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.076629 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-hxhjs\"" Oct 30 16:20:07.076807 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.076723 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 16:20:07.077668 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.077644 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-lghnm" Oct 30 16:20:07.080823 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.080800 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 16:20:07.080823 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.080810 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-5kvnn\"" Oct 30 16:20:07.080985 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.080904 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-d9kr8" Oct 30 16:20:07.081425 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.081278 2580 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 16:20:07.081425 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.081336 2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-6lc6s" Oct 30 16:20:07.083650 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.083635 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g69ll" Oct 30 16:20:07.087004 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.086986 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-c8x4k" Oct 30 16:20:07.089911 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.089891 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-x9ddt" Oct 30 16:20:07.092482 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.092464 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b4ljl" Oct 30 16:20:07.092576 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:07.092538 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b4ljl" podUID="2d1ed7d8-37cc-4652-b225-3a715b755e48" Oct 30 16:20:07.093285 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.093271 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-6lc6s" Oct 30 16:20:07.128481 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.128460 2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 16:20:07.175794 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.175771 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 16:20:07.175910 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.175805 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 16:20:07.175910 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.175857 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 16:20:07.176032 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.176016 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 16:20:07.176084 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.176046 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 16:20:07.176084 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.176048 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 16:20:07.176220 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.176209 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 16:20:07.176253 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.176210 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 16:20:07.176304 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.176293 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-lngbh\"" Oct 30 16:20:07.176347 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.176298 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 16:20:07.176347 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.176331 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 16:20:07.176440 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.176349 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 16:20:07.176440 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.176354 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 16:20:07.176440 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.176380 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 16:20:07.176440 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.176390 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 16:20:07.176440 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.176397 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 16:20:07.176440 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.176384 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 16:20:07.176440 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.176417 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-l28xz\"" Oct 30 16:20:07.176440 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.176418 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-gfrzf\"" Oct 30 16:20:07.176440 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.176408 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 16:20:07.176732 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.176434 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 16:20:07.176732 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.176468 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-n7jsz\"" Oct 30 16:20:07.176732 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.176447 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 16:20:07.176732 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.176483 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 16:20:07.187821 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.187803 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-sg8tz\"" Oct 30 16:20:07.188403 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.188392 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 16:20:07.188447 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.188428 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-jrplp\"" Oct 30 16:20:07.222174 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.222145 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a6b95002-74f1-475d-a64a-6495733cf142-registration-dir\") pod \"aws-ebs-csi-driver-node-g69ll\" (UID: \"a6b95002-74f1-475d-a64a-6495733cf142\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g69ll" Oct 30 16:20:07.222174 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.222176 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/749b2b57-f20e-4cdb-8d89-d2bac352f751-run\") pod \"tuned-c8x4k\" (UID: \"749b2b57-f20e-4cdb-8d89-d2bac352f751\") " pod="openshift-cluster-node-tuning-operator/tuned-c8x4k" Oct 30 16:20:07.222285 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.222192 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/749b2b57-f20e-4cdb-8d89-d2bac352f751-tmp\") pod \"tuned-c8x4k\" (UID: \"749b2b57-f20e-4cdb-8d89-d2bac352f751\") " pod="openshift-cluster-node-tuning-operator/tuned-c8x4k" Oct 30 16:20:07.222285 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.222207 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f2de1ce0-1c1e-4276-80a9-66239e13e352-run-ovn\") pod \"ovnkube-node-2krz6\" (UID: \"f2de1ce0-1c1e-4276-80a9-66239e13e352\") " pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:07.222285 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.222255 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a6b95002-74f1-475d-a64a-6495733cf142-etc-selinux\") pod \"aws-ebs-csi-driver-node-g69ll\" (UID: \"a6b95002-74f1-475d-a64a-6495733cf142\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g69ll" Oct 30 16:20:07.222440 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.222295 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/33b888e2-d349-439b-9aa1-5b519e0e08f7-system-cni-dir\") pod \"multus-7f2mt\" (UID: \"33b888e2-d349-439b-9aa1-5b519e0e08f7\") " pod="openshift-multus/multus-7f2mt" Oct 30 16:20:07.222440 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.222315 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ec1217e-8623-42a6-a052-adee153936a2-tmp-dir\") pod \"node-resolver-lghnm\" (UID: \"9ec1217e-8623-42a6-a052-adee153936a2\") " pod="openshift-dns/node-resolver-lghnm" Oct 30 16:20:07.222440 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.222403 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrppt\" (UniqueName: \"kubernetes.io/projected/9ec1217e-8623-42a6-a052-adee153936a2-kube-api-access-nrppt\") pod \"node-resolver-lghnm\" (UID: \"9ec1217e-8623-42a6-a052-adee153936a2\") " pod="openshift-dns/node-resolver-lghnm" Oct 30 16:20:07.222440 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.222420 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/749b2b57-f20e-4cdb-8d89-d2bac352f751-etc-sysctl-d\") pod \"tuned-c8x4k\" (UID: \"749b2b57-f20e-4cdb-8d89-d2bac352f751\") " pod="openshift-cluster-node-tuning-operator/tuned-c8x4k" Oct 30 16:20:07.222440 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.222434 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/749b2b57-f20e-4cdb-8d89-d2bac352f751-host\") pod \"tuned-c8x4k\" (UID: \"749b2b57-f20e-4cdb-8d89-d2bac352f751\") " pod="openshift-cluster-node-tuning-operator/tuned-c8x4k" Oct 30 16:20:07.222688 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.222451 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/33b888e2-d349-439b-9aa1-5b519e0e08f7-host-var-lib-cni-bin\") pod \"multus-7f2mt\" (UID: \"33b888e2-d349-439b-9aa1-5b519e0e08f7\") " pod="openshift-multus/multus-7f2mt" Oct 30 16:20:07.222688 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.222466 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/33b888e2-d349-439b-9aa1-5b519e0e08f7-hostroot\") pod \"multus-7f2mt\" (UID: \"33b888e2-d349-439b-9aa1-5b519e0e08f7\") " pod="openshift-multus/multus-7f2mt" Oct 30 16:20:07.222688 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.222516 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9ea50669-2c08-403d-8d08-f9ffffa5b896-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-x9ddt\" (UID: \"9ea50669-2c08-403d-8d08-f9ffffa5b896\") " pod="openshift-multus/multus-additional-cni-plugins-x9ddt" Oct 30 16:20:07.222688 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.222552 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f2de1ce0-1c1e-4276-80a9-66239e13e352-run-openvswitch\") pod \"ovnkube-node-2krz6\" (UID: \"f2de1ce0-1c1e-4276-80a9-66239e13e352\") " pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:07.222688 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.222594 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f2de1ce0-1c1e-4276-80a9-66239e13e352-host-cni-netd\") pod \"ovnkube-node-2krz6\" (UID: \"f2de1ce0-1c1e-4276-80a9-66239e13e352\") " pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:07.222688 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.222638 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/029f4ecb-eb6b-4693-9e9e-ca61f554dd33-kubelet-config\") pod \"global-pull-secret-syncer-ncfzj\" (UID: \"029f4ecb-eb6b-4693-9e9e-ca61f554dd33\") " pod="kube-system/global-pull-secret-syncer-ncfzj" Oct 30 16:20:07.222688 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.222676 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a6b95002-74f1-475d-a64a-6495733cf142-kubelet-dir\") pod \"aws-ebs-csi-driver-node-g69ll\" (UID: \"a6b95002-74f1-475d-a64a-6495733cf142\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g69ll" Oct 30 16:20:07.222983 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.222700 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a6b95002-74f1-475d-a64a-6495733cf142-socket-dir\") pod \"aws-ebs-csi-driver-node-g69ll\" (UID: \"a6b95002-74f1-475d-a64a-6495733cf142\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g69ll" Oct 30 16:20:07.222983 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.222742 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/33b888e2-d349-439b-9aa1-5b519e0e08f7-multus-cni-dir\") pod \"multus-7f2mt\" (UID: \"33b888e2-d349-439b-9aa1-5b519e0e08f7\") " pod="openshift-multus/multus-7f2mt" Oct 30 16:20:07.222983 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.222773 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/33b888e2-d349-439b-9aa1-5b519e0e08f7-multus-conf-dir\") pod \"multus-7f2mt\" (UID: \"33b888e2-d349-439b-9aa1-5b519e0e08f7\") " pod="openshift-multus/multus-7f2mt" Oct 30 16:20:07.222983 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.222799 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f2de1ce0-1c1e-4276-80a9-66239e13e352-node-log\") pod \"ovnkube-node-2krz6\" (UID: \"f2de1ce0-1c1e-4276-80a9-66239e13e352\") " pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:07.222983 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.222839 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzp5d\" (UniqueName: \"kubernetes.io/projected/25c51397-40b9-432c-949d-12ba9f36532a-kube-api-access-qzp5d\") pod \"iptables-alerter-d9kr8\" (UID: \"25c51397-40b9-432c-949d-12ba9f36532a\") " pod="openshift-network-operator/iptables-alerter-d9kr8" Oct 30 16:20:07.222983 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.222878 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/33b888e2-d349-439b-9aa1-5b519e0e08f7-cnibin\") pod \"multus-7f2mt\" (UID: \"33b888e2-d349-439b-9aa1-5b519e0e08f7\") " pod="openshift-multus/multus-7f2mt" Oct 30 16:20:07.222983 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.222926 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/749b2b57-f20e-4cdb-8d89-d2bac352f751-etc-sysctl-conf\") pod \"tuned-c8x4k\" (UID: \"749b2b57-f20e-4cdb-8d89-d2bac352f751\") " pod="openshift-cluster-node-tuning-operator/tuned-c8x4k" Oct 30 16:20:07.222983 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.222952 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9ea50669-2c08-403d-8d08-f9ffffa5b896-system-cni-dir\") pod \"multus-additional-cni-plugins-x9ddt\" (UID: \"9ea50669-2c08-403d-8d08-f9ffffa5b896\") " pod="openshift-multus/multus-additional-cni-plugins-x9ddt" Oct 30 16:20:07.222983 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.222971 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f2de1ce0-1c1e-4276-80a9-66239e13e352-ovnkube-config\") pod \"ovnkube-node-2krz6\" (UID: \"f2de1ce0-1c1e-4276-80a9-66239e13e352\") " pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:07.222983 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.222985 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/749b2b57-f20e-4cdb-8d89-d2bac352f751-etc-sysconfig\") pod \"tuned-c8x4k\" (UID: \"749b2b57-f20e-4cdb-8d89-d2bac352f751\") " pod="openshift-cluster-node-tuning-operator/tuned-c8x4k" Oct 30 16:20:07.223330 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.223034 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9ea50669-2c08-403d-8d08-f9ffffa5b896-tuning-conf-dir\") pod \"multus-additional-cni-plugins-x9ddt\" (UID: \"9ea50669-2c08-403d-8d08-f9ffffa5b896\") " pod="openshift-multus/multus-additional-cni-plugins-x9ddt" Oct 30 16:20:07.223330 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.223052 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f2de1ce0-1c1e-4276-80a9-66239e13e352-host-run-netns\") pod \"ovnkube-node-2krz6\" (UID: \"f2de1ce0-1c1e-4276-80a9-66239e13e352\") " pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:07.223330 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.223066 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f3d36d6-5098-43cc-8f8a-bd83d5dea5fc-metrics-certs\") pod \"network-metrics-daemon-xj6cp\" (UID: \"6f3d36d6-5098-43cc-8f8a-bd83d5dea5fc\") " pod="openshift-multus/network-metrics-daemon-xj6cp" Oct 30 16:20:07.223330 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.223087 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f2de1ce0-1c1e-4276-80a9-66239e13e352-env-overrides\") pod \"ovnkube-node-2krz6\" (UID: \"f2de1ce0-1c1e-4276-80a9-66239e13e352\") " pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:07.223330 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.223109 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7d4j4\" (UniqueName: \"kubernetes.io/projected/f2de1ce0-1c1e-4276-80a9-66239e13e352-kube-api-access-7d4j4\") pod \"ovnkube-node-2krz6\" (UID: \"f2de1ce0-1c1e-4276-80a9-66239e13e352\") " pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:07.223330 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.223130 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/25c51397-40b9-432c-949d-12ba9f36532a-iptables-alerter-script\") pod \"iptables-alerter-d9kr8\" (UID: \"25c51397-40b9-432c-949d-12ba9f36532a\") " pod="openshift-network-operator/iptables-alerter-d9kr8" Oct 30 16:20:07.223330 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.223158 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zlg7\" (UniqueName: \"kubernetes.io/projected/a6b95002-74f1-475d-a64a-6495733cf142-kube-api-access-8zlg7\") pod \"aws-ebs-csi-driver-node-g69ll\" (UID: \"a6b95002-74f1-475d-a64a-6495733cf142\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g69ll" Oct 30 16:20:07.223330 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.223173 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/33b888e2-d349-439b-9aa1-5b519e0e08f7-host-run-multus-certs\") pod \"multus-7f2mt\" (UID: \"33b888e2-d349-439b-9aa1-5b519e0e08f7\") " pod="openshift-multus/multus-7f2mt" Oct 30 16:20:07.223330 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.223200 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/749b2b57-f20e-4cdb-8d89-d2bac352f751-sys\") pod \"tuned-c8x4k\" (UID: \"749b2b57-f20e-4cdb-8d89-d2bac352f751\") " pod="openshift-cluster-node-tuning-operator/tuned-c8x4k" Oct 30 16:20:07.223330 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.223229 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f2de1ce0-1c1e-4276-80a9-66239e13e352-ovnkube-script-lib\") pod \"ovnkube-node-2krz6\" (UID: \"f2de1ce0-1c1e-4276-80a9-66239e13e352\") " pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:07.223330 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.223253 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/33b888e2-d349-439b-9aa1-5b519e0e08f7-cni-binary-copy\") pod \"multus-7f2mt\" (UID: \"33b888e2-d349-439b-9aa1-5b519e0e08f7\") " pod="openshift-multus/multus-7f2mt" Oct 30 16:20:07.223330 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.223277 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/749b2b57-f20e-4cdb-8d89-d2bac352f751-etc-kubernetes\") pod \"tuned-c8x4k\" (UID: \"749b2b57-f20e-4cdb-8d89-d2bac352f751\") " pod="openshift-cluster-node-tuning-operator/tuned-c8x4k" Oct 30 16:20:07.223330 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.223302 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljv94\" (UniqueName: \"kubernetes.io/projected/2d1ed7d8-37cc-4652-b225-3a715b755e48-kube-api-access-ljv94\") pod \"network-check-target-b4ljl\" (UID: \"2d1ed7d8-37cc-4652-b225-3a715b755e48\") " pod="openshift-network-diagnostics/network-check-target-b4ljl" Oct 30 16:20:07.223330 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.223328 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f2de1ce0-1c1e-4276-80a9-66239e13e352-run-systemd\") pod \"ovnkube-node-2krz6\" (UID: \"f2de1ce0-1c1e-4276-80a9-66239e13e352\") " pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:07.223875 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.223352 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/33b888e2-d349-439b-9aa1-5b519e0e08f7-multus-socket-dir-parent\") pod \"multus-7f2mt\" (UID: \"33b888e2-d349-439b-9aa1-5b519e0e08f7\") " pod="openshift-multus/multus-7f2mt" Oct 30 16:20:07.223875 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.223373 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/33b888e2-d349-439b-9aa1-5b519e0e08f7-multus-daemon-config\") pod \"multus-7f2mt\" (UID: \"33b888e2-d349-439b-9aa1-5b519e0e08f7\") " pod="openshift-multus/multus-7f2mt" Oct 30 16:20:07.223875 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.223418 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/33b888e2-d349-439b-9aa1-5b519e0e08f7-etc-kubernetes\") pod \"multus-7f2mt\" (UID: \"33b888e2-d349-439b-9aa1-5b519e0e08f7\") " pod="openshift-multus/multus-7f2mt" Oct 30 16:20:07.223875 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.223463 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9ec1217e-8623-42a6-a052-adee153936a2-hosts-file\") pod \"node-resolver-lghnm\" (UID: \"9ec1217e-8623-42a6-a052-adee153936a2\") " pod="openshift-dns/node-resolver-lghnm" Oct 30 16:20:07.223875 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.223493 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d10078a0-c21c-4f18-9c5a-4915fb7c8af3-serviceca\") pod \"node-ca-2l5ht\" (UID: \"d10078a0-c21c-4f18-9c5a-4915fb7c8af3\") " pod="openshift-image-registry/node-ca-2l5ht" Oct 30 16:20:07.223875 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.223517 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f2de1ce0-1c1e-4276-80a9-66239e13e352-host-kubelet\") pod \"ovnkube-node-2krz6\" (UID: \"f2de1ce0-1c1e-4276-80a9-66239e13e352\") " pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:07.223875 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.223542 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f2de1ce0-1c1e-4276-80a9-66239e13e352-host-run-ovn-kubernetes\") pod \"ovnkube-node-2krz6\" (UID: \"f2de1ce0-1c1e-4276-80a9-66239e13e352\") " pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:07.223875 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.223573 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/749b2b57-f20e-4cdb-8d89-d2bac352f751-etc-systemd\") pod \"tuned-c8x4k\" (UID: \"749b2b57-f20e-4cdb-8d89-d2bac352f751\") " pod="openshift-cluster-node-tuning-operator/tuned-c8x4k" Oct 30 16:20:07.223875 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.223609 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9ea50669-2c08-403d-8d08-f9ffffa5b896-cni-binary-copy\") pod \"multus-additional-cni-plugins-x9ddt\" (UID: \"9ea50669-2c08-403d-8d08-f9ffffa5b896\") " pod="openshift-multus/multus-additional-cni-plugins-x9ddt" Oct 30 16:20:07.223875 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.223625 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5tsm\" (UniqueName: \"kubernetes.io/projected/6f3d36d6-5098-43cc-8f8a-bd83d5dea5fc-kube-api-access-v5tsm\") pod \"network-metrics-daemon-xj6cp\" (UID: \"6f3d36d6-5098-43cc-8f8a-bd83d5dea5fc\") " pod="openshift-multus/network-metrics-daemon-xj6cp" Oct 30 16:20:07.223875 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.223641 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/029f4ecb-eb6b-4693-9e9e-ca61f554dd33-original-pull-secret\") pod \"global-pull-secret-syncer-ncfzj\" (UID: \"029f4ecb-eb6b-4693-9e9e-ca61f554dd33\") " pod="kube-system/global-pull-secret-syncer-ncfzj" Oct 30 16:20:07.223875 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.223656 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a6b95002-74f1-475d-a64a-6495733cf142-sys-fs\") pod \"aws-ebs-csi-driver-node-g69ll\" (UID: \"a6b95002-74f1-475d-a64a-6495733cf142\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g69ll" Oct 30 16:20:07.223875 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.223669 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5sxw\" (UniqueName: \"kubernetes.io/projected/33b888e2-d349-439b-9aa1-5b519e0e08f7-kube-api-access-v5sxw\") pod \"multus-7f2mt\" (UID: \"33b888e2-d349-439b-9aa1-5b519e0e08f7\") " pod="openshift-multus/multus-7f2mt" Oct 30 16:20:07.223875 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.223714 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfgdd\" (UniqueName: \"kubernetes.io/projected/749b2b57-f20e-4cdb-8d89-d2bac352f751-kube-api-access-dfgdd\") pod \"tuned-c8x4k\" (UID: \"749b2b57-f20e-4cdb-8d89-d2bac352f751\") " pod="openshift-cluster-node-tuning-operator/tuned-c8x4k" Oct 30 16:20:07.223875 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.223754 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9ea50669-2c08-403d-8d08-f9ffffa5b896-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-x9ddt\" (UID: \"9ea50669-2c08-403d-8d08-f9ffffa5b896\") " pod="openshift-multus/multus-additional-cni-plugins-x9ddt" Oct 30 16:20:07.223875 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.223793 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpgwx\" (UniqueName: \"kubernetes.io/projected/9ea50669-2c08-403d-8d08-f9ffffa5b896-kube-api-access-zpgwx\") pod \"multus-additional-cni-plugins-x9ddt\" (UID: \"9ea50669-2c08-403d-8d08-f9ffffa5b896\") " pod="openshift-multus/multus-additional-cni-plugins-x9ddt" Oct 30 16:20:07.224399 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.223814 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f2de1ce0-1c1e-4276-80a9-66239e13e352-systemd-units\") pod \"ovnkube-node-2krz6\" (UID: \"f2de1ce0-1c1e-4276-80a9-66239e13e352\") " pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:07.224399 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.223828 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f2de1ce0-1c1e-4276-80a9-66239e13e352-log-socket\") pod \"ovnkube-node-2krz6\" (UID: \"f2de1ce0-1c1e-4276-80a9-66239e13e352\") " pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:07.224399 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.223843 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f2de1ce0-1c1e-4276-80a9-66239e13e352-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2krz6\" (UID: \"f2de1ce0-1c1e-4276-80a9-66239e13e352\") " pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:07.224399 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.223878 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f2de1ce0-1c1e-4276-80a9-66239e13e352-ovn-node-metrics-cert\") pod \"ovnkube-node-2krz6\" (UID: \"f2de1ce0-1c1e-4276-80a9-66239e13e352\") " pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:07.224399 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.223904 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/33b888e2-d349-439b-9aa1-5b519e0e08f7-host-var-lib-cni-multus\") pod \"multus-7f2mt\" (UID: \"33b888e2-d349-439b-9aa1-5b519e0e08f7\") " pod="openshift-multus/multus-7f2mt" Oct 30 16:20:07.224399 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.223931 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f2de1ce0-1c1e-4276-80a9-66239e13e352-host-slash\") pod \"ovnkube-node-2krz6\" (UID: \"f2de1ce0-1c1e-4276-80a9-66239e13e352\") " pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:07.224399 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.223957 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/33b888e2-d349-439b-9aa1-5b519e0e08f7-host-run-k8s-cni-cncf-io\") pod \"multus-7f2mt\" (UID: \"33b888e2-d349-439b-9aa1-5b519e0e08f7\") " pod="openshift-multus/multus-7f2mt" Oct 30 16:20:07.224399 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.223984 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/749b2b57-f20e-4cdb-8d89-d2bac352f751-lib-modules\") pod \"tuned-c8x4k\" (UID: \"749b2b57-f20e-4cdb-8d89-d2bac352f751\") " pod="openshift-cluster-node-tuning-operator/tuned-c8x4k" Oct 30 16:20:07.224399 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.224005 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f2de1ce0-1c1e-4276-80a9-66239e13e352-host-cni-bin\") pod \"ovnkube-node-2krz6\" (UID: \"f2de1ce0-1c1e-4276-80a9-66239e13e352\") " pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:07.224399 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.224019 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25jwd\" (UniqueName: \"kubernetes.io/projected/d10078a0-c21c-4f18-9c5a-4915fb7c8af3-kube-api-access-25jwd\") pod \"node-ca-2l5ht\" (UID: \"d10078a0-c21c-4f18-9c5a-4915fb7c8af3\") " pod="openshift-image-registry/node-ca-2l5ht" Oct 30 16:20:07.224399 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.224033 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4c515c55-0717-4974-9f1a-68e0a25984cd-agent-certs\") pod \"konnectivity-agent-sg9tg\" (UID: \"4c515c55-0717-4974-9f1a-68e0a25984cd\") " pod="kube-system/konnectivity-agent-sg9tg" Oct 30 16:20:07.224399 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.224077 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4c515c55-0717-4974-9f1a-68e0a25984cd-konnectivity-ca\") pod \"konnectivity-agent-sg9tg\" (UID: \"4c515c55-0717-4974-9f1a-68e0a25984cd\") " pod="kube-system/konnectivity-agent-sg9tg" Oct 30 16:20:07.224399 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.224101 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/029f4ecb-eb6b-4693-9e9e-ca61f554dd33-dbus\") pod \"global-pull-secret-syncer-ncfzj\" (UID: \"029f4ecb-eb6b-4693-9e9e-ca61f554dd33\") " pod="kube-system/global-pull-secret-syncer-ncfzj" Oct 30 16:20:07.224399 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.224130 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/33b888e2-d349-439b-9aa1-5b519e0e08f7-host-var-lib-kubelet\") pod \"multus-7f2mt\" (UID: \"33b888e2-d349-439b-9aa1-5b519e0e08f7\") " pod="openshift-multus/multus-7f2mt" Oct 30 16:20:07.224399 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.224154 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/749b2b57-f20e-4cdb-8d89-d2bac352f751-etc-modprobe-d\") pod \"tuned-c8x4k\" (UID: \"749b2b57-f20e-4cdb-8d89-d2bac352f751\") " pod="openshift-cluster-node-tuning-operator/tuned-c8x4k" Oct 30 16:20:07.224399 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.224188 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9ea50669-2c08-403d-8d08-f9ffffa5b896-cnibin\") pod \"multus-additional-cni-plugins-x9ddt\" (UID: \"9ea50669-2c08-403d-8d08-f9ffffa5b896\") " pod="openshift-multus/multus-additional-cni-plugins-x9ddt" Oct 30 16:20:07.224828 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.224214 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f2de1ce0-1c1e-4276-80a9-66239e13e352-var-lib-openvswitch\") pod \"ovnkube-node-2krz6\" (UID: \"f2de1ce0-1c1e-4276-80a9-66239e13e352\") " pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:07.224828 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.224231 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/25c51397-40b9-432c-949d-12ba9f36532a-host-slash\") pod \"iptables-alerter-d9kr8\" (UID: \"25c51397-40b9-432c-949d-12ba9f36532a\") " pod="openshift-network-operator/iptables-alerter-d9kr8" Oct 30 16:20:07.224828 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.224299 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a6b95002-74f1-475d-a64a-6495733cf142-device-dir\") pod \"aws-ebs-csi-driver-node-g69ll\" (UID: \"a6b95002-74f1-475d-a64a-6495733cf142\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g69ll" Oct 30 16:20:07.224828 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.224324 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/33b888e2-d349-439b-9aa1-5b519e0e08f7-host-run-netns\") pod \"multus-7f2mt\" (UID: \"33b888e2-d349-439b-9aa1-5b519e0e08f7\") " pod="openshift-multus/multus-7f2mt" Oct 30 16:20:07.224828 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.224365 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/749b2b57-f20e-4cdb-8d89-d2bac352f751-var-lib-kubelet\") pod \"tuned-c8x4k\" (UID: \"749b2b57-f20e-4cdb-8d89-d2bac352f751\") " pod="openshift-cluster-node-tuning-operator/tuned-c8x4k" Oct 30 16:20:07.224828 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.224382 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/749b2b57-f20e-4cdb-8d89-d2bac352f751-etc-tuned\") pod \"tuned-c8x4k\" (UID: \"749b2b57-f20e-4cdb-8d89-d2bac352f751\") " pod="openshift-cluster-node-tuning-operator/tuned-c8x4k" Oct 30 16:20:07.224828 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.224397 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d10078a0-c21c-4f18-9c5a-4915fb7c8af3-host\") pod \"node-ca-2l5ht\" (UID: \"d10078a0-c21c-4f18-9c5a-4915fb7c8af3\") " pod="openshift-image-registry/node-ca-2l5ht" Oct 30 16:20:07.224828 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.224412 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f2de1ce0-1c1e-4276-80a9-66239e13e352-etc-openvswitch\") pod \"ovnkube-node-2krz6\" (UID: \"f2de1ce0-1c1e-4276-80a9-66239e13e352\") " pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:07.267965 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.267901 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-136.ec2.internal" podStartSLOduration=1.267890656 podStartE2EDuration="1.267890656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 16:20:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:20:07.267794779 +0000 UTC m=+18.568082999" watchObservedRunningTime="2026-10-30 16:20:07.267890656 +0000 UTC m=+18.568178873" Oct 30 16:20:07.325377 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.325357 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/749b2b57-f20e-4cdb-8d89-d2bac352f751-etc-kubernetes\") pod \"tuned-c8x4k\" (UID: \"749b2b57-f20e-4cdb-8d89-d2bac352f751\") " pod="openshift-cluster-node-tuning-operator/tuned-c8x4k" Oct 30 16:20:07.325747 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.325384 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljv94\" (UniqueName: \"kubernetes.io/projected/2d1ed7d8-37cc-4652-b225-3a715b755e48-kube-api-access-ljv94\") pod \"network-check-target-b4ljl\" (UID: \"2d1ed7d8-37cc-4652-b225-3a715b755e48\") " pod="openshift-network-diagnostics/network-check-target-b4ljl" Oct 30 16:20:07.325747 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.325399 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f2de1ce0-1c1e-4276-80a9-66239e13e352-run-systemd\") pod \"ovnkube-node-2krz6\" (UID: \"f2de1ce0-1c1e-4276-80a9-66239e13e352\") " pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:07.325747 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.325421 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/33b888e2-d349-439b-9aa1-5b519e0e08f7-multus-socket-dir-parent\") pod \"multus-7f2mt\" (UID: \"33b888e2-d349-439b-9aa1-5b519e0e08f7\") " pod="openshift-multus/multus-7f2mt" Oct 30 16:20:07.325747 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.325443 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/33b888e2-d349-439b-9aa1-5b519e0e08f7-multus-daemon-config\") pod \"multus-7f2mt\" (UID: \"33b888e2-d349-439b-9aa1-5b519e0e08f7\") " pod="openshift-multus/multus-7f2mt" Oct 30 16:20:07.325747 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.325463 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/33b888e2-d349-439b-9aa1-5b519e0e08f7-etc-kubernetes\") pod \"multus-7f2mt\" (UID: \"33b888e2-d349-439b-9aa1-5b519e0e08f7\") " pod="openshift-multus/multus-7f2mt" Oct 30 16:20:07.325747 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.325489 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f2de1ce0-1c1e-4276-80a9-66239e13e352-run-systemd\") pod \"ovnkube-node-2krz6\" (UID: \"f2de1ce0-1c1e-4276-80a9-66239e13e352\") " pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:07.325747 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.325502 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/33b888e2-d349-439b-9aa1-5b519e0e08f7-multus-socket-dir-parent\") pod \"multus-7f2mt\" (UID: \"33b888e2-d349-439b-9aa1-5b519e0e08f7\") " pod="openshift-multus/multus-7f2mt" Oct 30 16:20:07.325747 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.325512 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/33b888e2-d349-439b-9aa1-5b519e0e08f7-etc-kubernetes\") pod \"multus-7f2mt\" (UID: \"33b888e2-d349-439b-9aa1-5b519e0e08f7\") " pod="openshift-multus/multus-7f2mt" Oct 30 16:20:07.325747 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.325514 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/749b2b57-f20e-4cdb-8d89-d2bac352f751-etc-kubernetes\") pod \"tuned-c8x4k\" (UID: \"749b2b57-f20e-4cdb-8d89-d2bac352f751\") " pod="openshift-cluster-node-tuning-operator/tuned-c8x4k" Oct 30 16:20:07.325747 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.325568 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9ec1217e-8623-42a6-a052-adee153936a2-hosts-file\") pod \"node-resolver-lghnm\" (UID: \"9ec1217e-8623-42a6-a052-adee153936a2\") " pod="openshift-dns/node-resolver-lghnm" Oct 30 16:20:07.325747 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.325620 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9ec1217e-8623-42a6-a052-adee153936a2-hosts-file\") pod \"node-resolver-lghnm\" (UID: \"9ec1217e-8623-42a6-a052-adee153936a2\") " pod="openshift-dns/node-resolver-lghnm" Oct 30 16:20:07.325747 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.325623 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d10078a0-c21c-4f18-9c5a-4915fb7c8af3-serviceca\") pod \"node-ca-2l5ht\" (UID: \"d10078a0-c21c-4f18-9c5a-4915fb7c8af3\") " pod="openshift-image-registry/node-ca-2l5ht" Oct 30 16:20:07.325747 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.325652 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f2de1ce0-1c1e-4276-80a9-66239e13e352-host-kubelet\") pod \"ovnkube-node-2krz6\" (UID: \"f2de1ce0-1c1e-4276-80a9-66239e13e352\") " pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:07.325747 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.325681 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f2de1ce0-1c1e-4276-80a9-66239e13e352-host-run-ovn-kubernetes\") pod \"ovnkube-node-2krz6\" (UID: \"f2de1ce0-1c1e-4276-80a9-66239e13e352\") " pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:07.325747 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.325713 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f2de1ce0-1c1e-4276-80a9-66239e13e352-host-kubelet\") pod \"ovnkube-node-2krz6\" (UID: \"f2de1ce0-1c1e-4276-80a9-66239e13e352\") " pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:07.325747 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.325735 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/749b2b57-f20e-4cdb-8d89-d2bac352f751-etc-systemd\") pod \"tuned-c8x4k\" (UID: \"749b2b57-f20e-4cdb-8d89-d2bac352f751\") " pod="openshift-cluster-node-tuning-operator/tuned-c8x4k" Oct 30 16:20:07.325747 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.325733 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f2de1ce0-1c1e-4276-80a9-66239e13e352-host-run-ovn-kubernetes\") pod \"ovnkube-node-2krz6\" (UID: \"f2de1ce0-1c1e-4276-80a9-66239e13e352\") " pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:07.326532 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.325754 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9ea50669-2c08-403d-8d08-f9ffffa5b896-cni-binary-copy\") pod \"multus-additional-cni-plugins-x9ddt\" (UID: \"9ea50669-2c08-403d-8d08-f9ffffa5b896\") " pod="openshift-multus/multus-additional-cni-plugins-x9ddt" Oct 30 16:20:07.326532 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.325811 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5tsm\" (UniqueName: \"kubernetes.io/projected/6f3d36d6-5098-43cc-8f8a-bd83d5dea5fc-kube-api-access-v5tsm\") pod \"network-metrics-daemon-xj6cp\" (UID: \"6f3d36d6-5098-43cc-8f8a-bd83d5dea5fc\") " pod="openshift-multus/network-metrics-daemon-xj6cp" Oct 30 16:20:07.326532 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.325774 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/749b2b57-f20e-4cdb-8d89-d2bac352f751-etc-systemd\") pod \"tuned-c8x4k\" (UID: \"749b2b57-f20e-4cdb-8d89-d2bac352f751\") " pod="openshift-cluster-node-tuning-operator/tuned-c8x4k" Oct 30 16:20:07.326532 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.325845 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/029f4ecb-eb6b-4693-9e9e-ca61f554dd33-original-pull-secret\") pod \"global-pull-secret-syncer-ncfzj\" (UID: \"029f4ecb-eb6b-4693-9e9e-ca61f554dd33\") " pod="kube-system/global-pull-secret-syncer-ncfzj" Oct 30 16:20:07.326532 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.325876 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a6b95002-74f1-475d-a64a-6495733cf142-sys-fs\") pod \"aws-ebs-csi-driver-node-g69ll\" (UID: \"a6b95002-74f1-475d-a64a-6495733cf142\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g69ll" Oct 30 16:20:07.326532 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.325901 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5sxw\" (UniqueName: \"kubernetes.io/projected/33b888e2-d349-439b-9aa1-5b519e0e08f7-kube-api-access-v5sxw\") pod \"multus-7f2mt\" (UID: \"33b888e2-d349-439b-9aa1-5b519e0e08f7\") " pod="openshift-multus/multus-7f2mt" Oct 30 16:20:07.326532 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:07.325922 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 16:20:07.326532 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.325924 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfgdd\" (UniqueName: \"kubernetes.io/projected/749b2b57-f20e-4cdb-8d89-d2bac352f751-kube-api-access-dfgdd\") pod \"tuned-c8x4k\" (UID: \"749b2b57-f20e-4cdb-8d89-d2bac352f751\") " pod="openshift-cluster-node-tuning-operator/tuned-c8x4k" Oct 30 16:20:07.326532 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.325951 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9ea50669-2c08-403d-8d08-f9ffffa5b896-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-x9ddt\" (UID: \"9ea50669-2c08-403d-8d08-f9ffffa5b896\") " pod="openshift-multus/multus-additional-cni-plugins-x9ddt" Oct 30 16:20:07.326532 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.325976 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d10078a0-c21c-4f18-9c5a-4915fb7c8af3-serviceca\") pod \"node-ca-2l5ht\" (UID: \"d10078a0-c21c-4f18-9c5a-4915fb7c8af3\") " pod="openshift-image-registry/node-ca-2l5ht" Oct 30 16:20:07.326532 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:07.326000 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/029f4ecb-eb6b-4693-9e9e-ca61f554dd33-original-pull-secret podName:029f4ecb-eb6b-4693-9e9e-ca61f554dd33 nodeName:}" failed. No retries permitted until 2026-10-30 16:20:07.825961093 +0000 UTC m=+19.126249313 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/029f4ecb-eb6b-4693-9e9e-ca61f554dd33-original-pull-secret") pod "global-pull-secret-syncer-ncfzj" (UID: "029f4ecb-eb6b-4693-9e9e-ca61f554dd33") : object "kube-system"/"original-pull-secret" not registered Oct 30 16:20:07.326532 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.326019 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpgwx\" (UniqueName: \"kubernetes.io/projected/9ea50669-2c08-403d-8d08-f9ffffa5b896-kube-api-access-zpgwx\") pod \"multus-additional-cni-plugins-x9ddt\" (UID: \"9ea50669-2c08-403d-8d08-f9ffffa5b896\") " pod="openshift-multus/multus-additional-cni-plugins-x9ddt" Oct 30 16:20:07.326532 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.326028 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/33b888e2-d349-439b-9aa1-5b519e0e08f7-multus-daemon-config\") pod \"multus-7f2mt\" (UID: \"33b888e2-d349-439b-9aa1-5b519e0e08f7\") " pod="openshift-multus/multus-7f2mt" Oct 30 16:20:07.326532 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.325973 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a6b95002-74f1-475d-a64a-6495733cf142-sys-fs\") pod \"aws-ebs-csi-driver-node-g69ll\" (UID: \"a6b95002-74f1-475d-a64a-6495733cf142\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g69ll" Oct 30 16:20:07.326532 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.326149 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f2de1ce0-1c1e-4276-80a9-66239e13e352-systemd-units\") pod \"ovnkube-node-2krz6\" (UID: \"f2de1ce0-1c1e-4276-80a9-66239e13e352\") " pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:07.326532 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.326272 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9ea50669-2c08-403d-8d08-f9ffffa5b896-cni-binary-copy\") pod \"multus-additional-cni-plugins-x9ddt\" (UID: \"9ea50669-2c08-403d-8d08-f9ffffa5b896\") " pod="openshift-multus/multus-additional-cni-plugins-x9ddt" Oct 30 16:20:07.326532 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.326273 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f2de1ce0-1c1e-4276-80a9-66239e13e352-log-socket\") pod \"ovnkube-node-2krz6\" (UID: \"f2de1ce0-1c1e-4276-80a9-66239e13e352\") " pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:07.327309 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.326293 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f2de1ce0-1c1e-4276-80a9-66239e13e352-systemd-units\") pod \"ovnkube-node-2krz6\" (UID: \"f2de1ce0-1c1e-4276-80a9-66239e13e352\") " pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:07.327309 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.326336 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f2de1ce0-1c1e-4276-80a9-66239e13e352-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2krz6\" (UID: \"f2de1ce0-1c1e-4276-80a9-66239e13e352\") " pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:07.327309 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.326342 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f2de1ce0-1c1e-4276-80a9-66239e13e352-log-socket\") pod \"ovnkube-node-2krz6\" (UID: \"f2de1ce0-1c1e-4276-80a9-66239e13e352\") " pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:07.327309 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.326358 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f2de1ce0-1c1e-4276-80a9-66239e13e352-ovn-node-metrics-cert\") pod \"ovnkube-node-2krz6\" (UID: \"f2de1ce0-1c1e-4276-80a9-66239e13e352\") " pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:07.327309 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.326377 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/33b888e2-d349-439b-9aa1-5b519e0e08f7-host-var-lib-cni-multus\") pod \"multus-7f2mt\" (UID: \"33b888e2-d349-439b-9aa1-5b519e0e08f7\") " pod="openshift-multus/multus-7f2mt" Oct 30 16:20:07.327309 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.326394 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f2de1ce0-1c1e-4276-80a9-66239e13e352-host-slash\") pod \"ovnkube-node-2krz6\" (UID: \"f2de1ce0-1c1e-4276-80a9-66239e13e352\") " pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:07.327309 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.326408 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/33b888e2-d349-439b-9aa1-5b519e0e08f7-host-run-k8s-cni-cncf-io\") pod \"multus-7f2mt\" (UID: \"33b888e2-d349-439b-9aa1-5b519e0e08f7\") " pod="openshift-multus/multus-7f2mt" Oct 30 16:20:07.327309 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.326377 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f2de1ce0-1c1e-4276-80a9-66239e13e352-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2krz6\" (UID: \"f2de1ce0-1c1e-4276-80a9-66239e13e352\") " pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:07.327309 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.326428 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/749b2b57-f20e-4cdb-8d89-d2bac352f751-lib-modules\") pod \"tuned-c8x4k\" (UID: \"749b2b57-f20e-4cdb-8d89-d2bac352f751\") " pod="openshift-cluster-node-tuning-operator/tuned-c8x4k" Oct 30 16:20:07.327309 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.326452 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/33b888e2-d349-439b-9aa1-5b519e0e08f7-host-var-lib-cni-multus\") pod \"multus-7f2mt\" (UID: \"33b888e2-d349-439b-9aa1-5b519e0e08f7\") " pod="openshift-multus/multus-7f2mt" Oct 30 16:20:07.327309 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.326463 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f2de1ce0-1c1e-4276-80a9-66239e13e352-host-slash\") pod \"ovnkube-node-2krz6\" (UID: \"f2de1ce0-1c1e-4276-80a9-66239e13e352\") " pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:07.327309 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.326472 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f2de1ce0-1c1e-4276-80a9-66239e13e352-host-cni-bin\") pod \"ovnkube-node-2krz6\" (UID: \"f2de1ce0-1c1e-4276-80a9-66239e13e352\") " pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:07.327309 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.326480 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9ea50669-2c08-403d-8d08-f9ffffa5b896-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-x9ddt\" (UID: \"9ea50669-2c08-403d-8d08-f9ffffa5b896\") " pod="openshift-multus/multus-additional-cni-plugins-x9ddt" Oct 30 16:20:07.327309 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.326499 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25jwd\" (UniqueName: \"kubernetes.io/projected/d10078a0-c21c-4f18-9c5a-4915fb7c8af3-kube-api-access-25jwd\") pod \"node-ca-2l5ht\" (UID: \"d10078a0-c21c-4f18-9c5a-4915fb7c8af3\") " pod="openshift-image-registry/node-ca-2l5ht" Oct 30 16:20:07.327309 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.326524 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4c515c55-0717-4974-9f1a-68e0a25984cd-agent-certs\") pod \"konnectivity-agent-sg9tg\" (UID: \"4c515c55-0717-4974-9f1a-68e0a25984cd\") " pod="kube-system/konnectivity-agent-sg9tg" Oct 30 16:20:07.327309 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.326529 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f2de1ce0-1c1e-4276-80a9-66239e13e352-host-cni-bin\") pod \"ovnkube-node-2krz6\" (UID: \"f2de1ce0-1c1e-4276-80a9-66239e13e352\") " pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:07.327309 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.326548 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4c515c55-0717-4974-9f1a-68e0a25984cd-konnectivity-ca\") pod \"konnectivity-agent-sg9tg\" (UID: \"4c515c55-0717-4974-9f1a-68e0a25984cd\") " pod="kube-system/konnectivity-agent-sg9tg" Oct 30 16:20:07.328092 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.326599 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/029f4ecb-eb6b-4693-9e9e-ca61f554dd33-dbus\") pod \"global-pull-secret-syncer-ncfzj\" (UID: \"029f4ecb-eb6b-4693-9e9e-ca61f554dd33\") " pod="kube-system/global-pull-secret-syncer-ncfzj" Oct 30 16:20:07.328092 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.326622 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/33b888e2-d349-439b-9aa1-5b519e0e08f7-host-var-lib-kubelet\") pod \"multus-7f2mt\" (UID: \"33b888e2-d349-439b-9aa1-5b519e0e08f7\") " pod="openshift-multus/multus-7f2mt" Oct 30 16:20:07.328092 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.326668 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/749b2b57-f20e-4cdb-8d89-d2bac352f751-etc-modprobe-d\") pod \"tuned-c8x4k\" (UID: \"749b2b57-f20e-4cdb-8d89-d2bac352f751\") " pod="openshift-cluster-node-tuning-operator/tuned-c8x4k" Oct 30 16:20:07.328092 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.326691 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9ea50669-2c08-403d-8d08-f9ffffa5b896-cnibin\") pod \"multus-additional-cni-plugins-x9ddt\" (UID: \"9ea50669-2c08-403d-8d08-f9ffffa5b896\") " pod="openshift-multus/multus-additional-cni-plugins-x9ddt" Oct 30 16:20:07.328092 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.326721 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f2de1ce0-1c1e-4276-80a9-66239e13e352-var-lib-openvswitch\") pod \"ovnkube-node-2krz6\" (UID: \"f2de1ce0-1c1e-4276-80a9-66239e13e352\") " pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:07.328092 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.326746 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/25c51397-40b9-432c-949d-12ba9f36532a-host-slash\") pod \"iptables-alerter-d9kr8\" (UID: \"25c51397-40b9-432c-949d-12ba9f36532a\") " pod="openshift-network-operator/iptables-alerter-d9kr8" Oct 30 16:20:07.328092 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.326755 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/029f4ecb-eb6b-4693-9e9e-ca61f554dd33-dbus\") pod \"global-pull-secret-syncer-ncfzj\" (UID: \"029f4ecb-eb6b-4693-9e9e-ca61f554dd33\") " pod="kube-system/global-pull-secret-syncer-ncfzj" Oct 30 16:20:07.328092 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.326768 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a6b95002-74f1-475d-a64a-6495733cf142-device-dir\") pod \"aws-ebs-csi-driver-node-g69ll\" (UID: \"a6b95002-74f1-475d-a64a-6495733cf142\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g69ll" Oct 30 16:20:07.328092 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.326797 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/33b888e2-d349-439b-9aa1-5b519e0e08f7-host-run-netns\") pod \"multus-7f2mt\" (UID: \"33b888e2-d349-439b-9aa1-5b519e0e08f7\") " pod="openshift-multus/multus-7f2mt" Oct 30 16:20:07.328092 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.326813 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/749b2b57-f20e-4cdb-8d89-d2bac352f751-var-lib-kubelet\") pod \"tuned-c8x4k\" (UID: \"749b2b57-f20e-4cdb-8d89-d2bac352f751\") " pod="openshift-cluster-node-tuning-operator/tuned-c8x4k" Oct 30 16:20:07.328092 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.326820 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a6b95002-74f1-475d-a64a-6495733cf142-device-dir\") pod \"aws-ebs-csi-driver-node-g69ll\" (UID: \"a6b95002-74f1-475d-a64a-6495733cf142\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g69ll" Oct 30 16:20:07.328092 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.326839 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9ea50669-2c08-403d-8d08-f9ffffa5b896-cnibin\") pod \"multus-additional-cni-plugins-x9ddt\" (UID: \"9ea50669-2c08-403d-8d08-f9ffffa5b896\") " pod="openshift-multus/multus-additional-cni-plugins-x9ddt" Oct 30 16:20:07.328092 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.326844 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/749b2b57-f20e-4cdb-8d89-d2bac352f751-etc-tuned\") pod \"tuned-c8x4k\" (UID: \"749b2b57-f20e-4cdb-8d89-d2bac352f751\") " pod="openshift-cluster-node-tuning-operator/tuned-c8x4k" Oct 30 16:20:07.328092 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.326852 2580 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 16:20:07.328092 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.326886 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d10078a0-c21c-4f18-9c5a-4915fb7c8af3-host\") pod \"node-ca-2l5ht\" (UID: \"d10078a0-c21c-4f18-9c5a-4915fb7c8af3\") " pod="openshift-image-registry/node-ca-2l5ht" Oct 30 16:20:07.328092 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.326899 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/25c51397-40b9-432c-949d-12ba9f36532a-host-slash\") pod \"iptables-alerter-d9kr8\" (UID: \"25c51397-40b9-432c-949d-12ba9f36532a\") " pod="openshift-network-operator/iptables-alerter-d9kr8" Oct 30 16:20:07.328092 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.326913 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f2de1ce0-1c1e-4276-80a9-66239e13e352-etc-openvswitch\") pod \"ovnkube-node-2krz6\" (UID: \"f2de1ce0-1c1e-4276-80a9-66239e13e352\") " pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:07.328092 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.326939 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/33b888e2-d349-439b-9aa1-5b519e0e08f7-host-run-netns\") pod \"multus-7f2mt\" (UID: \"33b888e2-d349-439b-9aa1-5b519e0e08f7\") " pod="openshift-multus/multus-7f2mt" Oct 30 16:20:07.328968 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.326938 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a6b95002-74f1-475d-a64a-6495733cf142-registration-dir\") pod \"aws-ebs-csi-driver-node-g69ll\" (UID: \"a6b95002-74f1-475d-a64a-6495733cf142\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g69ll" Oct 30 16:20:07.328968 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.326978 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a6b95002-74f1-475d-a64a-6495733cf142-registration-dir\") pod \"aws-ebs-csi-driver-node-g69ll\" (UID: \"a6b95002-74f1-475d-a64a-6495733cf142\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g69ll" Oct 30 16:20:07.328968 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.326970 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/749b2b57-f20e-4cdb-8d89-d2bac352f751-run\") pod \"tuned-c8x4k\" (UID: \"749b2b57-f20e-4cdb-8d89-d2bac352f751\") " pod="openshift-cluster-node-tuning-operator/tuned-c8x4k" Oct 30 16:20:07.328968 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.326865 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f2de1ce0-1c1e-4276-80a9-66239e13e352-var-lib-openvswitch\") pod \"ovnkube-node-2krz6\" (UID: \"f2de1ce0-1c1e-4276-80a9-66239e13e352\") " pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:07.328968 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.327057 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d10078a0-c21c-4f18-9c5a-4915fb7c8af3-host\") pod \"node-ca-2l5ht\" (UID: \"d10078a0-c21c-4f18-9c5a-4915fb7c8af3\") " pod="openshift-image-registry/node-ca-2l5ht" Oct 30 16:20:07.328968 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.327075 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/749b2b57-f20e-4cdb-8d89-d2bac352f751-lib-modules\") pod \"tuned-c8x4k\" (UID: \"749b2b57-f20e-4cdb-8d89-d2bac352f751\") " pod="openshift-cluster-node-tuning-operator/tuned-c8x4k" Oct 30 16:20:07.328968 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.327081 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4c515c55-0717-4974-9f1a-68e0a25984cd-konnectivity-ca\") pod \"konnectivity-agent-sg9tg\" (UID: \"4c515c55-0717-4974-9f1a-68e0a25984cd\") " pod="kube-system/konnectivity-agent-sg9tg" Oct 30 16:20:07.328968 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.327093 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f2de1ce0-1c1e-4276-80a9-66239e13e352-etc-openvswitch\") pod \"ovnkube-node-2krz6\" (UID: \"f2de1ce0-1c1e-4276-80a9-66239e13e352\") " pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:07.328968 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.327114 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/749b2b57-f20e-4cdb-8d89-d2bac352f751-etc-modprobe-d\") pod \"tuned-c8x4k\" (UID: \"749b2b57-f20e-4cdb-8d89-d2bac352f751\") " pod="openshift-cluster-node-tuning-operator/tuned-c8x4k" Oct 30 16:20:07.328968 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.327125 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/749b2b57-f20e-4cdb-8d89-d2bac352f751-tmp\") pod \"tuned-c8x4k\" (UID: \"749b2b57-f20e-4cdb-8d89-d2bac352f751\") " pod="openshift-cluster-node-tuning-operator/tuned-c8x4k" Oct 30 16:20:07.328968 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.327144 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/749b2b57-f20e-4cdb-8d89-d2bac352f751-run\") pod \"tuned-c8x4k\" (UID: \"749b2b57-f20e-4cdb-8d89-d2bac352f751\") " pod="openshift-cluster-node-tuning-operator/tuned-c8x4k" Oct 30 16:20:07.328968 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.327156 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/749b2b57-f20e-4cdb-8d89-d2bac352f751-var-lib-kubelet\") pod \"tuned-c8x4k\" (UID: \"749b2b57-f20e-4cdb-8d89-d2bac352f751\") " pod="openshift-cluster-node-tuning-operator/tuned-c8x4k" Oct 30 16:20:07.328968 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.327176 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/33b888e2-d349-439b-9aa1-5b519e0e08f7-host-var-lib-kubelet\") pod \"multus-7f2mt\" (UID: \"33b888e2-d349-439b-9aa1-5b519e0e08f7\") " pod="openshift-multus/multus-7f2mt" Oct 30 16:20:07.328968 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.327192 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f2de1ce0-1c1e-4276-80a9-66239e13e352-run-ovn\") pod \"ovnkube-node-2krz6\" (UID: \"f2de1ce0-1c1e-4276-80a9-66239e13e352\") " pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:07.328968 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.327199 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/33b888e2-d349-439b-9aa1-5b519e0e08f7-host-run-k8s-cni-cncf-io\") pod \"multus-7f2mt\" (UID: \"33b888e2-d349-439b-9aa1-5b519e0e08f7\") " pod="openshift-multus/multus-7f2mt" Oct 30 16:20:07.328968 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.327295 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a6b95002-74f1-475d-a64a-6495733cf142-etc-selinux\") pod \"aws-ebs-csi-driver-node-g69ll\" (UID: \"a6b95002-74f1-475d-a64a-6495733cf142\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g69ll" Oct 30 16:20:07.328968 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.327306 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f2de1ce0-1c1e-4276-80a9-66239e13e352-run-ovn\") pod \"ovnkube-node-2krz6\" (UID: \"f2de1ce0-1c1e-4276-80a9-66239e13e352\") " pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:07.328968 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.327346 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/33b888e2-d349-439b-9aa1-5b519e0e08f7-system-cni-dir\") pod \"multus-7f2mt\" (UID: \"33b888e2-d349-439b-9aa1-5b519e0e08f7\") " pod="openshift-multus/multus-7f2mt" Oct 30 16:20:07.329830 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.327381 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ec1217e-8623-42a6-a052-adee153936a2-tmp-dir\") pod \"node-resolver-lghnm\" (UID: \"9ec1217e-8623-42a6-a052-adee153936a2\") " pod="openshift-dns/node-resolver-lghnm" Oct 30 16:20:07.329830 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.327418 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a6b95002-74f1-475d-a64a-6495733cf142-etc-selinux\") pod \"aws-ebs-csi-driver-node-g69ll\" (UID: \"a6b95002-74f1-475d-a64a-6495733cf142\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g69ll" Oct 30 16:20:07.329830 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.327419 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrppt\" (UniqueName: \"kubernetes.io/projected/9ec1217e-8623-42a6-a052-adee153936a2-kube-api-access-nrppt\") pod \"node-resolver-lghnm\" (UID: \"9ec1217e-8623-42a6-a052-adee153936a2\") " pod="openshift-dns/node-resolver-lghnm" Oct 30 16:20:07.329830 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.327441 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/33b888e2-d349-439b-9aa1-5b519e0e08f7-system-cni-dir\") pod \"multus-7f2mt\" (UID: \"33b888e2-d349-439b-9aa1-5b519e0e08f7\") " pod="openshift-multus/multus-7f2mt" Oct 30 16:20:07.329830 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.327473 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/749b2b57-f20e-4cdb-8d89-d2bac352f751-etc-sysctl-d\") pod \"tuned-c8x4k\" (UID: \"749b2b57-f20e-4cdb-8d89-d2bac352f751\") " pod="openshift-cluster-node-tuning-operator/tuned-c8x4k" Oct 30 16:20:07.329830 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.327525 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/749b2b57-f20e-4cdb-8d89-d2bac352f751-host\") pod \"tuned-c8x4k\" (UID: \"749b2b57-f20e-4cdb-8d89-d2bac352f751\") " pod="openshift-cluster-node-tuning-operator/tuned-c8x4k" Oct 30 16:20:07.329830 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.327548 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/33b888e2-d349-439b-9aa1-5b519e0e08f7-host-var-lib-cni-bin\") pod \"multus-7f2mt\" (UID: \"33b888e2-d349-439b-9aa1-5b519e0e08f7\") " pod="openshift-multus/multus-7f2mt" Oct 30 16:20:07.329830 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.327602 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/33b888e2-d349-439b-9aa1-5b519e0e08f7-hostroot\") pod \"multus-7f2mt\" (UID: \"33b888e2-d349-439b-9aa1-5b519e0e08f7\") " pod="openshift-multus/multus-7f2mt" Oct 30 16:20:07.329830 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.327625 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9ea50669-2c08-403d-8d08-f9ffffa5b896-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-x9ddt\" (UID: \"9ea50669-2c08-403d-8d08-f9ffffa5b896\") " pod="openshift-multus/multus-additional-cni-plugins-x9ddt" Oct 30 16:20:07.329830 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.327673 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f2de1ce0-1c1e-4276-80a9-66239e13e352-run-openvswitch\") pod \"ovnkube-node-2krz6\" (UID: \"f2de1ce0-1c1e-4276-80a9-66239e13e352\") " pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:07.329830 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.327677 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ec1217e-8623-42a6-a052-adee153936a2-tmp-dir\") pod \"node-resolver-lghnm\" (UID: \"9ec1217e-8623-42a6-a052-adee153936a2\") " pod="openshift-dns/node-resolver-lghnm" Oct 30 16:20:07.329830 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.327715 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/749b2b57-f20e-4cdb-8d89-d2bac352f751-etc-sysctl-d\") pod \"tuned-c8x4k\" (UID: \"749b2b57-f20e-4cdb-8d89-d2bac352f751\") " pod="openshift-cluster-node-tuning-operator/tuned-c8x4k" Oct 30 16:20:07.329830 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.327717 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f2de1ce0-1c1e-4276-80a9-66239e13e352-host-cni-netd\") pod \"ovnkube-node-2krz6\" (UID: \"f2de1ce0-1c1e-4276-80a9-66239e13e352\") " pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:07.329830 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.327726 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/33b888e2-d349-439b-9aa1-5b519e0e08f7-host-var-lib-cni-bin\") pod \"multus-7f2mt\" (UID: \"33b888e2-d349-439b-9aa1-5b519e0e08f7\") " pod="openshift-multus/multus-7f2mt" Oct 30 16:20:07.329830 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.327749 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f2de1ce0-1c1e-4276-80a9-66239e13e352-host-cni-netd\") pod \"ovnkube-node-2krz6\" (UID: \"f2de1ce0-1c1e-4276-80a9-66239e13e352\") " pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:07.329830 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.327784 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/029f4ecb-eb6b-4693-9e9e-ca61f554dd33-kubelet-config\") pod \"global-pull-secret-syncer-ncfzj\" (UID: \"029f4ecb-eb6b-4693-9e9e-ca61f554dd33\") " pod="kube-system/global-pull-secret-syncer-ncfzj" Oct 30 16:20:07.329830 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.327805 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a6b95002-74f1-475d-a64a-6495733cf142-kubelet-dir\") pod \"aws-ebs-csi-driver-node-g69ll\" (UID: \"a6b95002-74f1-475d-a64a-6495733cf142\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g69ll" Oct 30 16:20:07.330644 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.327825 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a6b95002-74f1-475d-a64a-6495733cf142-socket-dir\") pod \"aws-ebs-csi-driver-node-g69ll\" (UID: \"a6b95002-74f1-475d-a64a-6495733cf142\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g69ll" Oct 30 16:20:07.330644 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.327840 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f2de1ce0-1c1e-4276-80a9-66239e13e352-run-openvswitch\") pod \"ovnkube-node-2krz6\" (UID: \"f2de1ce0-1c1e-4276-80a9-66239e13e352\") " pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:07.330644 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.327844 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/33b888e2-d349-439b-9aa1-5b519e0e08f7-multus-cni-dir\") pod \"multus-7f2mt\" (UID: \"33b888e2-d349-439b-9aa1-5b519e0e08f7\") " pod="openshift-multus/multus-7f2mt" Oct 30 16:20:07.330644 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.327874 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/33b888e2-d349-439b-9aa1-5b519e0e08f7-multus-conf-dir\") pod \"multus-7f2mt\" (UID: \"33b888e2-d349-439b-9aa1-5b519e0e08f7\") " pod="openshift-multus/multus-7f2mt" Oct 30 16:20:07.330644 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.327896 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f2de1ce0-1c1e-4276-80a9-66239e13e352-node-log\") pod \"ovnkube-node-2krz6\" (UID: \"f2de1ce0-1c1e-4276-80a9-66239e13e352\") " pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:07.330644 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.327916 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzp5d\" (UniqueName: \"kubernetes.io/projected/25c51397-40b9-432c-949d-12ba9f36532a-kube-api-access-qzp5d\") pod \"iptables-alerter-d9kr8\" (UID: \"25c51397-40b9-432c-949d-12ba9f36532a\") " pod="openshift-network-operator/iptables-alerter-d9kr8" Oct 30 16:20:07.330644 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.327935 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/33b888e2-d349-439b-9aa1-5b519e0e08f7-cnibin\") pod \"multus-7f2mt\" (UID: \"33b888e2-d349-439b-9aa1-5b519e0e08f7\") " pod="openshift-multus/multus-7f2mt" Oct 30 16:20:07.330644 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.327957 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/749b2b57-f20e-4cdb-8d89-d2bac352f751-etc-sysctl-conf\") pod \"tuned-c8x4k\" (UID: \"749b2b57-f20e-4cdb-8d89-d2bac352f751\") " pod="openshift-cluster-node-tuning-operator/tuned-c8x4k" Oct 30 16:20:07.330644 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.328025 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9ea50669-2c08-403d-8d08-f9ffffa5b896-system-cni-dir\") pod \"multus-additional-cni-plugins-x9ddt\" (UID: \"9ea50669-2c08-403d-8d08-f9ffffa5b896\") " pod="openshift-multus/multus-additional-cni-plugins-x9ddt" Oct 30 16:20:07.330644 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.328048 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f2de1ce0-1c1e-4276-80a9-66239e13e352-ovnkube-config\") pod \"ovnkube-node-2krz6\" (UID: \"f2de1ce0-1c1e-4276-80a9-66239e13e352\") " pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:07.330644 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.328048 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a6b95002-74f1-475d-a64a-6495733cf142-socket-dir\") pod \"aws-ebs-csi-driver-node-g69ll\" (UID: \"a6b95002-74f1-475d-a64a-6495733cf142\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g69ll" Oct 30 16:20:07.330644 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.328068 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/749b2b57-f20e-4cdb-8d89-d2bac352f751-etc-sysconfig\") pod \"tuned-c8x4k\" (UID: \"749b2b57-f20e-4cdb-8d89-d2bac352f751\") " pod="openshift-cluster-node-tuning-operator/tuned-c8x4k" Oct 30 16:20:07.330644 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.328090 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9ea50669-2c08-403d-8d08-f9ffffa5b896-tuning-conf-dir\") pod \"multus-additional-cni-plugins-x9ddt\" (UID: \"9ea50669-2c08-403d-8d08-f9ffffa5b896\") " pod="openshift-multus/multus-additional-cni-plugins-x9ddt" Oct 30 16:20:07.330644 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.328097 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/33b888e2-d349-439b-9aa1-5b519e0e08f7-cnibin\") pod \"multus-7f2mt\" (UID: \"33b888e2-d349-439b-9aa1-5b519e0e08f7\") " pod="openshift-multus/multus-7f2mt" Oct 30 16:20:07.330644 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.328112 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f2de1ce0-1c1e-4276-80a9-66239e13e352-host-run-netns\") pod \"ovnkube-node-2krz6\" (UID: \"f2de1ce0-1c1e-4276-80a9-66239e13e352\") " pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:07.330644 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.328134 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f3d36d6-5098-43cc-8f8a-bd83d5dea5fc-metrics-certs\") pod \"network-metrics-daemon-xj6cp\" (UID: \"6f3d36d6-5098-43cc-8f8a-bd83d5dea5fc\") " pod="openshift-multus/network-metrics-daemon-xj6cp" Oct 30 16:20:07.330644 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.328160 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f2de1ce0-1c1e-4276-80a9-66239e13e352-env-overrides\") pod \"ovnkube-node-2krz6\" (UID: \"f2de1ce0-1c1e-4276-80a9-66239e13e352\") " pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:07.331274 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.328168 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/749b2b57-f20e-4cdb-8d89-d2bac352f751-etc-sysctl-conf\") pod \"tuned-c8x4k\" (UID: \"749b2b57-f20e-4cdb-8d89-d2bac352f751\") " pod="openshift-cluster-node-tuning-operator/tuned-c8x4k" Oct 30 16:20:07.331274 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.328183 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7d4j4\" (UniqueName: \"kubernetes.io/projected/f2de1ce0-1c1e-4276-80a9-66239e13e352-kube-api-access-7d4j4\") pod \"ovnkube-node-2krz6\" (UID: \"f2de1ce0-1c1e-4276-80a9-66239e13e352\") " pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:07.331274 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.328196 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9ea50669-2c08-403d-8d08-f9ffffa5b896-system-cni-dir\") pod \"multus-additional-cni-plugins-x9ddt\" (UID: \"9ea50669-2c08-403d-8d08-f9ffffa5b896\") " pod="openshift-multus/multus-additional-cni-plugins-x9ddt" Oct 30 16:20:07.331274 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.328252 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a6b95002-74f1-475d-a64a-6495733cf142-kubelet-dir\") pod \"aws-ebs-csi-driver-node-g69ll\" (UID: \"a6b95002-74f1-475d-a64a-6495733cf142\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g69ll" Oct 30 16:20:07.331274 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:07.328269 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 16:20:07.331274 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.328285 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/33b888e2-d349-439b-9aa1-5b519e0e08f7-multus-conf-dir\") pod \"multus-7f2mt\" (UID: \"33b888e2-d349-439b-9aa1-5b519e0e08f7\") " pod="openshift-multus/multus-7f2mt" Oct 30 16:20:07.331274 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.328285 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9ea50669-2c08-403d-8d08-f9ffffa5b896-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-x9ddt\" (UID: \"9ea50669-2c08-403d-8d08-f9ffffa5b896\") " pod="openshift-multus/multus-additional-cni-plugins-x9ddt" Oct 30 16:20:07.331274 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:07.328323 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f3d36d6-5098-43cc-8f8a-bd83d5dea5fc-metrics-certs podName:6f3d36d6-5098-43cc-8f8a-bd83d5dea5fc nodeName:}" failed. No retries permitted until 2026-10-30 16:20:07.828307814 +0000 UTC m=+19.128596014 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6f3d36d6-5098-43cc-8f8a-bd83d5dea5fc-metrics-certs") pod "network-metrics-daemon-xj6cp" (UID: "6f3d36d6-5098-43cc-8f8a-bd83d5dea5fc") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 16:20:07.331274 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.328321 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f2de1ce0-1c1e-4276-80a9-66239e13e352-node-log\") pod \"ovnkube-node-2krz6\" (UID: \"f2de1ce0-1c1e-4276-80a9-66239e13e352\") " pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:07.331274 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.328347 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/25c51397-40b9-432c-949d-12ba9f36532a-iptables-alerter-script\") pod \"iptables-alerter-d9kr8\" (UID: \"25c51397-40b9-432c-949d-12ba9f36532a\") " pod="openshift-network-operator/iptables-alerter-d9kr8" Oct 30 16:20:07.331274 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.328356 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/33b888e2-d349-439b-9aa1-5b519e0e08f7-multus-cni-dir\") pod \"multus-7f2mt\" (UID: \"33b888e2-d349-439b-9aa1-5b519e0e08f7\") " pod="openshift-multus/multus-7f2mt" Oct 30 16:20:07.331274 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.328375 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zlg7\" (UniqueName: \"kubernetes.io/projected/a6b95002-74f1-475d-a64a-6495733cf142-kube-api-access-8zlg7\") pod \"aws-ebs-csi-driver-node-g69ll\" (UID: \"a6b95002-74f1-475d-a64a-6495733cf142\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g69ll" Oct 30 16:20:07.331274 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.328400 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/33b888e2-d349-439b-9aa1-5b519e0e08f7-host-run-multus-certs\") pod \"multus-7f2mt\" (UID: \"33b888e2-d349-439b-9aa1-5b519e0e08f7\") " pod="openshift-multus/multus-7f2mt" Oct 30 16:20:07.331274 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.328424 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/749b2b57-f20e-4cdb-8d89-d2bac352f751-sys\") pod \"tuned-c8x4k\" (UID: \"749b2b57-f20e-4cdb-8d89-d2bac352f751\") " pod="openshift-cluster-node-tuning-operator/tuned-c8x4k" Oct 30 16:20:07.331274 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.328450 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f2de1ce0-1c1e-4276-80a9-66239e13e352-ovnkube-script-lib\") pod \"ovnkube-node-2krz6\" (UID: \"f2de1ce0-1c1e-4276-80a9-66239e13e352\") " pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:07.331274 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.328518 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/33b888e2-d349-439b-9aa1-5b519e0e08f7-hostroot\") pod \"multus-7f2mt\" (UID: \"33b888e2-d349-439b-9aa1-5b519e0e08f7\") " pod="openshift-multus/multus-7f2mt" Oct 30 16:20:07.331274 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.328553 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f2de1ce0-1c1e-4276-80a9-66239e13e352-env-overrides\") pod \"ovnkube-node-2krz6\" (UID: \"f2de1ce0-1c1e-4276-80a9-66239e13e352\") " pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:07.331800 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.328556 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/33b888e2-d349-439b-9aa1-5b519e0e08f7-cni-binary-copy\") pod \"multus-7f2mt\" (UID: \"33b888e2-d349-439b-9aa1-5b519e0e08f7\") " pod="openshift-multus/multus-7f2mt" Oct 30 16:20:07.331800 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.328597 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/749b2b57-f20e-4cdb-8d89-d2bac352f751-etc-sysconfig\") pod \"tuned-c8x4k\" (UID: \"749b2b57-f20e-4cdb-8d89-d2bac352f751\") " pod="openshift-cluster-node-tuning-operator/tuned-c8x4k" Oct 30 16:20:07.331800 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.328749 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9ea50669-2c08-403d-8d08-f9ffffa5b896-tuning-conf-dir\") pod \"multus-additional-cni-plugins-x9ddt\" (UID: \"9ea50669-2c08-403d-8d08-f9ffffa5b896\") " pod="openshift-multus/multus-additional-cni-plugins-x9ddt" Oct 30 16:20:07.331800 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.328785 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f2de1ce0-1c1e-4276-80a9-66239e13e352-host-run-netns\") pod \"ovnkube-node-2krz6\" (UID: \"f2de1ce0-1c1e-4276-80a9-66239e13e352\") " pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:07.331800 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.328958 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/33b888e2-d349-439b-9aa1-5b519e0e08f7-host-run-multus-certs\") pod \"multus-7f2mt\" (UID: \"33b888e2-d349-439b-9aa1-5b519e0e08f7\") " pod="openshift-multus/multus-7f2mt" Oct 30 16:20:07.331800 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.329015 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/749b2b57-f20e-4cdb-8d89-d2bac352f751-sys\") pod \"tuned-c8x4k\" (UID: \"749b2b57-f20e-4cdb-8d89-d2bac352f751\") " pod="openshift-cluster-node-tuning-operator/tuned-c8x4k" Oct 30 16:20:07.331800 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.329064 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/029f4ecb-eb6b-4693-9e9e-ca61f554dd33-kubelet-config\") pod \"global-pull-secret-syncer-ncfzj\" (UID: \"029f4ecb-eb6b-4693-9e9e-ca61f554dd33\") " pod="kube-system/global-pull-secret-syncer-ncfzj" Oct 30 16:20:07.331800 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.329114 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/749b2b57-f20e-4cdb-8d89-d2bac352f751-host\") pod \"tuned-c8x4k\" (UID: \"749b2b57-f20e-4cdb-8d89-d2bac352f751\") " pod="openshift-cluster-node-tuning-operator/tuned-c8x4k" Oct 30 16:20:07.331800 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.329828 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/33b888e2-d349-439b-9aa1-5b519e0e08f7-cni-binary-copy\") pod \"multus-7f2mt\" (UID: \"33b888e2-d349-439b-9aa1-5b519e0e08f7\") " pod="openshift-multus/multus-7f2mt" Oct 30 16:20:07.331800 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.329988 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f2de1ce0-1c1e-4276-80a9-66239e13e352-ovnkube-config\") pod \"ovnkube-node-2krz6\" (UID: \"f2de1ce0-1c1e-4276-80a9-66239e13e352\") " pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:07.331800 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.330016 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f2de1ce0-1c1e-4276-80a9-66239e13e352-ovnkube-script-lib\") pod \"ovnkube-node-2krz6\" (UID: \"f2de1ce0-1c1e-4276-80a9-66239e13e352\") " pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:07.331800 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.330106 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/25c51397-40b9-432c-949d-12ba9f36532a-iptables-alerter-script\") pod \"iptables-alerter-d9kr8\" (UID: \"25c51397-40b9-432c-949d-12ba9f36532a\") " pod="openshift-network-operator/iptables-alerter-d9kr8" Oct 30 16:20:07.331800 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.330978 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f2de1ce0-1c1e-4276-80a9-66239e13e352-ovn-node-metrics-cert\") pod \"ovnkube-node-2krz6\" (UID: \"f2de1ce0-1c1e-4276-80a9-66239e13e352\") " pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:07.331800 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.331059 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/749b2b57-f20e-4cdb-8d89-d2bac352f751-tmp\") pod \"tuned-c8x4k\" (UID: \"749b2b57-f20e-4cdb-8d89-d2bac352f751\") " pod="openshift-cluster-node-tuning-operator/tuned-c8x4k" Oct 30 16:20:07.331800 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.331113 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/749b2b57-f20e-4cdb-8d89-d2bac352f751-etc-tuned\") pod \"tuned-c8x4k\" (UID: \"749b2b57-f20e-4cdb-8d89-d2bac352f751\") " pod="openshift-cluster-node-tuning-operator/tuned-c8x4k" Oct 30 16:20:07.331800 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.331123 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4c515c55-0717-4974-9f1a-68e0a25984cd-agent-certs\") pod \"konnectivity-agent-sg9tg\" (UID: \"4c515c55-0717-4974-9f1a-68e0a25984cd\") " pod="kube-system/konnectivity-agent-sg9tg" Oct 30 16:20:07.349508 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:07.349486 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 16:20:07.349508 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:07.349505 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 16:20:07.349659 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:07.349516 2580 projected.go:196] Error preparing data for projected volume kube-api-access-ljv94 for pod openshift-network-diagnostics/network-check-target-b4ljl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 16:20:07.349659 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:07.349558 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2d1ed7d8-37cc-4652-b225-3a715b755e48-kube-api-access-ljv94 podName:2d1ed7d8-37cc-4652-b225-3a715b755e48 nodeName:}" failed. No retries permitted until 2026-10-30 16:20:07.849547094 +0000 UTC m=+19.149835315 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-ljv94" (UniqueName: "kubernetes.io/projected/2d1ed7d8-37cc-4652-b225-3a715b755e48-kube-api-access-ljv94") pod "network-check-target-b4ljl" (UID: "2d1ed7d8-37cc-4652-b225-3a715b755e48") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 16:20:07.352409 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.351971 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5tsm\" (UniqueName: \"kubernetes.io/projected/6f3d36d6-5098-43cc-8f8a-bd83d5dea5fc-kube-api-access-v5tsm\") pod \"network-metrics-daemon-xj6cp\" (UID: \"6f3d36d6-5098-43cc-8f8a-bd83d5dea5fc\") " pod="openshift-multus/network-metrics-daemon-xj6cp" Oct 30 16:20:07.352852 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.352826 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfgdd\" (UniqueName: \"kubernetes.io/projected/749b2b57-f20e-4cdb-8d89-d2bac352f751-kube-api-access-dfgdd\") pod \"tuned-c8x4k\" (UID: \"749b2b57-f20e-4cdb-8d89-d2bac352f751\") " pod="openshift-cluster-node-tuning-operator/tuned-c8x4k" Oct 30 16:20:07.352949 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.352830 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5sxw\" (UniqueName: \"kubernetes.io/projected/33b888e2-d349-439b-9aa1-5b519e0e08f7-kube-api-access-v5sxw\") pod \"multus-7f2mt\" (UID: \"33b888e2-d349-439b-9aa1-5b519e0e08f7\") " pod="openshift-multus/multus-7f2mt" Oct 30 16:20:07.353206 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.353175 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpgwx\" (UniqueName: \"kubernetes.io/projected/9ea50669-2c08-403d-8d08-f9ffffa5b896-kube-api-access-zpgwx\") pod \"multus-additional-cni-plugins-x9ddt\" (UID: \"9ea50669-2c08-403d-8d08-f9ffffa5b896\") " pod="openshift-multus/multus-additional-cni-plugins-x9ddt" Oct 30 16:20:07.353291 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.353255 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zlg7\" (UniqueName: \"kubernetes.io/projected/a6b95002-74f1-475d-a64a-6495733cf142-kube-api-access-8zlg7\") pod \"aws-ebs-csi-driver-node-g69ll\" (UID: \"a6b95002-74f1-475d-a64a-6495733cf142\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g69ll" Oct 30 16:20:07.353291 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.353269 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzp5d\" (UniqueName: \"kubernetes.io/projected/25c51397-40b9-432c-949d-12ba9f36532a-kube-api-access-qzp5d\") pod \"iptables-alerter-d9kr8\" (UID: \"25c51397-40b9-432c-949d-12ba9f36532a\") " pod="openshift-network-operator/iptables-alerter-d9kr8" Oct 30 16:20:07.353366 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.353319 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7d4j4\" (UniqueName: \"kubernetes.io/projected/f2de1ce0-1c1e-4276-80a9-66239e13e352-kube-api-access-7d4j4\") pod \"ovnkube-node-2krz6\" (UID: \"f2de1ce0-1c1e-4276-80a9-66239e13e352\") " pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:07.353710 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.353696 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25jwd\" (UniqueName: \"kubernetes.io/projected/d10078a0-c21c-4f18-9c5a-4915fb7c8af3-kube-api-access-25jwd\") pod \"node-ca-2l5ht\" (UID: \"d10078a0-c21c-4f18-9c5a-4915fb7c8af3\") " pod="openshift-image-registry/node-ca-2l5ht" Oct 30 16:20:07.354483 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.354467 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrppt\" (UniqueName: \"kubernetes.io/projected/9ec1217e-8623-42a6-a052-adee153936a2-kube-api-access-nrppt\") pod \"node-resolver-lghnm\" (UID: \"9ec1217e-8623-42a6-a052-adee153936a2\") " pod="openshift-dns/node-resolver-lghnm" Oct 30 16:20:07.371058 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.371043 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2l5ht" Oct 30 16:20:07.377252 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.377235 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-sg9tg" Oct 30 16:20:07.379808 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.379791 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7f2mt" Oct 30 16:20:07.380531 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:20:07.380513 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd10078a0_c21c_4f18_9c5a_4915fb7c8af3.slice/crio-2e25f0b762d57dbd49faf26aa31b5c096227a38a4195247540dada265b1c1c50 WatchSource:0}: Error finding container 2e25f0b762d57dbd49faf26aa31b5c096227a38a4195247540dada265b1c1c50: Status 404 returned error can't find the container with id 2e25f0b762d57dbd49faf26aa31b5c096227a38a4195247540dada265b1c1c50 Oct 30 16:20:07.386239 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:20:07.386217 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c515c55_0717_4974_9f1a_68e0a25984cd.slice/crio-8dbde36737364320b9db4f9b6b3ec9cb036413547443ff1f770d9244b3824331 WatchSource:0}: Error finding container 8dbde36737364320b9db4f9b6b3ec9cb036413547443ff1f770d9244b3824331: Status 404 returned error can't find the container with id 8dbde36737364320b9db4f9b6b3ec9cb036413547443ff1f770d9244b3824331 Oct 30 16:20:07.388637 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:20:07.388618 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33b888e2_d349_439b_9aa1_5b519e0e08f7.slice/crio-c5f4d46d9d932ab42101bc160c22e1b28cc23fed7c0d7fccf9a8c2bcd51e905e WatchSource:0}: Error finding container c5f4d46d9d932ab42101bc160c22e1b28cc23fed7c0d7fccf9a8c2bcd51e905e: Status 404 returned error can't find the container with id c5f4d46d9d932ab42101bc160c22e1b28cc23fed7c0d7fccf9a8c2bcd51e905e Oct 30 16:20:07.399528 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.399512 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-lghnm" Oct 30 16:20:07.401818 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.401802 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-d9kr8" Oct 30 16:20:07.407878 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.407756 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:07.408775 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:20:07.408755 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ec1217e_8623_42a6_a052_adee153936a2.slice/crio-20e0e4f0bb761cfba7ee62e5fb09dd6dace934d281fb100229b83952b32fba9a WatchSource:0}: Error finding container 20e0e4f0bb761cfba7ee62e5fb09dd6dace934d281fb100229b83952b32fba9a: Status 404 returned error can't find the container with id 20e0e4f0bb761cfba7ee62e5fb09dd6dace934d281fb100229b83952b32fba9a Oct 30 16:20:07.414062 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.414047 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-c8x4k" Oct 30 16:20:07.417004 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:20:07.416981 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25c51397_40b9_432c_949d_12ba9f36532a.slice/crio-1de6e79aacb2bda8e4dae807b79eb5116dc5be949804852582c2f1f5105e4e70 WatchSource:0}: Error finding container 1de6e79aacb2bda8e4dae807b79eb5116dc5be949804852582c2f1f5105e4e70: Status 404 returned error can't find the container with id 1de6e79aacb2bda8e4dae807b79eb5116dc5be949804852582c2f1f5105e4e70 Oct 30 16:20:07.419599 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:20:07.419557 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2de1ce0_1c1e_4276_80a9_66239e13e352.slice/crio-41f89ab13b121f44360e1a4bbeef89b6d52d3402131286344cb4b94f8811e2f4 WatchSource:0}: Error finding container 41f89ab13b121f44360e1a4bbeef89b6d52d3402131286344cb4b94f8811e2f4: Status 404 returned error can't find the container with id 41f89ab13b121f44360e1a4bbeef89b6d52d3402131286344cb4b94f8811e2f4 Oct 30 16:20:07.420137 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.420122 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g69ll" Oct 30 16:20:07.423375 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:20:07.423355 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod749b2b57_f20e_4cdb_8d89_d2bac352f751.slice/crio-8f3c56e9db0a1f1b641e5ba13c8569ba3635c4f16b2329d31e44aa09a0bb0d34 WatchSource:0}: Error finding container 8f3c56e9db0a1f1b641e5ba13c8569ba3635c4f16b2329d31e44aa09a0bb0d34: Status 404 returned error can't find the container with id 8f3c56e9db0a1f1b641e5ba13c8569ba3635c4f16b2329d31e44aa09a0bb0d34 Oct 30 16:20:07.428571 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:20:07.428552 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6b95002_74f1_475d_a64a_6495733cf142.slice/crio-3a0fe4e644ac14b6dbc705911600a33fa60f54ec76fdc1c3cb9a789c664f0eca WatchSource:0}: Error finding container 3a0fe4e644ac14b6dbc705911600a33fa60f54ec76fdc1c3cb9a789c664f0eca: Status 404 returned error can't find the container with id 3a0fe4e644ac14b6dbc705911600a33fa60f54ec76fdc1c3cb9a789c664f0eca Oct 30 16:20:07.430388 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.430371 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-x9ddt" Oct 30 16:20:07.438517 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:20:07.438501 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ea50669_2c08_403d_8d08_f9ffffa5b896.slice/crio-4e2f429b17f5c625d3707f009d7076d2a08fe8c8c4bfe2c2e4b0d03d3d36c00f WatchSource:0}: Error finding container 4e2f429b17f5c625d3707f009d7076d2a08fe8c8c4bfe2c2e4b0d03d3d36c00f: Status 404 returned error can't find the container with id 4e2f429b17f5c625d3707f009d7076d2a08fe8c8c4bfe2c2e4b0d03d3d36c00f Oct 30 16:20:07.474810 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.474789 2580 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 16:20:07.831552 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.831518 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f3d36d6-5098-43cc-8f8a-bd83d5dea5fc-metrics-certs\") pod \"network-metrics-daemon-xj6cp\" (UID: \"6f3d36d6-5098-43cc-8f8a-bd83d5dea5fc\") " pod="openshift-multus/network-metrics-daemon-xj6cp" Oct 30 16:20:07.831728 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.831603 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/029f4ecb-eb6b-4693-9e9e-ca61f554dd33-original-pull-secret\") pod \"global-pull-secret-syncer-ncfzj\" (UID: \"029f4ecb-eb6b-4693-9e9e-ca61f554dd33\") " pod="kube-system/global-pull-secret-syncer-ncfzj" Oct 30 16:20:07.831800 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:07.831740 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 16:20:07.831855 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:07.831801 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/029f4ecb-eb6b-4693-9e9e-ca61f554dd33-original-pull-secret podName:029f4ecb-eb6b-4693-9e9e-ca61f554dd33 nodeName:}" failed. No retries permitted until 2026-10-30 16:20:08.831782345 +0000 UTC m=+20.132070566 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/029f4ecb-eb6b-4693-9e9e-ca61f554dd33-original-pull-secret") pod "global-pull-secret-syncer-ncfzj" (UID: "029f4ecb-eb6b-4693-9e9e-ca61f554dd33") : object "kube-system"/"original-pull-secret" not registered Oct 30 16:20:07.832150 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:07.832130 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 16:20:07.832218 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:07.832188 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f3d36d6-5098-43cc-8f8a-bd83d5dea5fc-metrics-certs podName:6f3d36d6-5098-43cc-8f8a-bd83d5dea5fc nodeName:}" failed. No retries permitted until 2026-10-30 16:20:08.832171567 +0000 UTC m=+20.132459781 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6f3d36d6-5098-43cc-8f8a-bd83d5dea5fc-metrics-certs") pod "network-metrics-daemon-xj6cp" (UID: "6f3d36d6-5098-43cc-8f8a-bd83d5dea5fc") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 16:20:07.932388 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:07.932349 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljv94\" (UniqueName: \"kubernetes.io/projected/2d1ed7d8-37cc-4652-b225-3a715b755e48-kube-api-access-ljv94\") pod \"network-check-target-b4ljl\" (UID: \"2d1ed7d8-37cc-4652-b225-3a715b755e48\") " pod="openshift-network-diagnostics/network-check-target-b4ljl" Oct 30 16:20:07.932568 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:07.932520 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 16:20:07.932568 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:07.932540 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 16:20:07.932568 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:07.932553 2580 projected.go:196] Error preparing data for projected volume kube-api-access-ljv94 for pod openshift-network-diagnostics/network-check-target-b4ljl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 16:20:07.932729 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:07.932627 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2d1ed7d8-37cc-4652-b225-3a715b755e48-kube-api-access-ljv94 podName:2d1ed7d8-37cc-4652-b225-3a715b755e48 nodeName:}" failed. No retries permitted until 2026-10-30 16:20:08.932607038 +0000 UTC m=+20.232895259 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-ljv94" (UniqueName: "kubernetes.io/projected/2d1ed7d8-37cc-4652-b225-3a715b755e48-kube-api-access-ljv94") pod "network-check-target-b4ljl" (UID: "2d1ed7d8-37cc-4652-b225-3a715b755e48") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 16:20:08.096642 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:08.096532 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 16:15:07 +0000 UTC" deadline="2028-05-31 06:52:13.321331134 +0000 UTC" Oct 30 16:20:08.096642 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:08.096596 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13886h32m5.224739178s" Oct 30 16:20:08.200987 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:08.200941 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2l5ht" event={"ID":"d10078a0-c21c-4f18-9c5a-4915fb7c8af3","Type":"ContainerStarted","Data":"2e25f0b762d57dbd49faf26aa31b5c096227a38a4195247540dada265b1c1c50"} Oct 30 16:20:08.205093 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:08.205059 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x9ddt" event={"ID":"9ea50669-2c08-403d-8d08-f9ffffa5b896","Type":"ContainerStarted","Data":"4e2f429b17f5c625d3707f009d7076d2a08fe8c8c4bfe2c2e4b0d03d3d36c00f"} Oct 30 16:20:08.211824 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:08.211782 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g69ll" event={"ID":"a6b95002-74f1-475d-a64a-6495733cf142","Type":"ContainerStarted","Data":"3a0fe4e644ac14b6dbc705911600a33fa60f54ec76fdc1c3cb9a789c664f0eca"} Oct 30 16:20:08.218006 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:08.217974 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-c8x4k" event={"ID":"749b2b57-f20e-4cdb-8d89-d2bac352f751","Type":"ContainerStarted","Data":"8f3c56e9db0a1f1b641e5ba13c8569ba3635c4f16b2329d31e44aa09a0bb0d34"} Oct 30 16:20:08.230256 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:08.230225 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7f2mt" event={"ID":"33b888e2-d349-439b-9aa1-5b519e0e08f7","Type":"ContainerStarted","Data":"c5f4d46d9d932ab42101bc160c22e1b28cc23fed7c0d7fccf9a8c2bcd51e905e"} Oct 30 16:20:08.242264 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:08.242234 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-sg9tg" event={"ID":"4c515c55-0717-4974-9f1a-68e0a25984cd","Type":"ContainerStarted","Data":"8dbde36737364320b9db4f9b6b3ec9cb036413547443ff1f770d9244b3824331"} Oct 30 16:20:08.254600 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:08.250614 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-131-136.ec2.internal_9ce644f9016009147990b020805ff06d/setup/0.log" Oct 30 16:20:08.254600 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:08.250653 2580 generic.go:356] "Generic (PLEG): container finished" podID="9ce644f9016009147990b020805ff06d" containerID="e62b2d0f673552954712774014cc22bed99f8d3f3e61d4488511b83f0aeffd54" exitCode=0 Oct 30 16:20:08.254600 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:08.250718 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-136.ec2.internal" event={"ID":"9ce644f9016009147990b020805ff06d","Type":"ContainerDied","Data":"e62b2d0f673552954712774014cc22bed99f8d3f3e61d4488511b83f0aeffd54"} Oct 30 16:20:08.254600 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:08.250759 2580 scope.go:122] "RemoveContainer" containerID="9bcc072de12fbce6c0c507299ed0e1da4cec7250e7543c41308f4a85176f3179" Oct 30 16:20:08.254600 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:08.251161 2580 scope.go:122] "RemoveContainer" containerID="9bcc072de12fbce6c0c507299ed0e1da4cec7250e7543c41308f4a85176f3179" Oct 30 16:20:08.259051 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:08.259028 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" event={"ID":"f2de1ce0-1c1e-4276-80a9-66239e13e352","Type":"ContainerStarted","Data":"41f89ab13b121f44360e1a4bbeef89b6d52d3402131286344cb4b94f8811e2f4"} Oct 30 16:20:08.260635 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:08.260605 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-d9kr8" event={"ID":"25c51397-40b9-432c-949d-12ba9f36532a","Type":"ContainerStarted","Data":"1de6e79aacb2bda8e4dae807b79eb5116dc5be949804852582c2f1f5105e4e70"} Oct 30 16:20:08.269230 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:08.269195 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-lghnm" event={"ID":"9ec1217e-8623-42a6-a052-adee153936a2","Type":"ContainerStarted","Data":"20e0e4f0bb761cfba7ee62e5fb09dd6dace934d281fb100229b83952b32fba9a"} Oct 30 16:20:08.302927 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:08.302892 2580 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_setup_kube-rbac-proxy-crio-ip-10-0-131-136.ec2.internal_openshift-machine-config-operator_9ce644f9016009147990b020805ff06d_0 in pod sandbox 622b6489c48a29a9d5eae0856d7d353098c922549fe22891407212db1d417313 from index: no such id: '9bcc072de12fbce6c0c507299ed0e1da4cec7250e7543c41308f4a85176f3179'" containerID="9bcc072de12fbce6c0c507299ed0e1da4cec7250e7543c41308f4a85176f3179" Oct 30 16:20:08.303024 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:08.302957 2580 kuberuntime_container.go:987] "Unhandled Error" err="failed to remove pod init container \"setup\": rpc error: code = Unknown desc = failed to delete container k8s_setup_kube-rbac-proxy-crio-ip-10-0-131-136.ec2.internal_openshift-machine-config-operator_9ce644f9016009147990b020805ff06d_0 in pod sandbox 622b6489c48a29a9d5eae0856d7d353098c922549fe22891407212db1d417313 from index: no such id: '9bcc072de12fbce6c0c507299ed0e1da4cec7250e7543c41308f4a85176f3179'; Skipping pod \"kube-rbac-proxy-crio-ip-10-0-131-136.ec2.internal_openshift-machine-config-operator(9ce644f9016009147990b020805ff06d)\"" logger="UnhandledError" Oct 30 16:20:08.838962 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:08.838883 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f3d36d6-5098-43cc-8f8a-bd83d5dea5fc-metrics-certs\") pod \"network-metrics-daemon-xj6cp\" (UID: \"6f3d36d6-5098-43cc-8f8a-bd83d5dea5fc\") " pod="openshift-multus/network-metrics-daemon-xj6cp" Oct 30 16:20:08.838962 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:08.838942 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/029f4ecb-eb6b-4693-9e9e-ca61f554dd33-original-pull-secret\") pod \"global-pull-secret-syncer-ncfzj\" (UID: \"029f4ecb-eb6b-4693-9e9e-ca61f554dd33\") " pod="kube-system/global-pull-secret-syncer-ncfzj" Oct 30 16:20:08.839481 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:08.839028 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 16:20:08.839481 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:08.839090 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/029f4ecb-eb6b-4693-9e9e-ca61f554dd33-original-pull-secret podName:029f4ecb-eb6b-4693-9e9e-ca61f554dd33 nodeName:}" failed. No retries permitted until 2026-10-30 16:20:10.839073889 +0000 UTC m=+22.139362089 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/029f4ecb-eb6b-4693-9e9e-ca61f554dd33-original-pull-secret") pod "global-pull-secret-syncer-ncfzj" (UID: "029f4ecb-eb6b-4693-9e9e-ca61f554dd33") : object "kube-system"/"original-pull-secret" not registered Oct 30 16:20:08.839481 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:08.839445 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 16:20:08.839670 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:08.839495 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f3d36d6-5098-43cc-8f8a-bd83d5dea5fc-metrics-certs podName:6f3d36d6-5098-43cc-8f8a-bd83d5dea5fc nodeName:}" failed. No retries permitted until 2026-10-30 16:20:10.839479518 +0000 UTC m=+22.139767721 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6f3d36d6-5098-43cc-8f8a-bd83d5dea5fc-metrics-certs") pod "network-metrics-daemon-xj6cp" (UID: "6f3d36d6-5098-43cc-8f8a-bd83d5dea5fc") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 16:20:08.941914 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:08.940989 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljv94\" (UniqueName: \"kubernetes.io/projected/2d1ed7d8-37cc-4652-b225-3a715b755e48-kube-api-access-ljv94\") pod \"network-check-target-b4ljl\" (UID: \"2d1ed7d8-37cc-4652-b225-3a715b755e48\") " pod="openshift-network-diagnostics/network-check-target-b4ljl" Oct 30 16:20:08.941914 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:08.941323 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 16:20:08.941914 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:08.941367 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 16:20:08.941914 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:08.941380 2580 projected.go:196] Error preparing data for projected volume kube-api-access-ljv94 for pod openshift-network-diagnostics/network-check-target-b4ljl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 16:20:08.941914 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:08.941441 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2d1ed7d8-37cc-4652-b225-3a715b755e48-kube-api-access-ljv94 podName:2d1ed7d8-37cc-4652-b225-3a715b755e48 nodeName:}" failed. No retries permitted until 2026-10-30 16:20:10.941422204 +0000 UTC m=+22.241710407 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-ljv94" (UniqueName: "kubernetes.io/projected/2d1ed7d8-37cc-4652-b225-3a715b755e48-kube-api-access-ljv94") pod "network-check-target-b4ljl" (UID: "2d1ed7d8-37cc-4652-b225-3a715b755e48") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 16:20:09.096813 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:09.096693 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 16:15:07 +0000 UTC" deadline="2028-05-16 21:55:10.391309305 +0000 UTC" Oct 30 16:20:09.096813 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:09.096738 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13541h35m1.294575148s" Oct 30 16:20:09.161519 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:09.161491 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xj6cp" Oct 30 16:20:09.161692 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:09.161640 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xj6cp" podUID="6f3d36d6-5098-43cc-8f8a-bd83d5dea5fc" Oct 30 16:20:09.161776 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:09.161711 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ncfzj" Oct 30 16:20:09.161833 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:09.161812 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ncfzj" podUID="029f4ecb-eb6b-4693-9e9e-ca61f554dd33" Oct 30 16:20:09.161940 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:09.161895 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b4ljl" Oct 30 16:20:09.161989 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:09.161967 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b4ljl" podUID="2d1ed7d8-37cc-4652-b225-3a715b755e48" Oct 30 16:20:09.293162 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:09.293114 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-136.ec2.internal" event={"ID":"9ce644f9016009147990b020805ff06d","Type":"ContainerStarted","Data":"b0f31a88ac74561bf5fb58c82d0cbdad3966b5c838f996c996dde6b60acabcc5"} Oct 30 16:20:10.309499 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:10.309267 2580 generic.go:356] "Generic (PLEG): container finished" podID="9ea50669-2c08-403d-8d08-f9ffffa5b896" containerID="400482828bf07e07d191072e688d0b20b11a611875a527e690e216b28be6b46f" exitCode=0 Oct 30 16:20:10.309499 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:10.309464 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x9ddt" event={"ID":"9ea50669-2c08-403d-8d08-f9ffffa5b896","Type":"ContainerDied","Data":"400482828bf07e07d191072e688d0b20b11a611875a527e690e216b28be6b46f"} Oct 30 16:20:10.376558 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:10.376499 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-136.ec2.internal" podStartSLOduration=4.376481544 podStartE2EDuration="4.376481544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 16:20:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:20:09.331791188 +0000 UTC m=+20.632079423" watchObservedRunningTime="2026-10-30 16:20:10.376481544 +0000 UTC m=+21.676769766" Oct 30 16:20:10.857771 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:10.856994 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f3d36d6-5098-43cc-8f8a-bd83d5dea5fc-metrics-certs\") pod \"network-metrics-daemon-xj6cp\" (UID: \"6f3d36d6-5098-43cc-8f8a-bd83d5dea5fc\") " pod="openshift-multus/network-metrics-daemon-xj6cp" Oct 30 16:20:10.857771 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:10.857066 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/029f4ecb-eb6b-4693-9e9e-ca61f554dd33-original-pull-secret\") pod \"global-pull-secret-syncer-ncfzj\" (UID: \"029f4ecb-eb6b-4693-9e9e-ca61f554dd33\") " pod="kube-system/global-pull-secret-syncer-ncfzj" Oct 30 16:20:10.857771 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:10.857197 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 16:20:10.857771 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:10.857252 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/029f4ecb-eb6b-4693-9e9e-ca61f554dd33-original-pull-secret podName:029f4ecb-eb6b-4693-9e9e-ca61f554dd33 nodeName:}" failed. No retries permitted until 2026-10-30 16:20:14.857234311 +0000 UTC m=+26.157522521 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/029f4ecb-eb6b-4693-9e9e-ca61f554dd33-original-pull-secret") pod "global-pull-secret-syncer-ncfzj" (UID: "029f4ecb-eb6b-4693-9e9e-ca61f554dd33") : object "kube-system"/"original-pull-secret" not registered Oct 30 16:20:10.857771 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:10.857675 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 16:20:10.857771 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:10.857727 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f3d36d6-5098-43cc-8f8a-bd83d5dea5fc-metrics-certs podName:6f3d36d6-5098-43cc-8f8a-bd83d5dea5fc nodeName:}" failed. No retries permitted until 2026-10-30 16:20:14.857711321 +0000 UTC m=+26.157999525 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6f3d36d6-5098-43cc-8f8a-bd83d5dea5fc-metrics-certs") pod "network-metrics-daemon-xj6cp" (UID: "6f3d36d6-5098-43cc-8f8a-bd83d5dea5fc") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 16:20:10.958056 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:10.957411 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljv94\" (UniqueName: \"kubernetes.io/projected/2d1ed7d8-37cc-4652-b225-3a715b755e48-kube-api-access-ljv94\") pod \"network-check-target-b4ljl\" (UID: \"2d1ed7d8-37cc-4652-b225-3a715b755e48\") " pod="openshift-network-diagnostics/network-check-target-b4ljl" Oct 30 16:20:10.958056 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:10.957628 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 16:20:10.958056 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:10.957652 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 16:20:10.958056 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:10.957666 2580 projected.go:196] Error preparing data for projected volume kube-api-access-ljv94 for pod openshift-network-diagnostics/network-check-target-b4ljl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 16:20:10.958056 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:10.957721 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2d1ed7d8-37cc-4652-b225-3a715b755e48-kube-api-access-ljv94 podName:2d1ed7d8-37cc-4652-b225-3a715b755e48 nodeName:}" failed. No retries permitted until 2026-10-30 16:20:14.957702772 +0000 UTC m=+26.257990971 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-ljv94" (UniqueName: "kubernetes.io/projected/2d1ed7d8-37cc-4652-b225-3a715b755e48-kube-api-access-ljv94") pod "network-check-target-b4ljl" (UID: "2d1ed7d8-37cc-4652-b225-3a715b755e48") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 16:20:11.161046 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:11.160973 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xj6cp" Oct 30 16:20:11.161209 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:11.161117 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xj6cp" podUID="6f3d36d6-5098-43cc-8f8a-bd83d5dea5fc" Oct 30 16:20:11.161517 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:11.161496 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ncfzj" Oct 30 16:20:11.161646 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:11.161610 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ncfzj" podUID="029f4ecb-eb6b-4693-9e9e-ca61f554dd33" Oct 30 16:20:11.161708 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:11.161669 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b4ljl" Oct 30 16:20:11.161756 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:11.161735 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b4ljl" podUID="2d1ed7d8-37cc-4652-b225-3a715b755e48" Oct 30 16:20:11.318854 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:11.318777 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-sg9tg" event={"ID":"4c515c55-0717-4974-9f1a-68e0a25984cd","Type":"ContainerStarted","Data":"377c8eb2f779ff7def063735c34e50407e1b9f681d57655db6a977c52584268b"} Oct 30 16:20:11.356679 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:11.356630 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-sg9tg" podStartSLOduration=5.94721619 podStartE2EDuration="9.356615901s" totalImagesPullingTime="3.409399711s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:20:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:20:11.35586971 +0000 UTC m=+22.656157928" watchObservedRunningTime="2026-10-30 16:20:11.356615901 +0000 UTC m=+22.656904119" Oct 30 16:20:12.324916 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:12.324864 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2l5ht" event={"ID":"d10078a0-c21c-4f18-9c5a-4915fb7c8af3","Type":"ContainerStarted","Data":"751d2b54a048a1321d1de23ccd46e077ac92c98f25139a07040a9f8517c93770"} Oct 30 16:20:12.378129 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:12.378088 2580 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-sg9tg" Oct 30 16:20:12.378634 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:12.378616 2580 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-sg9tg" Oct 30 16:20:12.415950 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:12.415896 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-2l5ht" podStartSLOduration=6.030849826 podStartE2EDuration="10.415881689s" totalImagesPullingTime="4.385031863s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 16:20:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:20:12.36222091 +0000 UTC m=+23.662509129" watchObservedRunningTime="2026-10-30 16:20:12.415881689 +0000 UTC m=+23.716169908" Oct 30 16:20:13.170075 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:13.168709 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ncfzj" Oct 30 16:20:13.170075 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:13.168889 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ncfzj" podUID="029f4ecb-eb6b-4693-9e9e-ca61f554dd33" Oct 30 16:20:13.170075 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:13.169422 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xj6cp" Oct 30 16:20:13.170075 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:13.169597 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xj6cp" podUID="6f3d36d6-5098-43cc-8f8a-bd83d5dea5fc" Oct 30 16:20:13.170075 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:13.169676 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b4ljl" Oct 30 16:20:13.170075 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:13.169776 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b4ljl" podUID="2d1ed7d8-37cc-4652-b225-3a715b755e48" Oct 30 16:20:13.328920 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:13.328887 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g69ll" event={"ID":"a6b95002-74f1-475d-a64a-6495733cf142","Type":"ContainerStarted","Data":"e6c3a95b2cc036bccd6fa760859e05cdaea2675886ceec92cd656ed54ceddad4"} Oct 30 16:20:14.331764 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:14.331735 2580 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 16:20:14.641605 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:14.641534 2580 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 16:20:14.896924 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:14.896617 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f3d36d6-5098-43cc-8f8a-bd83d5dea5fc-metrics-certs\") pod \"network-metrics-daemon-xj6cp\" (UID: \"6f3d36d6-5098-43cc-8f8a-bd83d5dea5fc\") " pod="openshift-multus/network-metrics-daemon-xj6cp" Oct 30 16:20:14.896924 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:14.896686 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/029f4ecb-eb6b-4693-9e9e-ca61f554dd33-original-pull-secret\") pod \"global-pull-secret-syncer-ncfzj\" (UID: \"029f4ecb-eb6b-4693-9e9e-ca61f554dd33\") " pod="kube-system/global-pull-secret-syncer-ncfzj" Oct 30 16:20:14.896924 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:14.896808 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 16:20:14.896924 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:14.896856 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/029f4ecb-eb6b-4693-9e9e-ca61f554dd33-original-pull-secret podName:029f4ecb-eb6b-4693-9e9e-ca61f554dd33 nodeName:}" failed. No retries permitted until 2026-10-30 16:20:22.896840452 +0000 UTC m=+34.197128647 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/029f4ecb-eb6b-4693-9e9e-ca61f554dd33-original-pull-secret") pod "global-pull-secret-syncer-ncfzj" (UID: "029f4ecb-eb6b-4693-9e9e-ca61f554dd33") : object "kube-system"/"original-pull-secret" not registered Oct 30 16:20:14.897249 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:14.897228 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 16:20:14.897296 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:14.897273 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f3d36d6-5098-43cc-8f8a-bd83d5dea5fc-metrics-certs podName:6f3d36d6-5098-43cc-8f8a-bd83d5dea5fc nodeName:}" failed. No retries permitted until 2026-10-30 16:20:22.897258445 +0000 UTC m=+34.197546645 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6f3d36d6-5098-43cc-8f8a-bd83d5dea5fc-metrics-certs") pod "network-metrics-daemon-xj6cp" (UID: "6f3d36d6-5098-43cc-8f8a-bd83d5dea5fc") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 16:20:14.997158 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:14.997133 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljv94\" (UniqueName: \"kubernetes.io/projected/2d1ed7d8-37cc-4652-b225-3a715b755e48-kube-api-access-ljv94\") pod \"network-check-target-b4ljl\" (UID: \"2d1ed7d8-37cc-4652-b225-3a715b755e48\") " pod="openshift-network-diagnostics/network-check-target-b4ljl" Oct 30 16:20:14.997279 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:14.997265 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 16:20:14.997320 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:14.997289 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 16:20:14.997320 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:14.997302 2580 projected.go:196] Error preparing data for projected volume kube-api-access-ljv94 for pod openshift-network-diagnostics/network-check-target-b4ljl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 16:20:14.997414 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:14.997369 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2d1ed7d8-37cc-4652-b225-3a715b755e48-kube-api-access-ljv94 podName:2d1ed7d8-37cc-4652-b225-3a715b755e48 nodeName:}" failed. No retries permitted until 2026-10-30 16:20:22.997351581 +0000 UTC m=+34.297639781 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-ljv94" (UniqueName: "kubernetes.io/projected/2d1ed7d8-37cc-4652-b225-3a715b755e48-kube-api-access-ljv94") pod "network-check-target-b4ljl" (UID: "2d1ed7d8-37cc-4652-b225-3a715b755e48") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 16:20:15.098282 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:15.098167 2580 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T16:20:14.641562896Z","UUID":"0eabb056-cd5a-4c47-8d67-6f98360498e6","Handler":null,"Name":"","Endpoint":""} Oct 30 16:20:15.101965 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:15.101941 2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 16:20:15.102096 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:15.101984 2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 16:20:15.162045 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:15.161973 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ncfzj" Oct 30 16:20:15.162171 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:15.162096 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ncfzj" podUID="029f4ecb-eb6b-4693-9e9e-ca61f554dd33" Oct 30 16:20:15.162171 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:15.162162 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b4ljl" Oct 30 16:20:15.162250 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:15.162225 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b4ljl" podUID="2d1ed7d8-37cc-4652-b225-3a715b755e48" Oct 30 16:20:15.162283 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:15.162259 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xj6cp" Oct 30 16:20:15.162340 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:15.162310 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xj6cp" podUID="6f3d36d6-5098-43cc-8f8a-bd83d5dea5fc" Oct 30 16:20:15.335798 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:15.335768 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-lghnm" event={"ID":"9ec1217e-8623-42a6-a052-adee153936a2","Type":"ContainerStarted","Data":"f3b55b8b8eb9620255a166550d947a03833cb061f34b208eacb06f092883efda"} Oct 30 16:20:15.338556 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:15.338009 2580 generic.go:356] "Generic (PLEG): container finished" podID="9ea50669-2c08-403d-8d08-f9ffffa5b896" containerID="0b0a760f70d4716ef6980b4da47d24ffb414f98d4954fe814e8b4264a8ebe822" exitCode=0 Oct 30 16:20:15.338556 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:15.338086 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x9ddt" event={"ID":"9ea50669-2c08-403d-8d08-f9ffffa5b896","Type":"ContainerDied","Data":"0b0a760f70d4716ef6980b4da47d24ffb414f98d4954fe814e8b4264a8ebe822"} Oct 30 16:20:15.341688 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:15.341316 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g69ll" event={"ID":"a6b95002-74f1-475d-a64a-6495733cf142","Type":"ContainerStarted","Data":"74967749470c05cd27ff688e9feacbe2dfbfcd1d749fa403cb1b32156f0f68ae"} Oct 30 16:20:15.343910 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:15.343885 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7f2mt" event={"ID":"33b888e2-d349-439b-9aa1-5b519e0e08f7","Type":"ContainerStarted","Data":"8fc4c83f1191203ec70c2bf15f68d66caa76754be3bcab8f41e7aa53150799b4"} Oct 30 16:20:15.373603 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:15.373114 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-lghnm" podStartSLOduration=5.811403258 podStartE2EDuration="13.373098666s" totalImagesPullingTime="7.561695408s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 16:20:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:20:15.372516576 +0000 UTC m=+26.672804795" watchObservedRunningTime="2026-10-30 16:20:15.373098666 +0000 UTC m=+26.673386886" Oct 30 16:20:16.348789 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:16.348458 2580 generic.go:356] "Generic (PLEG): container finished" podID="9ea50669-2c08-403d-8d08-f9ffffa5b896" containerID="4cc1581bac68fafdd21e5a05433eb540aadd5de0eb78eff5e9c58a771b98c2b9" exitCode=0 Oct 30 16:20:16.348789 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:16.348624 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x9ddt" event={"ID":"9ea50669-2c08-403d-8d08-f9ffffa5b896","Type":"ContainerDied","Data":"4cc1581bac68fafdd21e5a05433eb540aadd5de0eb78eff5e9c58a771b98c2b9"} Oct 30 16:20:16.351360 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:16.351285 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g69ll" event={"ID":"a6b95002-74f1-475d-a64a-6495733cf142","Type":"ContainerStarted","Data":"1312c3b5a249b760b85689ec9a0a3b8aa34fed88e86e3f46d5989e6207a638f4"} Oct 30 16:20:16.389966 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:16.389916 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-7f2mt" podStartSLOduration=7.098686086 podStartE2EDuration="14.389901548s" totalImagesPullingTime="7.291215462s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 16:20:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:20:15.455151556 +0000 UTC m=+26.755439791" watchObservedRunningTime="2026-10-30 16:20:16.389901548 +0000 UTC m=+27.690189781" Oct 30 16:20:16.430300 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:16.430252 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g69ll" podStartSLOduration=6.646329769 podStartE2EDuration="14.430238328s" totalImagesPullingTime="7.783908559s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 16:20:02 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:20:16.429837298 +0000 UTC m=+27.730125517" watchObservedRunningTime="2026-10-30 16:20:16.430238328 +0000 UTC m=+27.730526542" Oct 30 16:20:17.159071 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:17.159042 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ncfzj" Oct 30 16:20:17.159373 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:17.159075 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b4ljl" Oct 30 16:20:17.159373 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:17.159044 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xj6cp" Oct 30 16:20:17.159373 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:17.159159 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ncfzj" podUID="029f4ecb-eb6b-4693-9e9e-ca61f554dd33" Oct 30 16:20:17.159373 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:17.159235 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b4ljl" podUID="2d1ed7d8-37cc-4652-b225-3a715b755e48" Oct 30 16:20:17.159373 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:17.159301 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xj6cp" podUID="6f3d36d6-5098-43cc-8f8a-bd83d5dea5fc" Oct 30 16:20:17.354938 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:17.354902 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-d9kr8" event={"ID":"25c51397-40b9-432c-949d-12ba9f36532a","Type":"ContainerStarted","Data":"c7641b824c8d585b0c581219b862a1fc6448d20ee5f703e4d17bfaf5722d478c"} Oct 30 16:20:17.357608 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:17.357565 2580 generic.go:356] "Generic (PLEG): container finished" podID="9ea50669-2c08-403d-8d08-f9ffffa5b896" containerID="8737afe289702ed6701d13e3de20816e1549986e2c852c5f88ab20a38c49f52e" exitCode=0 Oct 30 16:20:17.357740 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:17.357650 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x9ddt" event={"ID":"9ea50669-2c08-403d-8d08-f9ffffa5b896","Type":"ContainerDied","Data":"8737afe289702ed6701d13e3de20816e1549986e2c852c5f88ab20a38c49f52e"} Oct 30 16:20:17.359538 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:17.359458 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-c8x4k" event={"ID":"749b2b57-f20e-4cdb-8d89-d2bac352f751","Type":"ContainerStarted","Data":"bafd72fb133876cc286a017ed810a109e1bb9fe83fb67b87952acad9661c488b"} Oct 30 16:20:17.392103 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:17.392059 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-d9kr8" podStartSLOduration=7.523783923 podStartE2EDuration="15.39204396s" totalImagesPullingTime="7.868260037s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 16:20:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:20:17.391766264 +0000 UTC m=+28.692054493" watchObservedRunningTime="2026-10-30 16:20:17.39204396 +0000 UTC m=+28.692332179" Oct 30 16:20:17.472157 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:17.472074 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-c8x4k" podStartSLOduration=6.028905728 podStartE2EDuration="15.472058248s" totalImagesPullingTime="9.44315252s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 16:20:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:20:17.471860444 +0000 UTC m=+28.772148663" watchObservedRunningTime="2026-10-30 16:20:17.472058248 +0000 UTC m=+28.772346465" Oct 30 16:20:19.162541 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:19.162505 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ncfzj" Oct 30 16:20:19.162541 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:19.162543 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b4ljl" Oct 30 16:20:19.163036 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:19.162682 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ncfzj" podUID="029f4ecb-eb6b-4693-9e9e-ca61f554dd33" Oct 30 16:20:19.163036 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:19.162732 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xj6cp" Oct 30 16:20:19.163036 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:19.162811 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b4ljl" podUID="2d1ed7d8-37cc-4652-b225-3a715b755e48" Oct 30 16:20:19.163036 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:19.162892 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xj6cp" podUID="6f3d36d6-5098-43cc-8f8a-bd83d5dea5fc" Oct 30 16:20:19.366770 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:19.366724 2580 generic.go:356] "Generic (PLEG): container finished" podID="9ea50669-2c08-403d-8d08-f9ffffa5b896" containerID="ea7603f6eb8322434165482e2da9cee993244aba808b6258fb700f4ecfe0cb45" exitCode=0 Oct 30 16:20:19.366935 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:19.366787 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x9ddt" event={"ID":"9ea50669-2c08-403d-8d08-f9ffffa5b896","Type":"ContainerDied","Data":"ea7603f6eb8322434165482e2da9cee993244aba808b6258fb700f4ecfe0cb45"} Oct 30 16:20:20.372145 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:20.372112 2580 generic.go:356] "Generic (PLEG): container finished" podID="9ea50669-2c08-403d-8d08-f9ffffa5b896" containerID="6b9a56d52e98361c9aef362ebf341a8dbf87b064a9aa3c93539874bdc1c1c792" exitCode=0 Oct 30 16:20:20.372650 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:20.372173 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x9ddt" event={"ID":"9ea50669-2c08-403d-8d08-f9ffffa5b896","Type":"ContainerDied","Data":"6b9a56d52e98361c9aef362ebf341a8dbf87b064a9aa3c93539874bdc1c1c792"} Oct 30 16:20:21.158204 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:21.158173 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b4ljl" Oct 30 16:20:21.158204 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:21.158209 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xj6cp" Oct 30 16:20:21.158427 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:21.158174 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ncfzj" Oct 30 16:20:21.158427 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:21.158289 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b4ljl" podUID="2d1ed7d8-37cc-4652-b225-3a715b755e48" Oct 30 16:20:21.158427 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:21.158358 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ncfzj" podUID="029f4ecb-eb6b-4693-9e9e-ca61f554dd33" Oct 30 16:20:21.158427 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:21.158423 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xj6cp" podUID="6f3d36d6-5098-43cc-8f8a-bd83d5dea5fc" Oct 30 16:20:21.377678 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:21.377463 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x9ddt" event={"ID":"9ea50669-2c08-403d-8d08-f9ffffa5b896","Type":"ContainerStarted","Data":"d33fc060cfb53aa6a1000e07d667eabf40f9fc5e2b15ad32dad3a64d7783a310"} Oct 30 16:20:21.379791 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:21.379770 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" event={"ID":"f2de1ce0-1c1e-4276-80a9-66239e13e352","Type":"ContainerStarted","Data":"d22d8d6ba7334a439ef020847e96d6a23d3d1fe79547de2891327c908d9af1da"} Oct 30 16:20:21.379859 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:21.379828 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" event={"ID":"f2de1ce0-1c1e-4276-80a9-66239e13e352","Type":"ContainerStarted","Data":"a74a37ff9d5430fd665d37d45d6b0a045ff303a2306bc91fcaa380cee79d3578"} Oct 30 16:20:21.379859 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:21.379857 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" event={"ID":"f2de1ce0-1c1e-4276-80a9-66239e13e352","Type":"ContainerStarted","Data":"e8b4ca667deec712ea9faca744d5d73b08ffe79c5a3d287d3542e26b3a398f58"} Oct 30 16:20:21.379932 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:21.379884 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" event={"ID":"f2de1ce0-1c1e-4276-80a9-66239e13e352","Type":"ContainerStarted","Data":"01fb684f99e043eaa5f489710e6cc3f5c746fc4418a5b0fcc4fb0b523117d005"} Oct 30 16:20:21.436525 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:21.436479 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-x9ddt" podStartSLOduration=9.870879781 podStartE2EDuration="19.436466786s" totalImagesPullingTime="9.565587005s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 16:20:02 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:20:21.431958728 +0000 UTC m=+32.732246946" watchObservedRunningTime="2026-10-30 16:20:21.436466786 +0000 UTC m=+32.736755004" Oct 30 16:20:21.502630 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:21.502609 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-sg9tg" Oct 30 16:20:21.502761 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:21.502720 2580 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 16:20:21.503128 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:21.503103 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-sg9tg" Oct 30 16:20:22.385034 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:22.385001 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" event={"ID":"f2de1ce0-1c1e-4276-80a9-66239e13e352","Type":"ContainerStarted","Data":"745c6874ce1784b0e76b10d4eae664370596c77b1661213847105aed62faf5e1"} Oct 30 16:20:22.385485 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:22.385072 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" event={"ID":"f2de1ce0-1c1e-4276-80a9-66239e13e352","Type":"ContainerStarted","Data":"f8870f26c42c448dcf31c0478ba5e95a3ee2752552b3b88bc00b65d697705ebf"} Oct 30 16:20:22.954942 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:22.954912 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f3d36d6-5098-43cc-8f8a-bd83d5dea5fc-metrics-certs\") pod \"network-metrics-daemon-xj6cp\" (UID: \"6f3d36d6-5098-43cc-8f8a-bd83d5dea5fc\") " pod="openshift-multus/network-metrics-daemon-xj6cp" Oct 30 16:20:22.955114 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:22.954957 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/029f4ecb-eb6b-4693-9e9e-ca61f554dd33-original-pull-secret\") pod \"global-pull-secret-syncer-ncfzj\" (UID: \"029f4ecb-eb6b-4693-9e9e-ca61f554dd33\") " pod="kube-system/global-pull-secret-syncer-ncfzj" Oct 30 16:20:22.955114 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:22.955051 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 16:20:22.955114 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:22.955097 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 16:20:22.955114 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:22.955105 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f3d36d6-5098-43cc-8f8a-bd83d5dea5fc-metrics-certs podName:6f3d36d6-5098-43cc-8f8a-bd83d5dea5fc nodeName:}" failed. No retries permitted until 2026-10-30 16:20:38.955091391 +0000 UTC m=+50.255379599 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6f3d36d6-5098-43cc-8f8a-bd83d5dea5fc-metrics-certs") pod "network-metrics-daemon-xj6cp" (UID: "6f3d36d6-5098-43cc-8f8a-bd83d5dea5fc") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 16:20:22.955304 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:22.955133 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/029f4ecb-eb6b-4693-9e9e-ca61f554dd33-original-pull-secret podName:029f4ecb-eb6b-4693-9e9e-ca61f554dd33 nodeName:}" failed. No retries permitted until 2026-10-30 16:20:38.955122469 +0000 UTC m=+50.255410665 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/029f4ecb-eb6b-4693-9e9e-ca61f554dd33-original-pull-secret") pod "global-pull-secret-syncer-ncfzj" (UID: "029f4ecb-eb6b-4693-9e9e-ca61f554dd33") : object "kube-system"/"original-pull-secret" not registered Oct 30 16:20:23.055468 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:23.055442 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljv94\" (UniqueName: \"kubernetes.io/projected/2d1ed7d8-37cc-4652-b225-3a715b755e48-kube-api-access-ljv94\") pod \"network-check-target-b4ljl\" (UID: \"2d1ed7d8-37cc-4652-b225-3a715b755e48\") " pod="openshift-network-diagnostics/network-check-target-b4ljl" Oct 30 16:20:23.055620 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:23.055549 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 16:20:23.055620 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:23.055561 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 16:20:23.055620 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:23.055572 2580 projected.go:196] Error preparing data for projected volume kube-api-access-ljv94 for pod openshift-network-diagnostics/network-check-target-b4ljl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 16:20:23.055712 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:23.055626 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2d1ed7d8-37cc-4652-b225-3a715b755e48-kube-api-access-ljv94 podName:2d1ed7d8-37cc-4652-b225-3a715b755e48 nodeName:}" failed. No retries permitted until 2026-10-30 16:20:39.055614867 +0000 UTC m=+50.355903076 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-ljv94" (UniqueName: "kubernetes.io/projected/2d1ed7d8-37cc-4652-b225-3a715b755e48-kube-api-access-ljv94") pod "network-check-target-b4ljl" (UID: "2d1ed7d8-37cc-4652-b225-3a715b755e48") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 16:20:23.158672 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:23.158651 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b4ljl" Oct 30 16:20:23.158882 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:23.158685 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ncfzj" Oct 30 16:20:23.158882 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:23.158739 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b4ljl" podUID="2d1ed7d8-37cc-4652-b225-3a715b755e48" Oct 30 16:20:23.158882 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:23.158807 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xj6cp" Oct 30 16:20:23.158993 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:23.158916 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xj6cp" podUID="6f3d36d6-5098-43cc-8f8a-bd83d5dea5fc" Oct 30 16:20:23.159029 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:23.158991 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ncfzj" podUID="029f4ecb-eb6b-4693-9e9e-ca61f554dd33" Oct 30 16:20:24.392859 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:24.392824 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" event={"ID":"f2de1ce0-1c1e-4276-80a9-66239e13e352","Type":"ContainerStarted","Data":"012a16ad70b063d87ff6ea733dd352eb135e11baa597c8a1ae69a2284ff30bb1"} Oct 30 16:20:25.158199 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:25.158169 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xj6cp" Oct 30 16:20:25.158199 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:25.158201 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b4ljl" Oct 30 16:20:25.158412 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:25.158304 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xj6cp" podUID="6f3d36d6-5098-43cc-8f8a-bd83d5dea5fc" Oct 30 16:20:25.158412 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:25.158351 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ncfzj" Oct 30 16:20:25.158516 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:25.158422 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b4ljl" podUID="2d1ed7d8-37cc-4652-b225-3a715b755e48" Oct 30 16:20:25.158516 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:25.158499 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ncfzj" podUID="029f4ecb-eb6b-4693-9e9e-ca61f554dd33" Oct 30 16:20:26.400830 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:26.400797 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" event={"ID":"f2de1ce0-1c1e-4276-80a9-66239e13e352","Type":"ContainerStarted","Data":"a94095ac9175aa22bd9860b327c64be1640d8f04ae70c39374d9eff4f8afddb1"} Oct 30 16:20:26.401291 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:26.401034 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:26.401291 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:26.401108 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:26.416535 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:26.416512 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:26.483315 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:26.483277 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" podStartSLOduration=10.735511317 podStartE2EDuration="24.483267676s" totalImagesPullingTime="13.747756359s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 16:20:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:20:26.482769719 +0000 UTC m=+37.783057973" watchObservedRunningTime="2026-10-30 16:20:26.483267676 +0000 UTC m=+37.783555893" Oct 30 16:20:27.158279 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:27.158241 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xj6cp" Oct 30 16:20:27.158279 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:27.158263 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b4ljl" Oct 30 16:20:27.158522 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:27.158361 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xj6cp" podUID="6f3d36d6-5098-43cc-8f8a-bd83d5dea5fc" Oct 30 16:20:27.158657 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:27.158628 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ncfzj" Oct 30 16:20:27.158793 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:27.158770 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b4ljl" podUID="2d1ed7d8-37cc-4652-b225-3a715b755e48" Oct 30 16:20:27.158889 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:27.158870 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ncfzj" podUID="029f4ecb-eb6b-4693-9e9e-ca61f554dd33" Oct 30 16:20:27.402984 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:27.402950 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:27.417298 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:27.417246 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:27.871109 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:27.871037 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ncfzj"] Oct 30 16:20:27.871314 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:27.871150 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ncfzj" Oct 30 16:20:27.871314 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:27.871261 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ncfzj" podUID="029f4ecb-eb6b-4693-9e9e-ca61f554dd33" Oct 30 16:20:27.877367 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:27.877340 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-b4ljl"] Oct 30 16:20:27.877479 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:27.877418 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b4ljl" Oct 30 16:20:27.877521 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:27.877487 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b4ljl" podUID="2d1ed7d8-37cc-4652-b225-3a715b755e48" Oct 30 16:20:27.877838 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:27.877820 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xj6cp"] Oct 30 16:20:27.877906 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:27.877895 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xj6cp" Oct 30 16:20:27.877980 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:27.877965 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xj6cp" podUID="6f3d36d6-5098-43cc-8f8a-bd83d5dea5fc" Oct 30 16:20:29.158968 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:29.158936 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ncfzj" Oct 30 16:20:29.159334 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:29.159099 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ncfzj" podUID="029f4ecb-eb6b-4693-9e9e-ca61f554dd33" Oct 30 16:20:29.159334 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:29.159155 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b4ljl" Oct 30 16:20:29.159334 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:29.159240 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b4ljl" podUID="2d1ed7d8-37cc-4652-b225-3a715b755e48" Oct 30 16:20:30.158766 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:30.158744 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xj6cp" Oct 30 16:20:30.158929 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:30.158872 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xj6cp" podUID="6f3d36d6-5098-43cc-8f8a-bd83d5dea5fc" Oct 30 16:20:31.158364 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:31.158334 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b4ljl" Oct 30 16:20:31.158833 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:31.158341 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ncfzj" Oct 30 16:20:31.158833 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:31.158430 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b4ljl" podUID="2d1ed7d8-37cc-4652-b225-3a715b755e48" Oct 30 16:20:31.158833 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:31.158512 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ncfzj" podUID="029f4ecb-eb6b-4693-9e9e-ca61f554dd33" Oct 30 16:20:32.158801 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:32.158774 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xj6cp" Oct 30 16:20:32.159264 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:32.158898 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xj6cp" podUID="6f3d36d6-5098-43cc-8f8a-bd83d5dea5fc" Oct 30 16:20:33.158463 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.158430 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b4ljl" Oct 30 16:20:33.158605 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:33.158564 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b4ljl" podUID="2d1ed7d8-37cc-4652-b225-3a715b755e48" Oct 30 16:20:33.158657 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.158608 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ncfzj" Oct 30 16:20:33.158702 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:33.158689 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ncfzj" podUID="029f4ecb-eb6b-4693-9e9e-ca61f554dd33" Oct 30 16:20:33.469323 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.466790 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-136.ec2.internal" event="NodeReady" Oct 30 16:20:33.469323 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.467118 2580 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 16:20:33.516573 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.516548 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-cpxbb"] Oct 30 16:20:33.534801 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.534777 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-4dfsk"] Oct 30 16:20:33.534995 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.534881 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-cpxbb" Oct 30 16:20:33.545144 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.544961 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-fhzbv"] Oct 30 16:20:33.545223 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.545060 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-4dfsk" Oct 30 16:20:33.549794 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.549779 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 30 16:20:33.549889 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.549821 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 30 16:20:33.558124 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.558083 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-p4rk6"] Oct 30 16:20:33.558223 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.558206 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-fhzbv" Oct 30 16:20:33.576128 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.575894 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-cswcz"] Oct 30 16:20:33.577887 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.576681 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-p4rk6" Oct 30 16:20:33.587683 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.587662 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-sfq4m"] Oct 30 16:20:33.587799 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.587783 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-cswcz" Oct 30 16:20:33.591878 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.591862 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 30 16:20:33.596542 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.596525 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-4nn7z"] Oct 30 16:20:33.596664 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.596650 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sfq4m" Oct 30 16:20:33.606441 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.606424 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-df85cd6ff-6pd8x"] Oct 30 16:20:33.606594 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.606557 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-4nn7z" Oct 30 16:20:33.609598 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.609570 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 30 16:20:33.612790 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.612768 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-5fw2s"] Oct 30 16:20:33.612914 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.612891 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-df85cd6ff-6pd8x" Oct 30 16:20:33.622490 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.622474 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-4wrf8"] Oct 30 16:20:33.622594 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.622567 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-5fw2s" Oct 30 16:20:33.631378 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.631361 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/154b2258-1b09-444b-ad80-2b194195e2cb-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-p4rk6\" (UID: \"154b2258-1b09-444b-ad80-2b194195e2cb\") " pod="openshift-insights/insights-operator-7c4986d449-p4rk6" Oct 30 16:20:33.631470 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.631383 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/06332900-7903-4275-8e33-de06de7f6580-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-cswcz\" (UID: \"06332900-7903-4275-8e33-de06de7f6580\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-cswcz" Oct 30 16:20:33.631470 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.631401 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnv4j\" (UniqueName: \"kubernetes.io/projected/a6341cee-9ba8-41ce-bb75-29c9cb4e6d29-kube-api-access-gnv4j\") pod \"kas-connection-checker-685895c7c9-fhzbv\" (UID: \"a6341cee-9ba8-41ce-bb75-29c9cb4e6d29\") " pod="kube-system/kas-connection-checker-685895c7c9-fhzbv" Oct 30 16:20:33.631470 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.631415 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/154b2258-1b09-444b-ad80-2b194195e2cb-snapshots\") pod \"insights-operator-7c4986d449-p4rk6\" (UID: \"154b2258-1b09-444b-ad80-2b194195e2cb\") " pod="openshift-insights/insights-operator-7c4986d449-p4rk6" Oct 30 16:20:33.631653 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.631530 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/154b2258-1b09-444b-ad80-2b194195e2cb-tmp\") pod \"insights-operator-7c4986d449-p4rk6\" (UID: \"154b2258-1b09-444b-ad80-2b194195e2cb\") " pod="openshift-insights/insights-operator-7c4986d449-p4rk6" Oct 30 16:20:33.631653 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.631571 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kgld\" (UniqueName: \"kubernetes.io/projected/154b2258-1b09-444b-ad80-2b194195e2cb-kube-api-access-4kgld\") pod \"insights-operator-7c4986d449-p4rk6\" (UID: \"154b2258-1b09-444b-ad80-2b194195e2cb\") " pod="openshift-insights/insights-operator-7c4986d449-p4rk6" Oct 30 16:20:33.631653 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.631629 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/06332900-7903-4275-8e33-de06de7f6580-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-cswcz\" (UID: \"06332900-7903-4275-8e33-de06de7f6580\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-cswcz" Oct 30 16:20:33.631806 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.631660 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/154b2258-1b09-444b-ad80-2b194195e2cb-service-ca-bundle\") pod \"insights-operator-7c4986d449-p4rk6\" (UID: \"154b2258-1b09-444b-ad80-2b194195e2cb\") " pod="openshift-insights/insights-operator-7c4986d449-p4rk6" Oct 30 16:20:33.631806 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.631725 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4d6v2\" (UniqueName: \"kubernetes.io/projected/06332900-7903-4275-8e33-de06de7f6580-kube-api-access-4d6v2\") pod \"cluster-samples-operator-6ddb6c8b94-cswcz\" (UID: \"06332900-7903-4275-8e33-de06de7f6580\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-cswcz" Oct 30 16:20:33.631806 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.631755 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4hr8\" (UniqueName: \"kubernetes.io/projected/90d73ca9-5d88-4467-99fa-d4e1a67f3828-kube-api-access-b4hr8\") pod \"kas-connection-checker-685895c7c9-cpxbb\" (UID: \"90d73ca9-5d88-4467-99fa-d4e1a67f3828\") " pod="kube-system/kas-connection-checker-685895c7c9-cpxbb" Oct 30 16:20:33.631806 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.631785 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/154b2258-1b09-444b-ad80-2b194195e2cb-serving-cert\") pod \"insights-operator-7c4986d449-p4rk6\" (UID: \"154b2258-1b09-444b-ad80-2b194195e2cb\") " pod="openshift-insights/insights-operator-7c4986d449-p4rk6" Oct 30 16:20:33.631983 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.631821 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mkkh\" (UniqueName: \"kubernetes.io/projected/d719cbcb-9874-4b85-8fe7-e1bf53511235-kube-api-access-2mkkh\") pod \"kas-connection-checker-685895c7c9-4dfsk\" (UID: \"d719cbcb-9874-4b85-8fe7-e1bf53511235\") " pod="kube-system/kas-connection-checker-685895c7c9-4dfsk" Oct 30 16:20:33.632051 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.632036 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-f8dddb4c-dcbgb"] Oct 30 16:20:33.632166 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.632152 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-4wrf8" Oct 30 16:20:33.640569 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.640531 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jrdmg"] Oct 30 16:20:33.640672 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.640631 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-f8dddb4c-dcbgb" Oct 30 16:20:33.647905 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.647890 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-992nb"] Oct 30 16:20:33.648021 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.648002 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jrdmg" Oct 30 16:20:33.650651 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.650631 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 30 16:20:33.650753 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.650635 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 16:20:33.650753 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.650699 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 16:20:33.650753 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.650713 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 30 16:20:33.650753 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.650740 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 16:20:33.650753 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.650751 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 16:20:33.651003 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.650811 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 16:20:33.651003 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.650812 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 30 16:20:33.651003 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.650854 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 30 16:20:33.651003 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.650912 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 30 16:20:33.651003 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.650964 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 30 16:20:33.651003 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.650987 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 30 16:20:33.651003 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.650997 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 30 16:20:33.651218 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.651007 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 30 16:20:33.651218 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.650988 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 16:20:33.651276 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.651247 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 16:20:33.651306 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.651282 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 30 16:20:33.651306 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.651301 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 30 16:20:33.651358 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.651314 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 30 16:20:33.655249 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.655232 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-65d65dd574-nbb2b"] Oct 30 16:20:33.655406 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.655392 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-992nb" Oct 30 16:20:33.661795 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.661779 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-nwvwc"] Oct 30 16:20:33.661887 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.661875 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-65d65dd574-nbb2b" Oct 30 16:20:33.665513 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.665496 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 30 16:20:33.670058 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.670033 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-qg6t6"] Oct 30 16:20:33.670154 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.670136 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-nwvwc" Oct 30 16:20:33.674985 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.674953 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 30 16:20:33.675083 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.675034 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 16:20:33.680984 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.680967 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-l4mhk"] Oct 30 16:20:33.681089 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.681076 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qg6t6" Oct 30 16:20:33.692309 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.692287 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 30 16:20:33.692389 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.692308 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 30 16:20:33.692467 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.692455 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 30 16:20:33.692571 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.692556 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 16:20:33.692643 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.692570 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 16:20:33.692643 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.692613 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 30 16:20:33.692643 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.692629 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 30 16:20:33.692643 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.692639 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 30 16:20:33.692770 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.692641 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 30 16:20:33.692770 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.692659 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-pmlj2\"" Oct 30 16:20:33.692770 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.692664 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 30 16:20:33.692770 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.692707 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 16:20:33.693648 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.693636 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-cpxbb"] Oct 30 16:20:33.693683 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.693663 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-4dfsk"] Oct 30 16:20:33.693683 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.693678 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-fhzbv"] Oct 30 16:20:33.693741 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.693699 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-4nn7z"] Oct 30 16:20:33.693741 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.693719 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-p4rk6"] Oct 30 16:20:33.693741 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.693738 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-sfq4m"] Oct 30 16:20:33.693820 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.693746 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-l4mhk" Oct 30 16:20:33.693820 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.693759 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-992nb"] Oct 30 16:20:33.693820 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.693779 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jrdmg"] Oct 30 16:20:33.693820 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.693797 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-df85cd6ff-6pd8x"] Oct 30 16:20:33.693820 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.693818 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-l4mhk"] Oct 30 16:20:33.693995 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.693836 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-cswcz"] Oct 30 16:20:33.693995 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.693861 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-65d65dd574-nbb2b"] Oct 30 16:20:33.693995 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.693880 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-f8dddb4c-dcbgb"] Oct 30 16:20:33.693995 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.693905 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-5fw2s"] Oct 30 16:20:33.693995 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.693928 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-nwvwc"] Oct 30 16:20:33.693995 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.693950 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qg6t6"] Oct 30 16:20:33.693995 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.693977 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-4wrf8"] Oct 30 16:20:33.696919 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.696896 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 16:20:33.697094 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.697077 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-s8kck\"" Oct 30 16:20:33.711067 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.711031 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-m9dmv\"" Oct 30 16:20:33.711067 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.711037 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 16:20:33.732238 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.732191 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/154b2258-1b09-444b-ad80-2b194195e2cb-tmp\") pod \"insights-operator-7c4986d449-p4rk6\" (UID: \"154b2258-1b09-444b-ad80-2b194195e2cb\") " pod="openshift-insights/insights-operator-7c4986d449-p4rk6" Oct 30 16:20:33.732238 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.732220 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kgld\" (UniqueName: \"kubernetes.io/projected/154b2258-1b09-444b-ad80-2b194195e2cb-kube-api-access-4kgld\") pod \"insights-operator-7c4986d449-p4rk6\" (UID: \"154b2258-1b09-444b-ad80-2b194195e2cb\") " pod="openshift-insights/insights-operator-7c4986d449-p4rk6" Oct 30 16:20:33.732238 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.732237 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qc9ck\" (UniqueName: \"kubernetes.io/projected/774953d5-23d3-4db1-aec9-3b78ef81017d-kube-api-access-qc9ck\") pod \"ingress-canary-l4mhk\" (UID: \"774953d5-23d3-4db1-aec9-3b78ef81017d\") " pod="openshift-ingress-canary/ingress-canary-l4mhk" Oct 30 16:20:33.732428 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.732254 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/209633e5-92f7-4a7f-9d68-3d48473ed2bf-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jrdmg\" (UID: \"209633e5-92f7-4a7f-9d68-3d48473ed2bf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jrdmg" Oct 30 16:20:33.732428 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.732270 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/06332900-7903-4275-8e33-de06de7f6580-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-cswcz\" (UID: \"06332900-7903-4275-8e33-de06de7f6580\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-cswcz" Oct 30 16:20:33.732428 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.732285 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9fafe556-9653-49df-a851-100b51bb7f47-registry-certificates\") pod \"image-registry-f8dddb4c-dcbgb\" (UID: \"9fafe556-9653-49df-a851-100b51bb7f47\") " pod="openshift-image-registry/image-registry-f8dddb4c-dcbgb" Oct 30 16:20:33.732428 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.732353 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4d6v2\" (UniqueName: \"kubernetes.io/projected/06332900-7903-4275-8e33-de06de7f6580-kube-api-access-4d6v2\") pod \"cluster-samples-operator-6ddb6c8b94-cswcz\" (UID: \"06332900-7903-4275-8e33-de06de7f6580\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-cswcz" Oct 30 16:20:33.732606 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.732523 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4f0f46f4-5cda-4d67-b6f3-47df06eb844c-trusted-ca\") pod \"console-operator-7cd56b8dbc-4wrf8\" (UID: \"4f0f46f4-5cda-4d67-b6f3-47df06eb844c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-4wrf8" Oct 30 16:20:33.732606 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.732565 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a0cd0943-8d62-41e4-b3e7-29eaabec7277-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-sfq4m\" (UID: \"a0cd0943-8d62-41e4-b3e7-29eaabec7277\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sfq4m" Oct 30 16:20:33.732681 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.732610 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9fafe556-9653-49df-a851-100b51bb7f47-installation-pull-secrets\") pod \"image-registry-f8dddb4c-dcbgb\" (UID: \"9fafe556-9653-49df-a851-100b51bb7f47\") " pod="openshift-image-registry/image-registry-f8dddb4c-dcbgb" Oct 30 16:20:33.732681 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.732634 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4hr8\" (UniqueName: \"kubernetes.io/projected/90d73ca9-5d88-4467-99fa-d4e1a67f3828-kube-api-access-b4hr8\") pod \"kas-connection-checker-685895c7c9-cpxbb\" (UID: \"90d73ca9-5d88-4467-99fa-d4e1a67f3828\") " pod="kube-system/kas-connection-checker-685895c7c9-cpxbb" Oct 30 16:20:33.732681 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.732654 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e90fa4d-66a8-4689-8f41-ca090d6697a9-tmp-dir\") pod \"dns-default-qg6t6\" (UID: \"3e90fa4d-66a8-4689-8f41-ca090d6697a9\") " pod="openshift-dns/dns-default-qg6t6" Oct 30 16:20:33.732821 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.732762 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f79e659b-79c9-43ee-8733-c685f60e87fb-registry-certificates\") pod \"image-registry-65d65dd574-nbb2b\" (UID: \"f79e659b-79c9-43ee-8733-c685f60e87fb\") " pod="openshift-image-registry/image-registry-65d65dd574-nbb2b" Oct 30 16:20:33.732821 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.732787 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-httvd\" (UniqueName: \"kubernetes.io/projected/9fafe556-9653-49df-a851-100b51bb7f47-kube-api-access-httvd\") pod \"image-registry-f8dddb4c-dcbgb\" (UID: \"9fafe556-9653-49df-a851-100b51bb7f47\") " pod="openshift-image-registry/image-registry-f8dddb4c-dcbgb" Oct 30 16:20:33.732821 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.732809 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ef87558b-2ba2-41ce-9a0b-9e4b18663a6c-serving-cert\") pod \"service-ca-operator-544d749c8c-4nn7z\" (UID: \"ef87558b-2ba2-41ce-9a0b-9e4b18663a6c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-4nn7z" Oct 30 16:20:33.732962 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.732824 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/b99d8a1f-000b-48ff-b4b0-7a62bc79c032-default-certificate\") pod \"router-default-df85cd6ff-6pd8x\" (UID: \"b99d8a1f-000b-48ff-b4b0-7a62bc79c032\") " pod="openshift-ingress/router-default-df85cd6ff-6pd8x" Oct 30 16:20:33.732962 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.732845 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9fafe556-9653-49df-a851-100b51bb7f47-image-registry-private-configuration\") pod \"image-registry-f8dddb4c-dcbgb\" (UID: \"9fafe556-9653-49df-a851-100b51bb7f47\") " pod="openshift-image-registry/image-registry-f8dddb4c-dcbgb" Oct 30 16:20:33.732962 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.732867 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/154b2258-1b09-444b-ad80-2b194195e2cb-tmp\") pod \"insights-operator-7c4986d449-p4rk6\" (UID: \"154b2258-1b09-444b-ad80-2b194195e2cb\") " pod="openshift-insights/insights-operator-7c4986d449-p4rk6" Oct 30 16:20:33.732962 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.732874 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f79e659b-79c9-43ee-8733-c685f60e87fb-bound-sa-token\") pod \"image-registry-65d65dd574-nbb2b\" (UID: \"f79e659b-79c9-43ee-8733-c685f60e87fb\") " pod="openshift-image-registry/image-registry-65d65dd574-nbb2b" Oct 30 16:20:33.732962 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.732954 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnsg7\" (UniqueName: \"kubernetes.io/projected/4f0f46f4-5cda-4d67-b6f3-47df06eb844c-kube-api-access-bnsg7\") pod \"console-operator-7cd56b8dbc-4wrf8\" (UID: \"4f0f46f4-5cda-4d67-b6f3-47df06eb844c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-4wrf8" Oct 30 16:20:33.733102 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.732990 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-849kr\" (UniqueName: \"kubernetes.io/projected/4d9a19d6-94d4-4490-8f93-89ca745b768f-kube-api-access-849kr\") pod \"volume-data-source-validator-58c9f5f46d-5fw2s\" (UID: \"4d9a19d6-94d4-4490-8f93-89ca745b768f\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-5fw2s" Oct 30 16:20:33.733102 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.733013 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f79e659b-79c9-43ee-8733-c685f60e87fb-installation-pull-secrets\") pod \"image-registry-65d65dd574-nbb2b\" (UID: \"f79e659b-79c9-43ee-8733-c685f60e87fb\") " pod="openshift-image-registry/image-registry-65d65dd574-nbb2b" Oct 30 16:20:33.733102 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.733066 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/154b2258-1b09-444b-ad80-2b194195e2cb-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-p4rk6\" (UID: \"154b2258-1b09-444b-ad80-2b194195e2cb\") " pod="openshift-insights/insights-operator-7c4986d449-p4rk6" Oct 30 16:20:33.733181 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.733101 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4f0f46f4-5cda-4d67-b6f3-47df06eb844c-serving-cert\") pod \"console-operator-7cd56b8dbc-4wrf8\" (UID: \"4f0f46f4-5cda-4d67-b6f3-47df06eb844c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-4wrf8" Oct 30 16:20:33.733181 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.733151 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnv4j\" (UniqueName: \"kubernetes.io/projected/a6341cee-9ba8-41ce-bb75-29c9cb4e6d29-kube-api-access-gnv4j\") pod \"kas-connection-checker-685895c7c9-fhzbv\" (UID: \"a6341cee-9ba8-41ce-bb75-29c9cb4e6d29\") " pod="kube-system/kas-connection-checker-685895c7c9-fhzbv" Oct 30 16:20:33.733239 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.733192 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/154b2258-1b09-444b-ad80-2b194195e2cb-snapshots\") pod \"insights-operator-7c4986d449-p4rk6\" (UID: \"154b2258-1b09-444b-ad80-2b194195e2cb\") " pod="openshift-insights/insights-operator-7c4986d449-p4rk6" Oct 30 16:20:33.733239 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.733192 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/06332900-7903-4275-8e33-de06de7f6580-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-cswcz\" (UID: \"06332900-7903-4275-8e33-de06de7f6580\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-cswcz" Oct 30 16:20:33.733239 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.733227 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f79e659b-79c9-43ee-8733-c685f60e87fb-image-registry-private-configuration\") pod \"image-registry-65d65dd574-nbb2b\" (UID: \"f79e659b-79c9-43ee-8733-c685f60e87fb\") " pod="openshift-image-registry/image-registry-65d65dd574-nbb2b" Oct 30 16:20:33.733352 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.733333 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-478bs\" (UniqueName: \"kubernetes.io/projected/0d304765-4b62-4cba-bad2-452771d9b201-kube-api-access-478bs\") pod \"network-check-source-6f78d59f99-nwvwc\" (UID: \"0d304765-4b62-4cba-bad2-452771d9b201\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-nwvwc" Oct 30 16:20:33.733392 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.733366 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/b99d8a1f-000b-48ff-b4b0-7a62bc79c032-stats-auth\") pod \"router-default-df85cd6ff-6pd8x\" (UID: \"b99d8a1f-000b-48ff-b4b0-7a62bc79c032\") " pod="openshift-ingress/router-default-df85cd6ff-6pd8x" Oct 30 16:20:33.733392 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.733386 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9fafe556-9653-49df-a851-100b51bb7f47-trusted-ca\") pod \"image-registry-f8dddb4c-dcbgb\" (UID: \"9fafe556-9653-49df-a851-100b51bb7f47\") " pod="openshift-image-registry/image-registry-f8dddb4c-dcbgb" Oct 30 16:20:33.733454 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.733426 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9fafe556-9653-49df-a851-100b51bb7f47-bound-sa-token\") pod \"image-registry-f8dddb4c-dcbgb\" (UID: \"9fafe556-9653-49df-a851-100b51bb7f47\") " pod="openshift-image-registry/image-registry-f8dddb4c-dcbgb" Oct 30 16:20:33.733485 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.733458 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f79e659b-79c9-43ee-8733-c685f60e87fb-registry-tls\") pod \"image-registry-65d65dd574-nbb2b\" (UID: \"f79e659b-79c9-43ee-8733-c685f60e87fb\") " pod="openshift-image-registry/image-registry-65d65dd574-nbb2b" Oct 30 16:20:33.733533 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.733514 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1042378b-9f46-4e19-88e4-2628e532e8c3-nginx-conf\") pod \"networking-console-plugin-86c478bb88-992nb\" (UID: \"1042378b-9f46-4e19-88e4-2628e532e8c3\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-992nb" Oct 30 16:20:33.733590 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.733562 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b99d8a1f-000b-48ff-b4b0-7a62bc79c032-service-ca-bundle\") pod \"router-default-df85cd6ff-6pd8x\" (UID: \"b99d8a1f-000b-48ff-b4b0-7a62bc79c032\") " pod="openshift-ingress/router-default-df85cd6ff-6pd8x" Oct 30 16:20:33.733631 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.733616 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/209633e5-92f7-4a7f-9d68-3d48473ed2bf-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jrdmg\" (UID: \"209633e5-92f7-4a7f-9d68-3d48473ed2bf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jrdmg" Oct 30 16:20:33.733663 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.733643 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3e90fa4d-66a8-4689-8f41-ca090d6697a9-config-volume\") pod \"dns-default-qg6t6\" (UID: \"3e90fa4d-66a8-4689-8f41-ca090d6697a9\") " pod="openshift-dns/dns-default-qg6t6" Oct 30 16:20:33.733705 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.733689 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/154b2258-1b09-444b-ad80-2b194195e2cb-service-ca-bundle\") pod \"insights-operator-7c4986d449-p4rk6\" (UID: \"154b2258-1b09-444b-ad80-2b194195e2cb\") " pod="openshift-insights/insights-operator-7c4986d449-p4rk6" Oct 30 16:20:33.733740 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.733731 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjp8r\" (UniqueName: \"kubernetes.io/projected/3e90fa4d-66a8-4689-8f41-ca090d6697a9-kube-api-access-pjp8r\") pod \"dns-default-qg6t6\" (UID: \"3e90fa4d-66a8-4689-8f41-ca090d6697a9\") " pod="openshift-dns/dns-default-qg6t6" Oct 30 16:20:33.733772 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.733746 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/154b2258-1b09-444b-ad80-2b194195e2cb-snapshots\") pod \"insights-operator-7c4986d449-p4rk6\" (UID: \"154b2258-1b09-444b-ad80-2b194195e2cb\") " pod="openshift-insights/insights-operator-7c4986d449-p4rk6" Oct 30 16:20:33.733801 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.733773 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4f0f46f4-5cda-4d67-b6f3-47df06eb844c-config\") pod \"console-operator-7cd56b8dbc-4wrf8\" (UID: \"4f0f46f4-5cda-4d67-b6f3-47df06eb844c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-4wrf8" Oct 30 16:20:33.733832 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.733811 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1042378b-9f46-4e19-88e4-2628e532e8c3-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-992nb\" (UID: \"1042378b-9f46-4e19-88e4-2628e532e8c3\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-992nb" Oct 30 16:20:33.733832 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.733815 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/154b2258-1b09-444b-ad80-2b194195e2cb-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-p4rk6\" (UID: \"154b2258-1b09-444b-ad80-2b194195e2cb\") " pod="openshift-insights/insights-operator-7c4986d449-p4rk6" Oct 30 16:20:33.733889 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.733873 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z65wk\" (UniqueName: \"kubernetes.io/projected/209633e5-92f7-4a7f-9d68-3d48473ed2bf-kube-api-access-z65wk\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jrdmg\" (UID: \"209633e5-92f7-4a7f-9d68-3d48473ed2bf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jrdmg" Oct 30 16:20:33.733921 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.733900 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a0cd0943-8d62-41e4-b3e7-29eaabec7277-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-sfq4m\" (UID: \"a0cd0943-8d62-41e4-b3e7-29eaabec7277\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sfq4m" Oct 30 16:20:33.733953 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.733933 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/154b2258-1b09-444b-ad80-2b194195e2cb-serving-cert\") pod \"insights-operator-7c4986d449-p4rk6\" (UID: \"154b2258-1b09-444b-ad80-2b194195e2cb\") " pod="openshift-insights/insights-operator-7c4986d449-p4rk6" Oct 30 16:20:33.733983 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.733955 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mkkh\" (UniqueName: \"kubernetes.io/projected/d719cbcb-9874-4b85-8fe7-e1bf53511235-kube-api-access-2mkkh\") pod \"kas-connection-checker-685895c7c9-4dfsk\" (UID: \"d719cbcb-9874-4b85-8fe7-e1bf53511235\") " pod="kube-system/kas-connection-checker-685895c7c9-4dfsk" Oct 30 16:20:33.733983 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.733972 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9fafe556-9653-49df-a851-100b51bb7f47-registry-tls\") pod \"image-registry-f8dddb4c-dcbgb\" (UID: \"9fafe556-9653-49df-a851-100b51bb7f47\") " pod="openshift-image-registry/image-registry-f8dddb4c-dcbgb" Oct 30 16:20:33.734042 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.733988 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/774953d5-23d3-4db1-aec9-3b78ef81017d-cert\") pod \"ingress-canary-l4mhk\" (UID: \"774953d5-23d3-4db1-aec9-3b78ef81017d\") " pod="openshift-ingress-canary/ingress-canary-l4mhk" Oct 30 16:20:33.734042 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.734003 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b99d8a1f-000b-48ff-b4b0-7a62bc79c032-metrics-certs\") pod \"router-default-df85cd6ff-6pd8x\" (UID: \"b99d8a1f-000b-48ff-b4b0-7a62bc79c032\") " pod="openshift-ingress/router-default-df85cd6ff-6pd8x" Oct 30 16:20:33.734117 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.734040 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9fafe556-9653-49df-a851-100b51bb7f47-ca-trust-extracted\") pod \"image-registry-f8dddb4c-dcbgb\" (UID: \"9fafe556-9653-49df-a851-100b51bb7f47\") " pod="openshift-image-registry/image-registry-f8dddb4c-dcbgb" Oct 30 16:20:33.734117 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.734073 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/154b2258-1b09-444b-ad80-2b194195e2cb-service-ca-bundle\") pod \"insights-operator-7c4986d449-p4rk6\" (UID: \"154b2258-1b09-444b-ad80-2b194195e2cb\") " pod="openshift-insights/insights-operator-7c4986d449-p4rk6" Oct 30 16:20:33.734190 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.734138 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3e90fa4d-66a8-4689-8f41-ca090d6697a9-metrics-tls\") pod \"dns-default-qg6t6\" (UID: \"3e90fa4d-66a8-4689-8f41-ca090d6697a9\") " pod="openshift-dns/dns-default-qg6t6" Oct 30 16:20:33.734239 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.734185 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ef87558b-2ba2-41ce-9a0b-9e4b18663a6c-config\") pod \"service-ca-operator-544d749c8c-4nn7z\" (UID: \"ef87558b-2ba2-41ce-9a0b-9e4b18663a6c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-4nn7z" Oct 30 16:20:33.734278 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.734231 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhcx7\" (UniqueName: \"kubernetes.io/projected/b99d8a1f-000b-48ff-b4b0-7a62bc79c032-kube-api-access-bhcx7\") pod \"router-default-df85cd6ff-6pd8x\" (UID: \"b99d8a1f-000b-48ff-b4b0-7a62bc79c032\") " pod="openshift-ingress/router-default-df85cd6ff-6pd8x" Oct 30 16:20:33.734307 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.734284 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f79e659b-79c9-43ee-8733-c685f60e87fb-trusted-ca\") pod \"image-registry-65d65dd574-nbb2b\" (UID: \"f79e659b-79c9-43ee-8733-c685f60e87fb\") " pod="openshift-image-registry/image-registry-65d65dd574-nbb2b" Oct 30 16:20:33.734337 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.734317 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f79e659b-79c9-43ee-8733-c685f60e87fb-ca-trust-extracted\") pod \"image-registry-65d65dd574-nbb2b\" (UID: \"f79e659b-79c9-43ee-8733-c685f60e87fb\") " pod="openshift-image-registry/image-registry-65d65dd574-nbb2b" Oct 30 16:20:33.734370 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.734356 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlqc8\" (UniqueName: \"kubernetes.io/projected/ef87558b-2ba2-41ce-9a0b-9e4b18663a6c-kube-api-access-qlqc8\") pod \"service-ca-operator-544d749c8c-4nn7z\" (UID: \"ef87558b-2ba2-41ce-9a0b-9e4b18663a6c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-4nn7z" Oct 30 16:20:33.734426 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.734412 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t54w7\" (UniqueName: \"kubernetes.io/projected/a0cd0943-8d62-41e4-b3e7-29eaabec7277-kube-api-access-t54w7\") pod \"cluster-monitoring-operator-884576c57-sfq4m\" (UID: \"a0cd0943-8d62-41e4-b3e7-29eaabec7277\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sfq4m" Oct 30 16:20:33.734479 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.734441 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/06332900-7903-4275-8e33-de06de7f6580-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-cswcz\" (UID: \"06332900-7903-4275-8e33-de06de7f6580\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-cswcz" Oct 30 16:20:33.734479 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.734460 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qh2h\" (UniqueName: \"kubernetes.io/projected/f79e659b-79c9-43ee-8733-c685f60e87fb-kube-api-access-5qh2h\") pod \"image-registry-65d65dd574-nbb2b\" (UID: \"f79e659b-79c9-43ee-8733-c685f60e87fb\") " pod="openshift-image-registry/image-registry-65d65dd574-nbb2b" Oct 30 16:20:33.734571 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:33.734552 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 16:20:33.734652 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:33.734641 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06332900-7903-4275-8e33-de06de7f6580-samples-operator-tls podName:06332900-7903-4275-8e33-de06de7f6580 nodeName:}" failed. No retries permitted until 2026-10-30 16:20:34.234622224 +0000 UTC m=+45.534910443 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/06332900-7903-4275-8e33-de06de7f6580-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-cswcz" (UID: "06332900-7903-4275-8e33-de06de7f6580") : secret "samples-operator-tls" not found Oct 30 16:20:33.737573 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.737557 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/154b2258-1b09-444b-ad80-2b194195e2cb-serving-cert\") pod \"insights-operator-7c4986d449-p4rk6\" (UID: \"154b2258-1b09-444b-ad80-2b194195e2cb\") " pod="openshift-insights/insights-operator-7c4986d449-p4rk6" Oct 30 16:20:33.750479 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.750457 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 16:20:33.750569 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.750487 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 30 16:20:33.750569 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.750502 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 30 16:20:33.750569 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.750513 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 16:20:33.750762 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.750685 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 30 16:20:33.750762 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.750687 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 30 16:20:33.750762 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.750703 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 16:20:33.750762 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.750740 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 16:20:33.751226 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.751212 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 16:20:33.751694 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.751681 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 16:20:33.755619 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.755595 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kgld\" (UniqueName: \"kubernetes.io/projected/154b2258-1b09-444b-ad80-2b194195e2cb-kube-api-access-4kgld\") pod \"insights-operator-7c4986d449-p4rk6\" (UID: \"154b2258-1b09-444b-ad80-2b194195e2cb\") " pod="openshift-insights/insights-operator-7c4986d449-p4rk6" Oct 30 16:20:33.755928 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.755911 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnv4j\" (UniqueName: \"kubernetes.io/projected/a6341cee-9ba8-41ce-bb75-29c9cb4e6d29-kube-api-access-gnv4j\") pod \"kas-connection-checker-685895c7c9-fhzbv\" (UID: \"a6341cee-9ba8-41ce-bb75-29c9cb4e6d29\") " pod="kube-system/kas-connection-checker-685895c7c9-fhzbv" Oct 30 16:20:33.756034 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.755974 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4hr8\" (UniqueName: \"kubernetes.io/projected/90d73ca9-5d88-4467-99fa-d4e1a67f3828-kube-api-access-b4hr8\") pod \"kas-connection-checker-685895c7c9-cpxbb\" (UID: \"90d73ca9-5d88-4467-99fa-d4e1a67f3828\") " pod="kube-system/kas-connection-checker-685895c7c9-cpxbb" Oct 30 16:20:33.756226 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.756206 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4d6v2\" (UniqueName: \"kubernetes.io/projected/06332900-7903-4275-8e33-de06de7f6580-kube-api-access-4d6v2\") pod \"cluster-samples-operator-6ddb6c8b94-cswcz\" (UID: \"06332900-7903-4275-8e33-de06de7f6580\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-cswcz" Oct 30 16:20:33.757849 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.757829 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mkkh\" (UniqueName: \"kubernetes.io/projected/d719cbcb-9874-4b85-8fe7-e1bf53511235-kube-api-access-2mkkh\") pod \"kas-connection-checker-685895c7c9-4dfsk\" (UID: \"d719cbcb-9874-4b85-8fe7-e1bf53511235\") " pod="kube-system/kas-connection-checker-685895c7c9-4dfsk" Oct 30 16:20:33.835417 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.835385 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z65wk\" (UniqueName: \"kubernetes.io/projected/209633e5-92f7-4a7f-9d68-3d48473ed2bf-kube-api-access-z65wk\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jrdmg\" (UID: \"209633e5-92f7-4a7f-9d68-3d48473ed2bf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jrdmg" Oct 30 16:20:33.835417 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.835423 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a0cd0943-8d62-41e4-b3e7-29eaabec7277-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-sfq4m\" (UID: \"a0cd0943-8d62-41e4-b3e7-29eaabec7277\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sfq4m" Oct 30 16:20:33.835686 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.835442 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9fafe556-9653-49df-a851-100b51bb7f47-registry-tls\") pod \"image-registry-f8dddb4c-dcbgb\" (UID: \"9fafe556-9653-49df-a851-100b51bb7f47\") " pod="openshift-image-registry/image-registry-f8dddb4c-dcbgb" Oct 30 16:20:33.835686 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:33.835523 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 16:20:33.835686 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:33.835534 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-f8dddb4c-dcbgb: secret "image-registry-tls" not found Oct 30 16:20:33.835686 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:33.835541 2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 16:20:33.835686 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.835569 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/774953d5-23d3-4db1-aec9-3b78ef81017d-cert\") pod \"ingress-canary-l4mhk\" (UID: \"774953d5-23d3-4db1-aec9-3b78ef81017d\") " pod="openshift-ingress-canary/ingress-canary-l4mhk" Oct 30 16:20:33.835686 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:33.835597 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9fafe556-9653-49df-a851-100b51bb7f47-registry-tls podName:9fafe556-9653-49df-a851-100b51bb7f47 nodeName:}" failed. No retries permitted until 2026-10-30 16:20:34.335566147 +0000 UTC m=+45.635854368 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9fafe556-9653-49df-a851-100b51bb7f47-registry-tls") pod "image-registry-f8dddb4c-dcbgb" (UID: "9fafe556-9653-49df-a851-100b51bb7f47") : secret "image-registry-tls" not found Oct 30 16:20:33.835686 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:33.835625 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0cd0943-8d62-41e4-b3e7-29eaabec7277-cluster-monitoring-operator-tls podName:a0cd0943-8d62-41e4-b3e7-29eaabec7277 nodeName:}" failed. No retries permitted until 2026-10-30 16:20:34.335615268 +0000 UTC m=+45.635903484 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a0cd0943-8d62-41e4-b3e7-29eaabec7277-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-sfq4m" (UID: "a0cd0943-8d62-41e4-b3e7-29eaabec7277") : secret "cluster-monitoring-operator-tls" not found Oct 30 16:20:33.835686 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.835644 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b99d8a1f-000b-48ff-b4b0-7a62bc79c032-metrics-certs\") pod \"router-default-df85cd6ff-6pd8x\" (UID: \"b99d8a1f-000b-48ff-b4b0-7a62bc79c032\") " pod="openshift-ingress/router-default-df85cd6ff-6pd8x" Oct 30 16:20:33.835686 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.835661 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9fafe556-9653-49df-a851-100b51bb7f47-ca-trust-extracted\") pod \"image-registry-f8dddb4c-dcbgb\" (UID: \"9fafe556-9653-49df-a851-100b51bb7f47\") " pod="openshift-image-registry/image-registry-f8dddb4c-dcbgb" Oct 30 16:20:33.835686 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.835676 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3e90fa4d-66a8-4689-8f41-ca090d6697a9-metrics-tls\") pod \"dns-default-qg6t6\" (UID: \"3e90fa4d-66a8-4689-8f41-ca090d6697a9\") " pod="openshift-dns/dns-default-qg6t6" Oct 30 16:20:33.836177 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.835695 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ef87558b-2ba2-41ce-9a0b-9e4b18663a6c-config\") pod \"service-ca-operator-544d749c8c-4nn7z\" (UID: \"ef87558b-2ba2-41ce-9a0b-9e4b18663a6c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-4nn7z" Oct 30 16:20:33.836177 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.835720 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhcx7\" (UniqueName: \"kubernetes.io/projected/b99d8a1f-000b-48ff-b4b0-7a62bc79c032-kube-api-access-bhcx7\") pod \"router-default-df85cd6ff-6pd8x\" (UID: \"b99d8a1f-000b-48ff-b4b0-7a62bc79c032\") " pod="openshift-ingress/router-default-df85cd6ff-6pd8x" Oct 30 16:20:33.836177 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.835743 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f79e659b-79c9-43ee-8733-c685f60e87fb-trusted-ca\") pod \"image-registry-65d65dd574-nbb2b\" (UID: \"f79e659b-79c9-43ee-8733-c685f60e87fb\") " pod="openshift-image-registry/image-registry-65d65dd574-nbb2b" Oct 30 16:20:33.836177 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:33.835768 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 16:20:33.836177 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.835769 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f79e659b-79c9-43ee-8733-c685f60e87fb-ca-trust-extracted\") pod \"image-registry-65d65dd574-nbb2b\" (UID: \"f79e659b-79c9-43ee-8733-c685f60e87fb\") " pod="openshift-image-registry/image-registry-65d65dd574-nbb2b" Oct 30 16:20:33.836177 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.835797 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlqc8\" (UniqueName: \"kubernetes.io/projected/ef87558b-2ba2-41ce-9a0b-9e4b18663a6c-kube-api-access-qlqc8\") pod \"service-ca-operator-544d749c8c-4nn7z\" (UID: \"ef87558b-2ba2-41ce-9a0b-9e4b18663a6c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-4nn7z" Oct 30 16:20:33.836177 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:33.835807 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e90fa4d-66a8-4689-8f41-ca090d6697a9-metrics-tls podName:3e90fa4d-66a8-4689-8f41-ca090d6697a9 nodeName:}" failed. No retries permitted until 2026-10-30 16:20:34.335796169 +0000 UTC m=+45.636084377 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3e90fa4d-66a8-4689-8f41-ca090d6697a9-metrics-tls") pod "dns-default-qg6t6" (UID: "3e90fa4d-66a8-4689-8f41-ca090d6697a9") : secret "dns-default-metrics-tls" not found Oct 30 16:20:33.836177 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.835835 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t54w7\" (UniqueName: \"kubernetes.io/projected/a0cd0943-8d62-41e4-b3e7-29eaabec7277-kube-api-access-t54w7\") pod \"cluster-monitoring-operator-884576c57-sfq4m\" (UID: \"a0cd0943-8d62-41e4-b3e7-29eaabec7277\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sfq4m" Oct 30 16:20:33.836177 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.835878 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qh2h\" (UniqueName: \"kubernetes.io/projected/f79e659b-79c9-43ee-8733-c685f60e87fb-kube-api-access-5qh2h\") pod \"image-registry-65d65dd574-nbb2b\" (UID: \"f79e659b-79c9-43ee-8733-c685f60e87fb\") " pod="openshift-image-registry/image-registry-65d65dd574-nbb2b" Oct 30 16:20:33.836177 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.835932 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qc9ck\" (UniqueName: \"kubernetes.io/projected/774953d5-23d3-4db1-aec9-3b78ef81017d-kube-api-access-qc9ck\") pod \"ingress-canary-l4mhk\" (UID: \"774953d5-23d3-4db1-aec9-3b78ef81017d\") " pod="openshift-ingress-canary/ingress-canary-l4mhk" Oct 30 16:20:33.836177 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.835960 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/209633e5-92f7-4a7f-9d68-3d48473ed2bf-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jrdmg\" (UID: \"209633e5-92f7-4a7f-9d68-3d48473ed2bf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jrdmg" Oct 30 16:20:33.836177 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.835986 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9fafe556-9653-49df-a851-100b51bb7f47-registry-certificates\") pod \"image-registry-f8dddb4c-dcbgb\" (UID: \"9fafe556-9653-49df-a851-100b51bb7f47\") " pod="openshift-image-registry/image-registry-f8dddb4c-dcbgb" Oct 30 16:20:33.836177 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.836010 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4f0f46f4-5cda-4d67-b6f3-47df06eb844c-trusted-ca\") pod \"console-operator-7cd56b8dbc-4wrf8\" (UID: \"4f0f46f4-5cda-4d67-b6f3-47df06eb844c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-4wrf8" Oct 30 16:20:33.836177 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.836034 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a0cd0943-8d62-41e4-b3e7-29eaabec7277-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-sfq4m\" (UID: \"a0cd0943-8d62-41e4-b3e7-29eaabec7277\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sfq4m" Oct 30 16:20:33.836177 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.836038 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9fafe556-9653-49df-a851-100b51bb7f47-ca-trust-extracted\") pod \"image-registry-f8dddb4c-dcbgb\" (UID: \"9fafe556-9653-49df-a851-100b51bb7f47\") " pod="openshift-image-registry/image-registry-f8dddb4c-dcbgb" Oct 30 16:20:33.836177 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.836059 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9fafe556-9653-49df-a851-100b51bb7f47-installation-pull-secrets\") pod \"image-registry-f8dddb4c-dcbgb\" (UID: \"9fafe556-9653-49df-a851-100b51bb7f47\") " pod="openshift-image-registry/image-registry-f8dddb4c-dcbgb" Oct 30 16:20:33.837001 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:33.836074 2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 16:20:33.837001 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.836089 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e90fa4d-66a8-4689-8f41-ca090d6697a9-tmp-dir\") pod \"dns-default-qg6t6\" (UID: \"3e90fa4d-66a8-4689-8f41-ca090d6697a9\") " pod="openshift-dns/dns-default-qg6t6" Oct 30 16:20:33.837001 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.836118 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f79e659b-79c9-43ee-8733-c685f60e87fb-registry-certificates\") pod \"image-registry-65d65dd574-nbb2b\" (UID: \"f79e659b-79c9-43ee-8733-c685f60e87fb\") " pod="openshift-image-registry/image-registry-65d65dd574-nbb2b" Oct 30 16:20:33.837001 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.836336 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f79e659b-79c9-43ee-8733-c685f60e87fb-ca-trust-extracted\") pod \"image-registry-65d65dd574-nbb2b\" (UID: \"f79e659b-79c9-43ee-8733-c685f60e87fb\") " pod="openshift-image-registry/image-registry-65d65dd574-nbb2b" Oct 30 16:20:33.837001 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.836355 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-httvd\" (UniqueName: \"kubernetes.io/projected/9fafe556-9653-49df-a851-100b51bb7f47-kube-api-access-httvd\") pod \"image-registry-f8dddb4c-dcbgb\" (UID: \"9fafe556-9653-49df-a851-100b51bb7f47\") " pod="openshift-image-registry/image-registry-f8dddb4c-dcbgb" Oct 30 16:20:33.837001 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.836397 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ef87558b-2ba2-41ce-9a0b-9e4b18663a6c-serving-cert\") pod \"service-ca-operator-544d749c8c-4nn7z\" (UID: \"ef87558b-2ba2-41ce-9a0b-9e4b18663a6c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-4nn7z" Oct 30 16:20:33.837001 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.836395 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ef87558b-2ba2-41ce-9a0b-9e4b18663a6c-config\") pod \"service-ca-operator-544d749c8c-4nn7z\" (UID: \"ef87558b-2ba2-41ce-9a0b-9e4b18663a6c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-4nn7z" Oct 30 16:20:33.837001 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:33.836646 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b99d8a1f-000b-48ff-b4b0-7a62bc79c032-metrics-certs podName:b99d8a1f-000b-48ff-b4b0-7a62bc79c032 nodeName:}" failed. No retries permitted until 2026-10-30 16:20:34.336629246 +0000 UTC m=+45.636917441 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b99d8a1f-000b-48ff-b4b0-7a62bc79c032-metrics-certs") pod "router-default-df85cd6ff-6pd8x" (UID: "b99d8a1f-000b-48ff-b4b0-7a62bc79c032") : secret "router-metrics-certs-default" not found Oct 30 16:20:33.837001 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.836709 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e90fa4d-66a8-4689-8f41-ca090d6697a9-tmp-dir\") pod \"dns-default-qg6t6\" (UID: \"3e90fa4d-66a8-4689-8f41-ca090d6697a9\") " pod="openshift-dns/dns-default-qg6t6" Oct 30 16:20:33.837001 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.836723 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9fafe556-9653-49df-a851-100b51bb7f47-registry-certificates\") pod \"image-registry-f8dddb4c-dcbgb\" (UID: \"9fafe556-9653-49df-a851-100b51bb7f47\") " pod="openshift-image-registry/image-registry-f8dddb4c-dcbgb" Oct 30 16:20:33.837001 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.836724 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/209633e5-92f7-4a7f-9d68-3d48473ed2bf-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jrdmg\" (UID: \"209633e5-92f7-4a7f-9d68-3d48473ed2bf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jrdmg" Oct 30 16:20:33.837528 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.837122 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f79e659b-79c9-43ee-8733-c685f60e87fb-registry-certificates\") pod \"image-registry-65d65dd574-nbb2b\" (UID: \"f79e659b-79c9-43ee-8733-c685f60e87fb\") " pod="openshift-image-registry/image-registry-65d65dd574-nbb2b" Oct 30 16:20:33.837528 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.837164 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/b99d8a1f-000b-48ff-b4b0-7a62bc79c032-default-certificate\") pod \"router-default-df85cd6ff-6pd8x\" (UID: \"b99d8a1f-000b-48ff-b4b0-7a62bc79c032\") " pod="openshift-ingress/router-default-df85cd6ff-6pd8x" Oct 30 16:20:33.837528 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.837191 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4f0f46f4-5cda-4d67-b6f3-47df06eb844c-trusted-ca\") pod \"console-operator-7cd56b8dbc-4wrf8\" (UID: \"4f0f46f4-5cda-4d67-b6f3-47df06eb844c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-4wrf8" Oct 30 16:20:33.837528 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.837203 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9fafe556-9653-49df-a851-100b51bb7f47-image-registry-private-configuration\") pod \"image-registry-f8dddb4c-dcbgb\" (UID: \"9fafe556-9653-49df-a851-100b51bb7f47\") " pod="openshift-image-registry/image-registry-f8dddb4c-dcbgb" Oct 30 16:20:33.837528 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.837248 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f79e659b-79c9-43ee-8733-c685f60e87fb-bound-sa-token\") pod \"image-registry-65d65dd574-nbb2b\" (UID: \"f79e659b-79c9-43ee-8733-c685f60e87fb\") " pod="openshift-image-registry/image-registry-65d65dd574-nbb2b" Oct 30 16:20:33.837528 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.837297 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnsg7\" (UniqueName: \"kubernetes.io/projected/4f0f46f4-5cda-4d67-b6f3-47df06eb844c-kube-api-access-bnsg7\") pod \"console-operator-7cd56b8dbc-4wrf8\" (UID: \"4f0f46f4-5cda-4d67-b6f3-47df06eb844c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-4wrf8" Oct 30 16:20:33.837528 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.837324 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-849kr\" (UniqueName: \"kubernetes.io/projected/4d9a19d6-94d4-4490-8f93-89ca745b768f-kube-api-access-849kr\") pod \"volume-data-source-validator-58c9f5f46d-5fw2s\" (UID: \"4d9a19d6-94d4-4490-8f93-89ca745b768f\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-5fw2s" Oct 30 16:20:33.837528 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.837351 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f79e659b-79c9-43ee-8733-c685f60e87fb-installation-pull-secrets\") pod \"image-registry-65d65dd574-nbb2b\" (UID: \"f79e659b-79c9-43ee-8733-c685f60e87fb\") " pod="openshift-image-registry/image-registry-65d65dd574-nbb2b" Oct 30 16:20:33.837528 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.837395 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4f0f46f4-5cda-4d67-b6f3-47df06eb844c-serving-cert\") pod \"console-operator-7cd56b8dbc-4wrf8\" (UID: \"4f0f46f4-5cda-4d67-b6f3-47df06eb844c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-4wrf8" Oct 30 16:20:33.837528 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.837440 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f79e659b-79c9-43ee-8733-c685f60e87fb-image-registry-private-configuration\") pod \"image-registry-65d65dd574-nbb2b\" (UID: \"f79e659b-79c9-43ee-8733-c685f60e87fb\") " pod="openshift-image-registry/image-registry-65d65dd574-nbb2b" Oct 30 16:20:33.837528 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.837467 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-478bs\" (UniqueName: \"kubernetes.io/projected/0d304765-4b62-4cba-bad2-452771d9b201-kube-api-access-478bs\") pod \"network-check-source-6f78d59f99-nwvwc\" (UID: \"0d304765-4b62-4cba-bad2-452771d9b201\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-nwvwc" Oct 30 16:20:33.837528 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.837491 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/b99d8a1f-000b-48ff-b4b0-7a62bc79c032-stats-auth\") pod \"router-default-df85cd6ff-6pd8x\" (UID: \"b99d8a1f-000b-48ff-b4b0-7a62bc79c032\") " pod="openshift-ingress/router-default-df85cd6ff-6pd8x" Oct 30 16:20:33.837528 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.837519 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9fafe556-9653-49df-a851-100b51bb7f47-trusted-ca\") pod \"image-registry-f8dddb4c-dcbgb\" (UID: \"9fafe556-9653-49df-a851-100b51bb7f47\") " pod="openshift-image-registry/image-registry-f8dddb4c-dcbgb" Oct 30 16:20:33.838176 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.837543 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f79e659b-79c9-43ee-8733-c685f60e87fb-trusted-ca\") pod \"image-registry-65d65dd574-nbb2b\" (UID: \"f79e659b-79c9-43ee-8733-c685f60e87fb\") " pod="openshift-image-registry/image-registry-65d65dd574-nbb2b" Oct 30 16:20:33.838176 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.837544 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9fafe556-9653-49df-a851-100b51bb7f47-bound-sa-token\") pod \"image-registry-f8dddb4c-dcbgb\" (UID: \"9fafe556-9653-49df-a851-100b51bb7f47\") " pod="openshift-image-registry/image-registry-f8dddb4c-dcbgb" Oct 30 16:20:33.838176 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.837604 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f79e659b-79c9-43ee-8733-c685f60e87fb-registry-tls\") pod \"image-registry-65d65dd574-nbb2b\" (UID: \"f79e659b-79c9-43ee-8733-c685f60e87fb\") " pod="openshift-image-registry/image-registry-65d65dd574-nbb2b" Oct 30 16:20:33.838176 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.837639 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1042378b-9f46-4e19-88e4-2628e532e8c3-nginx-conf\") pod \"networking-console-plugin-86c478bb88-992nb\" (UID: \"1042378b-9f46-4e19-88e4-2628e532e8c3\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-992nb" Oct 30 16:20:33.838176 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.837665 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b99d8a1f-000b-48ff-b4b0-7a62bc79c032-service-ca-bundle\") pod \"router-default-df85cd6ff-6pd8x\" (UID: \"b99d8a1f-000b-48ff-b4b0-7a62bc79c032\") " pod="openshift-ingress/router-default-df85cd6ff-6pd8x" Oct 30 16:20:33.838176 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.837698 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/209633e5-92f7-4a7f-9d68-3d48473ed2bf-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jrdmg\" (UID: \"209633e5-92f7-4a7f-9d68-3d48473ed2bf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jrdmg" Oct 30 16:20:33.838176 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.837725 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3e90fa4d-66a8-4689-8f41-ca090d6697a9-config-volume\") pod \"dns-default-qg6t6\" (UID: \"3e90fa4d-66a8-4689-8f41-ca090d6697a9\") " pod="openshift-dns/dns-default-qg6t6" Oct 30 16:20:33.838176 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.837755 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjp8r\" (UniqueName: \"kubernetes.io/projected/3e90fa4d-66a8-4689-8f41-ca090d6697a9-kube-api-access-pjp8r\") pod \"dns-default-qg6t6\" (UID: \"3e90fa4d-66a8-4689-8f41-ca090d6697a9\") " pod="openshift-dns/dns-default-qg6t6" Oct 30 16:20:33.838176 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.837791 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4f0f46f4-5cda-4d67-b6f3-47df06eb844c-config\") pod \"console-operator-7cd56b8dbc-4wrf8\" (UID: \"4f0f46f4-5cda-4d67-b6f3-47df06eb844c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-4wrf8" Oct 30 16:20:33.838176 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.837818 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1042378b-9f46-4e19-88e4-2628e532e8c3-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-992nb\" (UID: \"1042378b-9f46-4e19-88e4-2628e532e8c3\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-992nb" Oct 30 16:20:33.838176 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:33.837954 2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 16:20:33.838176 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:33.838008 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1042378b-9f46-4e19-88e4-2628e532e8c3-networking-console-plugin-cert podName:1042378b-9f46-4e19-88e4-2628e532e8c3 nodeName:}" failed. No retries permitted until 2026-10-30 16:20:34.337991475 +0000 UTC m=+45.638279678 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1042378b-9f46-4e19-88e4-2628e532e8c3-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-992nb" (UID: "1042378b-9f46-4e19-88e4-2628e532e8c3") : secret "networking-console-plugin-cert" not found Oct 30 16:20:33.838788 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:33.838289 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 16:20:33.838788 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:33.838306 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-65d65dd574-nbb2b: secret "image-registry-tls" not found Oct 30 16:20:33.838788 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:33.838348 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f79e659b-79c9-43ee-8733-c685f60e87fb-registry-tls podName:f79e659b-79c9-43ee-8733-c685f60e87fb nodeName:}" failed. No retries permitted until 2026-10-30 16:20:34.338331961 +0000 UTC m=+45.638620162 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f79e659b-79c9-43ee-8733-c685f60e87fb-registry-tls") pod "image-registry-65d65dd574-nbb2b" (UID: "f79e659b-79c9-43ee-8733-c685f60e87fb") : secret "image-registry-tls" not found Oct 30 16:20:33.839393 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.839022 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1042378b-9f46-4e19-88e4-2628e532e8c3-nginx-conf\") pod \"networking-console-plugin-86c478bb88-992nb\" (UID: \"1042378b-9f46-4e19-88e4-2628e532e8c3\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-992nb" Oct 30 16:20:33.839393 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.839119 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9fafe556-9653-49df-a851-100b51bb7f47-trusted-ca\") pod \"image-registry-f8dddb4c-dcbgb\" (UID: \"9fafe556-9653-49df-a851-100b51bb7f47\") " pod="openshift-image-registry/image-registry-f8dddb4c-dcbgb" Oct 30 16:20:33.840552 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.839622 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3e90fa4d-66a8-4689-8f41-ca090d6697a9-config-volume\") pod \"dns-default-qg6t6\" (UID: \"3e90fa4d-66a8-4689-8f41-ca090d6697a9\") " pod="openshift-dns/dns-default-qg6t6" Oct 30 16:20:33.840552 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.839691 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4f0f46f4-5cda-4d67-b6f3-47df06eb844c-config\") pod \"console-operator-7cd56b8dbc-4wrf8\" (UID: \"4f0f46f4-5cda-4d67-b6f3-47df06eb844c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-4wrf8" Oct 30 16:20:33.840552 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.839726 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/774953d5-23d3-4db1-aec9-3b78ef81017d-cert\") pod \"ingress-canary-l4mhk\" (UID: \"774953d5-23d3-4db1-aec9-3b78ef81017d\") " pod="openshift-ingress-canary/ingress-canary-l4mhk" Oct 30 16:20:33.840552 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:33.840053 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b99d8a1f-000b-48ff-b4b0-7a62bc79c032-service-ca-bundle podName:b99d8a1f-000b-48ff-b4b0-7a62bc79c032 nodeName:}" failed. No retries permitted until 2026-10-30 16:20:34.340037833 +0000 UTC m=+45.640326029 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b99d8a1f-000b-48ff-b4b0-7a62bc79c032-service-ca-bundle") pod "router-default-df85cd6ff-6pd8x" (UID: "b99d8a1f-000b-48ff-b4b0-7a62bc79c032") : configmap references non-existent config key: service-ca.crt Oct 30 16:20:33.840552 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.840503 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a0cd0943-8d62-41e4-b3e7-29eaabec7277-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-sfq4m\" (UID: \"a0cd0943-8d62-41e4-b3e7-29eaabec7277\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sfq4m" Oct 30 16:20:33.840552 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.840507 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ef87558b-2ba2-41ce-9a0b-9e4b18663a6c-serving-cert\") pod \"service-ca-operator-544d749c8c-4nn7z\" (UID: \"ef87558b-2ba2-41ce-9a0b-9e4b18663a6c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-4nn7z" Oct 30 16:20:33.841612 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.841336 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/b99d8a1f-000b-48ff-b4b0-7a62bc79c032-default-certificate\") pod \"router-default-df85cd6ff-6pd8x\" (UID: \"b99d8a1f-000b-48ff-b4b0-7a62bc79c032\") " pod="openshift-ingress/router-default-df85cd6ff-6pd8x" Oct 30 16:20:33.841612 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.841361 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/b99d8a1f-000b-48ff-b4b0-7a62bc79c032-stats-auth\") pod \"router-default-df85cd6ff-6pd8x\" (UID: \"b99d8a1f-000b-48ff-b4b0-7a62bc79c032\") " pod="openshift-ingress/router-default-df85cd6ff-6pd8x" Oct 30 16:20:33.841916 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.841864 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9fafe556-9653-49df-a851-100b51bb7f47-installation-pull-secrets\") pod \"image-registry-f8dddb4c-dcbgb\" (UID: \"9fafe556-9653-49df-a851-100b51bb7f47\") " pod="openshift-image-registry/image-registry-f8dddb4c-dcbgb" Oct 30 16:20:33.841916 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.841864 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/209633e5-92f7-4a7f-9d68-3d48473ed2bf-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jrdmg\" (UID: \"209633e5-92f7-4a7f-9d68-3d48473ed2bf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jrdmg" Oct 30 16:20:33.842406 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.842389 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f79e659b-79c9-43ee-8733-c685f60e87fb-image-registry-private-configuration\") pod \"image-registry-65d65dd574-nbb2b\" (UID: \"f79e659b-79c9-43ee-8733-c685f60e87fb\") " pod="openshift-image-registry/image-registry-65d65dd574-nbb2b" Oct 30 16:20:33.842749 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.842732 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9fafe556-9653-49df-a851-100b51bb7f47-image-registry-private-configuration\") pod \"image-registry-f8dddb4c-dcbgb\" (UID: \"9fafe556-9653-49df-a851-100b51bb7f47\") " pod="openshift-image-registry/image-registry-f8dddb4c-dcbgb" Oct 30 16:20:33.842896 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.842879 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f79e659b-79c9-43ee-8733-c685f60e87fb-installation-pull-secrets\") pod \"image-registry-65d65dd574-nbb2b\" (UID: \"f79e659b-79c9-43ee-8733-c685f60e87fb\") " pod="openshift-image-registry/image-registry-65d65dd574-nbb2b" Oct 30 16:20:33.843121 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.843105 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4f0f46f4-5cda-4d67-b6f3-47df06eb844c-serving-cert\") pod \"console-operator-7cd56b8dbc-4wrf8\" (UID: \"4f0f46f4-5cda-4d67-b6f3-47df06eb844c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-4wrf8" Oct 30 16:20:33.858726 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.858699 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-cpxbb" Oct 30 16:20:33.862354 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.862319 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z65wk\" (UniqueName: \"kubernetes.io/projected/209633e5-92f7-4a7f-9d68-3d48473ed2bf-kube-api-access-z65wk\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jrdmg\" (UID: \"209633e5-92f7-4a7f-9d68-3d48473ed2bf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jrdmg" Oct 30 16:20:33.862706 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.862687 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlqc8\" (UniqueName: \"kubernetes.io/projected/ef87558b-2ba2-41ce-9a0b-9e4b18663a6c-kube-api-access-qlqc8\") pod \"service-ca-operator-544d749c8c-4nn7z\" (UID: \"ef87558b-2ba2-41ce-9a0b-9e4b18663a6c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-4nn7z" Oct 30 16:20:33.862874 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.862856 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhcx7\" (UniqueName: \"kubernetes.io/projected/b99d8a1f-000b-48ff-b4b0-7a62bc79c032-kube-api-access-bhcx7\") pod \"router-default-df85cd6ff-6pd8x\" (UID: \"b99d8a1f-000b-48ff-b4b0-7a62bc79c032\") " pod="openshift-ingress/router-default-df85cd6ff-6pd8x" Oct 30 16:20:33.864339 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.864320 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qc9ck\" (UniqueName: \"kubernetes.io/projected/774953d5-23d3-4db1-aec9-3b78ef81017d-kube-api-access-qc9ck\") pod \"ingress-canary-l4mhk\" (UID: \"774953d5-23d3-4db1-aec9-3b78ef81017d\") " pod="openshift-ingress-canary/ingress-canary-l4mhk" Oct 30 16:20:33.864501 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.864484 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjp8r\" (UniqueName: \"kubernetes.io/projected/3e90fa4d-66a8-4689-8f41-ca090d6697a9-kube-api-access-pjp8r\") pod \"dns-default-qg6t6\" (UID: \"3e90fa4d-66a8-4689-8f41-ca090d6697a9\") " pod="openshift-dns/dns-default-qg6t6" Oct 30 16:20:33.864775 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.864759 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-httvd\" (UniqueName: \"kubernetes.io/projected/9fafe556-9653-49df-a851-100b51bb7f47-kube-api-access-httvd\") pod \"image-registry-f8dddb4c-dcbgb\" (UID: \"9fafe556-9653-49df-a851-100b51bb7f47\") " pod="openshift-image-registry/image-registry-f8dddb4c-dcbgb" Oct 30 16:20:33.867293 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.867266 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f79e659b-79c9-43ee-8733-c685f60e87fb-bound-sa-token\") pod \"image-registry-65d65dd574-nbb2b\" (UID: \"f79e659b-79c9-43ee-8733-c685f60e87fb\") " pod="openshift-image-registry/image-registry-65d65dd574-nbb2b" Oct 30 16:20:33.867440 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.867412 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t54w7\" (UniqueName: \"kubernetes.io/projected/a0cd0943-8d62-41e4-b3e7-29eaabec7277-kube-api-access-t54w7\") pod \"cluster-monitoring-operator-884576c57-sfq4m\" (UID: \"a0cd0943-8d62-41e4-b3e7-29eaabec7277\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sfq4m" Oct 30 16:20:33.868735 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.868708 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-478bs\" (UniqueName: \"kubernetes.io/projected/0d304765-4b62-4cba-bad2-452771d9b201-kube-api-access-478bs\") pod \"network-check-source-6f78d59f99-nwvwc\" (UID: \"0d304765-4b62-4cba-bad2-452771d9b201\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-nwvwc" Oct 30 16:20:33.869357 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.869320 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9fafe556-9653-49df-a851-100b51bb7f47-bound-sa-token\") pod \"image-registry-f8dddb4c-dcbgb\" (UID: \"9fafe556-9653-49df-a851-100b51bb7f47\") " pod="openshift-image-registry/image-registry-f8dddb4c-dcbgb" Oct 30 16:20:33.869638 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.869613 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-849kr\" (UniqueName: \"kubernetes.io/projected/4d9a19d6-94d4-4490-8f93-89ca745b768f-kube-api-access-849kr\") pod \"volume-data-source-validator-58c9f5f46d-5fw2s\" (UID: \"4d9a19d6-94d4-4490-8f93-89ca745b768f\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-5fw2s" Oct 30 16:20:33.869736 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.869682 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qh2h\" (UniqueName: \"kubernetes.io/projected/f79e659b-79c9-43ee-8733-c685f60e87fb-kube-api-access-5qh2h\") pod \"image-registry-65d65dd574-nbb2b\" (UID: \"f79e659b-79c9-43ee-8733-c685f60e87fb\") " pod="openshift-image-registry/image-registry-65d65dd574-nbb2b" Oct 30 16:20:33.870226 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.870205 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnsg7\" (UniqueName: \"kubernetes.io/projected/4f0f46f4-5cda-4d67-b6f3-47df06eb844c-kube-api-access-bnsg7\") pod \"console-operator-7cd56b8dbc-4wrf8\" (UID: \"4f0f46f4-5cda-4d67-b6f3-47df06eb844c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-4wrf8" Oct 30 16:20:33.876372 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.876346 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-4dfsk" Oct 30 16:20:33.888694 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.888673 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-fhzbv" Oct 30 16:20:33.901421 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.901401 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-p4rk6" Oct 30 16:20:33.936714 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.936685 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-4nn7z" Oct 30 16:20:33.944029 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.943993 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-5fw2s" Oct 30 16:20:33.969778 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:33.967773 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-4wrf8" Oct 30 16:20:34.006671 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:34.003572 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jrdmg" Oct 30 16:20:34.024696 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:34.024440 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-nwvwc" Oct 30 16:20:34.039027 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:34.039001 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-l4mhk" Oct 30 16:20:34.159806 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:34.159782 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xj6cp" Oct 30 16:20:34.174738 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:34.174711 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 16:20:34.175392 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:34.175367 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-d8rcz\"" Oct 30 16:20:34.221325 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:34.221298 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-cpxbb"] Oct 30 16:20:34.226294 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:20:34.226272 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90d73ca9_5d88_4467_99fa_d4e1a67f3828.slice/crio-a9958733293e00825c710b46bc78d037c1b12987d541df429154b651e08f3e9e WatchSource:0}: Error finding container a9958733293e00825c710b46bc78d037c1b12987d541df429154b651e08f3e9e: Status 404 returned error can't find the container with id a9958733293e00825c710b46bc78d037c1b12987d541df429154b651e08f3e9e Oct 30 16:20:34.243879 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:34.243829 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/06332900-7903-4275-8e33-de06de7f6580-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-cswcz\" (UID: \"06332900-7903-4275-8e33-de06de7f6580\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-cswcz" Oct 30 16:20:34.244019 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:34.243998 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 16:20:34.244091 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:34.244078 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06332900-7903-4275-8e33-de06de7f6580-samples-operator-tls podName:06332900-7903-4275-8e33-de06de7f6580 nodeName:}" failed. No retries permitted until 2026-10-30 16:20:35.244058882 +0000 UTC m=+46.544347101 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/06332900-7903-4275-8e33-de06de7f6580-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-cswcz" (UID: "06332900-7903-4275-8e33-de06de7f6580") : secret "samples-operator-tls" not found Oct 30 16:20:34.251133 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:34.251104 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-fhzbv"] Oct 30 16:20:34.256983 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:20:34.256957 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6341cee_9ba8_41ce_bb75_29c9cb4e6d29.slice/crio-0fbe56808be3a5df04c4b47ef6cecad39b8b2bc2bf07dceacd9baed7d4592676 WatchSource:0}: Error finding container 0fbe56808be3a5df04c4b47ef6cecad39b8b2bc2bf07dceacd9baed7d4592676: Status 404 returned error can't find the container with id 0fbe56808be3a5df04c4b47ef6cecad39b8b2bc2bf07dceacd9baed7d4592676 Oct 30 16:20:34.269119 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:34.269092 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-4dfsk"] Oct 30 16:20:34.271287 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:34.271264 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-p4rk6"] Oct 30 16:20:34.272650 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:20:34.272627 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd719cbcb_9874_4b85_8fe7_e1bf53511235.slice/crio-02210e8bbafabc424bef434e3547303fff2acff99672b3416f13da5fa795721c WatchSource:0}: Error finding container 02210e8bbafabc424bef434e3547303fff2acff99672b3416f13da5fa795721c: Status 404 returned error can't find the container with id 02210e8bbafabc424bef434e3547303fff2acff99672b3416f13da5fa795721c Oct 30 16:20:34.278515 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:20:34.278496 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod154b2258_1b09_444b_ad80_2b194195e2cb.slice/crio-fd5592d8326b2be10d1373ce1a16560e5049b9a904bf066caa5d7d6dbf8dc9a6 WatchSource:0}: Error finding container fd5592d8326b2be10d1373ce1a16560e5049b9a904bf066caa5d7d6dbf8dc9a6: Status 404 returned error can't find the container with id fd5592d8326b2be10d1373ce1a16560e5049b9a904bf066caa5d7d6dbf8dc9a6 Oct 30 16:20:34.333815 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:34.333633 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-4nn7z"] Oct 30 16:20:34.339176 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:20:34.339134 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef87558b_2ba2_41ce_9a0b_9e4b18663a6c.slice/crio-40e9ed71cab3cd5a3993520de66e28ba52b20f5ea5c551646136cdcf0136c32b WatchSource:0}: Error finding container 40e9ed71cab3cd5a3993520de66e28ba52b20f5ea5c551646136cdcf0136c32b: Status 404 returned error can't find the container with id 40e9ed71cab3cd5a3993520de66e28ba52b20f5ea5c551646136cdcf0136c32b Oct 30 16:20:34.345186 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:34.345157 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f79e659b-79c9-43ee-8733-c685f60e87fb-registry-tls\") pod \"image-registry-65d65dd574-nbb2b\" (UID: \"f79e659b-79c9-43ee-8733-c685f60e87fb\") " pod="openshift-image-registry/image-registry-65d65dd574-nbb2b" Oct 30 16:20:34.345231 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:34.345197 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b99d8a1f-000b-48ff-b4b0-7a62bc79c032-service-ca-bundle\") pod \"router-default-df85cd6ff-6pd8x\" (UID: \"b99d8a1f-000b-48ff-b4b0-7a62bc79c032\") " pod="openshift-ingress/router-default-df85cd6ff-6pd8x" Oct 30 16:20:34.345267 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:34.345235 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1042378b-9f46-4e19-88e4-2628e532e8c3-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-992nb\" (UID: \"1042378b-9f46-4e19-88e4-2628e532e8c3\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-992nb" Oct 30 16:20:34.345296 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:34.345270 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a0cd0943-8d62-41e4-b3e7-29eaabec7277-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-sfq4m\" (UID: \"a0cd0943-8d62-41e4-b3e7-29eaabec7277\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sfq4m" Oct 30 16:20:34.345330 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:34.345294 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9fafe556-9653-49df-a851-100b51bb7f47-registry-tls\") pod \"image-registry-f8dddb4c-dcbgb\" (UID: \"9fafe556-9653-49df-a851-100b51bb7f47\") " pod="openshift-image-registry/image-registry-f8dddb4c-dcbgb" Oct 30 16:20:34.345330 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:34.345313 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b99d8a1f-000b-48ff-b4b0-7a62bc79c032-metrics-certs\") pod \"router-default-df85cd6ff-6pd8x\" (UID: \"b99d8a1f-000b-48ff-b4b0-7a62bc79c032\") " pod="openshift-ingress/router-default-df85cd6ff-6pd8x" Oct 30 16:20:34.345382 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:34.345328 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3e90fa4d-66a8-4689-8f41-ca090d6697a9-metrics-tls\") pod \"dns-default-qg6t6\" (UID: \"3e90fa4d-66a8-4689-8f41-ca090d6697a9\") " pod="openshift-dns/dns-default-qg6t6" Oct 30 16:20:34.345484 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:34.345471 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 16:20:34.345524 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:34.345513 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e90fa4d-66a8-4689-8f41-ca090d6697a9-metrics-tls podName:3e90fa4d-66a8-4689-8f41-ca090d6697a9 nodeName:}" failed. No retries permitted until 2026-10-30 16:20:35.345501617 +0000 UTC m=+46.645789812 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3e90fa4d-66a8-4689-8f41-ca090d6697a9-metrics-tls") pod "dns-default-qg6t6" (UID: "3e90fa4d-66a8-4689-8f41-ca090d6697a9") : secret "dns-default-metrics-tls" not found Oct 30 16:20:34.345806 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:34.345792 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 16:20:34.345806 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:34.345805 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-65d65dd574-nbb2b: secret "image-registry-tls" not found Oct 30 16:20:34.345916 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:34.345833 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f79e659b-79c9-43ee-8733-c685f60e87fb-registry-tls podName:f79e659b-79c9-43ee-8733-c685f60e87fb nodeName:}" failed. No retries permitted until 2026-10-30 16:20:35.345823302 +0000 UTC m=+46.646111511 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f79e659b-79c9-43ee-8733-c685f60e87fb-registry-tls") pod "image-registry-65d65dd574-nbb2b" (UID: "f79e659b-79c9-43ee-8733-c685f60e87fb") : secret "image-registry-tls" not found Oct 30 16:20:34.345916 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:34.345875 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b99d8a1f-000b-48ff-b4b0-7a62bc79c032-service-ca-bundle podName:b99d8a1f-000b-48ff-b4b0-7a62bc79c032 nodeName:}" failed. No retries permitted until 2026-10-30 16:20:35.345868849 +0000 UTC m=+46.646157058 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b99d8a1f-000b-48ff-b4b0-7a62bc79c032-service-ca-bundle") pod "router-default-df85cd6ff-6pd8x" (UID: "b99d8a1f-000b-48ff-b4b0-7a62bc79c032") : configmap references non-existent config key: service-ca.crt Oct 30 16:20:34.345916 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:34.345909 2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 16:20:34.346063 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:34.345929 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1042378b-9f46-4e19-88e4-2628e532e8c3-networking-console-plugin-cert podName:1042378b-9f46-4e19-88e4-2628e532e8c3 nodeName:}" failed. No retries permitted until 2026-10-30 16:20:35.345923255 +0000 UTC m=+46.646211464 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1042378b-9f46-4e19-88e4-2628e532e8c3-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-992nb" (UID: "1042378b-9f46-4e19-88e4-2628e532e8c3") : secret "networking-console-plugin-cert" not found Oct 30 16:20:34.346063 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:34.345961 2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 16:20:34.346063 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:34.345980 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0cd0943-8d62-41e4-b3e7-29eaabec7277-cluster-monitoring-operator-tls podName:a0cd0943-8d62-41e4-b3e7-29eaabec7277 nodeName:}" failed. No retries permitted until 2026-10-30 16:20:35.345974433 +0000 UTC m=+46.646262641 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a0cd0943-8d62-41e4-b3e7-29eaabec7277-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-sfq4m" (UID: "a0cd0943-8d62-41e4-b3e7-29eaabec7277") : secret "cluster-monitoring-operator-tls" not found Oct 30 16:20:34.346063 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:34.346012 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 16:20:34.346063 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:34.346019 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-f8dddb4c-dcbgb: secret "image-registry-tls" not found Oct 30 16:20:34.346063 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:34.346035 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9fafe556-9653-49df-a851-100b51bb7f47-registry-tls podName:9fafe556-9653-49df-a851-100b51bb7f47 nodeName:}" failed. No retries permitted until 2026-10-30 16:20:35.346030418 +0000 UTC m=+46.646318627 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9fafe556-9653-49df-a851-100b51bb7f47-registry-tls") pod "image-registry-f8dddb4c-dcbgb" (UID: "9fafe556-9653-49df-a851-100b51bb7f47") : secret "image-registry-tls" not found Oct 30 16:20:34.346371 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:34.346068 2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 16:20:34.346371 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:34.346084 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b99d8a1f-000b-48ff-b4b0-7a62bc79c032-metrics-certs podName:b99d8a1f-000b-48ff-b4b0-7a62bc79c032 nodeName:}" failed. No retries permitted until 2026-10-30 16:20:35.346079633 +0000 UTC m=+46.646367842 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b99d8a1f-000b-48ff-b4b0-7a62bc79c032-metrics-certs") pod "router-default-df85cd6ff-6pd8x" (UID: "b99d8a1f-000b-48ff-b4b0-7a62bc79c032") : secret "router-metrics-certs-default" not found Oct 30 16:20:34.358966 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:34.358935 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-nwvwc"] Oct 30 16:20:34.361704 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:34.361684 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-4wrf8"] Oct 30 16:20:34.364541 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:20:34.364513 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d304765_4b62_4cba_bad2_452771d9b201.slice/crio-f8e162f8c660d3797fd06f03e23a3233b9a1b7df45142ab247c7476073fce382 WatchSource:0}: Error finding container f8e162f8c660d3797fd06f03e23a3233b9a1b7df45142ab247c7476073fce382: Status 404 returned error can't find the container with id f8e162f8c660d3797fd06f03e23a3233b9a1b7df45142ab247c7476073fce382 Oct 30 16:20:34.365805 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:20:34.365777 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f0f46f4_5cda_4d67_b6f3_47df06eb844c.slice/crio-592f76c85aefce0da12f4d2efe9bea09041835fd390ba1b2cab47cd2e618bc66 WatchSource:0}: Error finding container 592f76c85aefce0da12f4d2efe9bea09041835fd390ba1b2cab47cd2e618bc66: Status 404 returned error can't find the container with id 592f76c85aefce0da12f4d2efe9bea09041835fd390ba1b2cab47cd2e618bc66 Oct 30 16:20:34.383636 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:34.383610 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-l4mhk"] Oct 30 16:20:34.389376 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:20:34.389343 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod774953d5_23d3_4db1_aec9_3b78ef81017d.slice/crio-4da5620bd44861a992d02fdba52cd60b17321f2c960afc3378c3307e3e4ce96a WatchSource:0}: Error finding container 4da5620bd44861a992d02fdba52cd60b17321f2c960afc3378c3307e3e4ce96a: Status 404 returned error can't find the container with id 4da5620bd44861a992d02fdba52cd60b17321f2c960afc3378c3307e3e4ce96a Oct 30 16:20:34.395147 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:34.395107 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-5fw2s"] Oct 30 16:20:34.398254 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:20:34.398226 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d9a19d6_94d4_4490_8f93_89ca745b768f.slice/crio-67f512b5ebf9ed2c68ab41c7daf01531b3f322cd10664fddaa19faa259b5e094 WatchSource:0}: Error finding container 67f512b5ebf9ed2c68ab41c7daf01531b3f322cd10664fddaa19faa259b5e094: Status 404 returned error can't find the container with id 67f512b5ebf9ed2c68ab41c7daf01531b3f322cd10664fddaa19faa259b5e094 Oct 30 16:20:34.402635 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:34.402612 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jrdmg"] Oct 30 16:20:34.406287 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:20:34.406265 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod209633e5_92f7_4a7f_9d68_3d48473ed2bf.slice/crio-d66366aaa32bcced3938846aed314a23cb0bb9031dc8c5c793c42a7665851493 WatchSource:0}: Error finding container d66366aaa32bcced3938846aed314a23cb0bb9031dc8c5c793c42a7665851493: Status 404 returned error can't find the container with id d66366aaa32bcced3938846aed314a23cb0bb9031dc8c5c793c42a7665851493 Oct 30 16:20:34.418042 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:34.418018 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-fhzbv" event={"ID":"a6341cee-9ba8-41ce-bb75-29c9cb4e6d29","Type":"ContainerStarted","Data":"0fbe56808be3a5df04c4b47ef6cecad39b8b2bc2bf07dceacd9baed7d4592676"} Oct 30 16:20:34.419614 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:34.419576 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-cpxbb" event={"ID":"90d73ca9-5d88-4467-99fa-d4e1a67f3828","Type":"ContainerStarted","Data":"59f3a5d1b4b5cebc4403a1607f7f4457fc253ff40884723415b2b9d80cea2460"} Oct 30 16:20:34.419707 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:34.419631 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-cpxbb" event={"ID":"90d73ca9-5d88-4467-99fa-d4e1a67f3828","Type":"ContainerStarted","Data":"a9958733293e00825c710b46bc78d037c1b12987d541df429154b651e08f3e9e"} Oct 30 16:20:34.420711 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:34.420692 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-l4mhk" event={"ID":"774953d5-23d3-4db1-aec9-3b78ef81017d","Type":"ContainerStarted","Data":"4da5620bd44861a992d02fdba52cd60b17321f2c960afc3378c3307e3e4ce96a"} Oct 30 16:20:34.421857 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:34.421839 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-5fw2s" event={"ID":"4d9a19d6-94d4-4490-8f93-89ca745b768f","Type":"ContainerStarted","Data":"67f512b5ebf9ed2c68ab41c7daf01531b3f322cd10664fddaa19faa259b5e094"} Oct 30 16:20:34.422839 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:34.422817 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-p4rk6" event={"ID":"154b2258-1b09-444b-ad80-2b194195e2cb","Type":"ContainerStarted","Data":"fd5592d8326b2be10d1373ce1a16560e5049b9a904bf066caa5d7d6dbf8dc9a6"} Oct 30 16:20:34.423809 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:34.423791 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-nwvwc" event={"ID":"0d304765-4b62-4cba-bad2-452771d9b201","Type":"ContainerStarted","Data":"f8e162f8c660d3797fd06f03e23a3233b9a1b7df45142ab247c7476073fce382"} Oct 30 16:20:34.424802 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:34.424784 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-4wrf8" event={"ID":"4f0f46f4-5cda-4d67-b6f3-47df06eb844c","Type":"ContainerStarted","Data":"592f76c85aefce0da12f4d2efe9bea09041835fd390ba1b2cab47cd2e618bc66"} Oct 30 16:20:34.425762 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:34.425743 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-4dfsk" event={"ID":"d719cbcb-9874-4b85-8fe7-e1bf53511235","Type":"ContainerStarted","Data":"02210e8bbafabc424bef434e3547303fff2acff99672b3416f13da5fa795721c"} Oct 30 16:20:34.426834 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:34.426809 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jrdmg" event={"ID":"209633e5-92f7-4a7f-9d68-3d48473ed2bf","Type":"ContainerStarted","Data":"d66366aaa32bcced3938846aed314a23cb0bb9031dc8c5c793c42a7665851493"} Oct 30 16:20:34.427863 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:34.427846 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-4nn7z" event={"ID":"ef87558b-2ba2-41ce-9a0b-9e4b18663a6c","Type":"ContainerStarted","Data":"40e9ed71cab3cd5a3993520de66e28ba52b20f5ea5c551646136cdcf0136c32b"} Oct 30 16:20:35.161844 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:35.161811 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ncfzj" Oct 30 16:20:35.162740 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:35.162266 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b4ljl" Oct 30 16:20:35.178462 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:35.178428 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-m64dw\"" Oct 30 16:20:35.178571 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:35.178549 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-46f47\"" Oct 30 16:20:35.178672 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:35.178658 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 16:20:35.254250 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:35.254216 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/06332900-7903-4275-8e33-de06de7f6580-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-cswcz\" (UID: \"06332900-7903-4275-8e33-de06de7f6580\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-cswcz" Oct 30 16:20:35.254404 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:35.254387 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 16:20:35.254466 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:35.254448 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06332900-7903-4275-8e33-de06de7f6580-samples-operator-tls podName:06332900-7903-4275-8e33-de06de7f6580 nodeName:}" failed. No retries permitted until 2026-10-30 16:20:37.254429986 +0000 UTC m=+48.554718185 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/06332900-7903-4275-8e33-de06de7f6580-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-cswcz" (UID: "06332900-7903-4275-8e33-de06de7f6580") : secret "samples-operator-tls" not found Oct 30 16:20:35.354748 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:35.354659 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f79e659b-79c9-43ee-8733-c685f60e87fb-registry-tls\") pod \"image-registry-65d65dd574-nbb2b\" (UID: \"f79e659b-79c9-43ee-8733-c685f60e87fb\") " pod="openshift-image-registry/image-registry-65d65dd574-nbb2b" Oct 30 16:20:35.354748 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:35.354699 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b99d8a1f-000b-48ff-b4b0-7a62bc79c032-service-ca-bundle\") pod \"router-default-df85cd6ff-6pd8x\" (UID: \"b99d8a1f-000b-48ff-b4b0-7a62bc79c032\") " pod="openshift-ingress/router-default-df85cd6ff-6pd8x" Oct 30 16:20:35.354748 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:35.354722 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1042378b-9f46-4e19-88e4-2628e532e8c3-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-992nb\" (UID: \"1042378b-9f46-4e19-88e4-2628e532e8c3\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-992nb" Oct 30 16:20:35.354748 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:35.354742 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a0cd0943-8d62-41e4-b3e7-29eaabec7277-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-sfq4m\" (UID: \"a0cd0943-8d62-41e4-b3e7-29eaabec7277\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sfq4m" Oct 30 16:20:35.354748 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:35.354760 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9fafe556-9653-49df-a851-100b51bb7f47-registry-tls\") pod \"image-registry-f8dddb4c-dcbgb\" (UID: \"9fafe556-9653-49df-a851-100b51bb7f47\") " pod="openshift-image-registry/image-registry-f8dddb4c-dcbgb" Oct 30 16:20:35.355103 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:35.354776 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b99d8a1f-000b-48ff-b4b0-7a62bc79c032-metrics-certs\") pod \"router-default-df85cd6ff-6pd8x\" (UID: \"b99d8a1f-000b-48ff-b4b0-7a62bc79c032\") " pod="openshift-ingress/router-default-df85cd6ff-6pd8x" Oct 30 16:20:35.355103 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:35.354797 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3e90fa4d-66a8-4689-8f41-ca090d6697a9-metrics-tls\") pod \"dns-default-qg6t6\" (UID: \"3e90fa4d-66a8-4689-8f41-ca090d6697a9\") " pod="openshift-dns/dns-default-qg6t6" Oct 30 16:20:35.355103 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:35.354930 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 16:20:35.355103 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:35.354976 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e90fa4d-66a8-4689-8f41-ca090d6697a9-metrics-tls podName:3e90fa4d-66a8-4689-8f41-ca090d6697a9 nodeName:}" failed. No retries permitted until 2026-10-30 16:20:37.354964068 +0000 UTC m=+48.655252263 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3e90fa4d-66a8-4689-8f41-ca090d6697a9-metrics-tls") pod "dns-default-qg6t6" (UID: "3e90fa4d-66a8-4689-8f41-ca090d6697a9") : secret "dns-default-metrics-tls" not found Oct 30 16:20:35.355373 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:35.355355 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 16:20:35.355432 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:35.355378 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-65d65dd574-nbb2b: secret "image-registry-tls" not found Oct 30 16:20:35.355432 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:35.355427 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f79e659b-79c9-43ee-8733-c685f60e87fb-registry-tls podName:f79e659b-79c9-43ee-8733-c685f60e87fb nodeName:}" failed. No retries permitted until 2026-10-30 16:20:37.35540908 +0000 UTC m=+48.655697284 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f79e659b-79c9-43ee-8733-c685f60e87fb-registry-tls") pod "image-registry-65d65dd574-nbb2b" (UID: "f79e659b-79c9-43ee-8733-c685f60e87fb") : secret "image-registry-tls" not found Oct 30 16:20:35.355531 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:35.355524 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b99d8a1f-000b-48ff-b4b0-7a62bc79c032-service-ca-bundle podName:b99d8a1f-000b-48ff-b4b0-7a62bc79c032 nodeName:}" failed. No retries permitted until 2026-10-30 16:20:37.355512919 +0000 UTC m=+48.655801116 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b99d8a1f-000b-48ff-b4b0-7a62bc79c032-service-ca-bundle") pod "router-default-df85cd6ff-6pd8x" (UID: "b99d8a1f-000b-48ff-b4b0-7a62bc79c032") : configmap references non-existent config key: service-ca.crt Oct 30 16:20:35.355601 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:35.355594 2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 16:20:35.355649 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:35.355633 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1042378b-9f46-4e19-88e4-2628e532e8c3-networking-console-plugin-cert podName:1042378b-9f46-4e19-88e4-2628e532e8c3 nodeName:}" failed. No retries permitted until 2026-10-30 16:20:37.355618724 +0000 UTC m=+48.655906921 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1042378b-9f46-4e19-88e4-2628e532e8c3-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-992nb" (UID: "1042378b-9f46-4e19-88e4-2628e532e8c3") : secret "networking-console-plugin-cert" not found Oct 30 16:20:35.355711 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:35.355688 2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 16:20:35.355762 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:35.355719 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0cd0943-8d62-41e4-b3e7-29eaabec7277-cluster-monitoring-operator-tls podName:a0cd0943-8d62-41e4-b3e7-29eaabec7277 nodeName:}" failed. No retries permitted until 2026-10-30 16:20:37.35570885 +0000 UTC m=+48.655997051 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a0cd0943-8d62-41e4-b3e7-29eaabec7277-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-sfq4m" (UID: "a0cd0943-8d62-41e4-b3e7-29eaabec7277") : secret "cluster-monitoring-operator-tls" not found Oct 30 16:20:35.355812 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:35.355774 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 16:20:35.355812 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:35.355783 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-f8dddb4c-dcbgb: secret "image-registry-tls" not found Oct 30 16:20:35.355812 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:35.355811 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9fafe556-9653-49df-a851-100b51bb7f47-registry-tls podName:9fafe556-9653-49df-a851-100b51bb7f47 nodeName:}" failed. No retries permitted until 2026-10-30 16:20:37.355801695 +0000 UTC m=+48.656089898 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9fafe556-9653-49df-a851-100b51bb7f47-registry-tls") pod "image-registry-f8dddb4c-dcbgb" (UID: "9fafe556-9653-49df-a851-100b51bb7f47") : secret "image-registry-tls" not found Oct 30 16:20:35.355956 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:35.355862 2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 16:20:35.355956 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:35.355892 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b99d8a1f-000b-48ff-b4b0-7a62bc79c032-metrics-certs podName:b99d8a1f-000b-48ff-b4b0-7a62bc79c032 nodeName:}" failed. No retries permitted until 2026-10-30 16:20:37.355882046 +0000 UTC m=+48.656170243 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b99d8a1f-000b-48ff-b4b0-7a62bc79c032-metrics-certs") pod "router-default-df85cd6ff-6pd8x" (UID: "b99d8a1f-000b-48ff-b4b0-7a62bc79c032") : secret "router-metrics-certs-default" not found Oct 30 16:20:35.438351 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:35.438271 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-fhzbv" event={"ID":"a6341cee-9ba8-41ce-bb75-29c9cb4e6d29","Type":"ContainerStarted","Data":"73fe252eed59f1cae3ed4c9ddf13bd1162ebe5aa162eda5219eeaa27629851a5"} Oct 30 16:20:35.442240 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:35.442202 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-4dfsk" event={"ID":"d719cbcb-9874-4b85-8fe7-e1bf53511235","Type":"ContainerStarted","Data":"65d5ec36c522c30bab495204ba2b3c5b57b213766a18562ae129b1d41c73e722"} Oct 30 16:20:35.520912 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:35.520704 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-cpxbb" podStartSLOduration=232.520686453 podStartE2EDuration="3m52.520686453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:16:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:20:35.519319155 +0000 UTC m=+46.819607375" watchObservedRunningTime="2026-10-30 16:20:35.520686453 +0000 UTC m=+46.820974672" Oct 30 16:20:35.521183 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:35.521143 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-fhzbv" podStartSLOduration=232.521131317 podStartE2EDuration="3m52.521131317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:16:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:20:35.479550057 +0000 UTC m=+46.779838486" watchObservedRunningTime="2026-10-30 16:20:35.521131317 +0000 UTC m=+46.821419529" Oct 30 16:20:35.560718 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:35.560662 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-4dfsk" podStartSLOduration=232.560647859 podStartE2EDuration="3m52.560647859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:16:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:20:35.55999552 +0000 UTC m=+46.860283739" watchObservedRunningTime="2026-10-30 16:20:35.560647859 +0000 UTC m=+46.860936103" Oct 30 16:20:37.284006 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:37.283964 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/06332900-7903-4275-8e33-de06de7f6580-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-cswcz\" (UID: \"06332900-7903-4275-8e33-de06de7f6580\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-cswcz" Oct 30 16:20:37.284459 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:37.284286 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 16:20:37.284459 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:37.284355 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06332900-7903-4275-8e33-de06de7f6580-samples-operator-tls podName:06332900-7903-4275-8e33-de06de7f6580 nodeName:}" failed. No retries permitted until 2026-10-30 16:20:41.284335849 +0000 UTC m=+52.584624064 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/06332900-7903-4275-8e33-de06de7f6580-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-cswcz" (UID: "06332900-7903-4275-8e33-de06de7f6580") : secret "samples-operator-tls" not found Oct 30 16:20:37.384851 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:37.384813 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f79e659b-79c9-43ee-8733-c685f60e87fb-registry-tls\") pod \"image-registry-65d65dd574-nbb2b\" (UID: \"f79e659b-79c9-43ee-8733-c685f60e87fb\") " pod="openshift-image-registry/image-registry-65d65dd574-nbb2b" Oct 30 16:20:37.385033 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:37.384860 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b99d8a1f-000b-48ff-b4b0-7a62bc79c032-service-ca-bundle\") pod \"router-default-df85cd6ff-6pd8x\" (UID: \"b99d8a1f-000b-48ff-b4b0-7a62bc79c032\") " pod="openshift-ingress/router-default-df85cd6ff-6pd8x" Oct 30 16:20:37.385033 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:37.384894 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1042378b-9f46-4e19-88e4-2628e532e8c3-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-992nb\" (UID: \"1042378b-9f46-4e19-88e4-2628e532e8c3\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-992nb" Oct 30 16:20:37.385033 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:37.384920 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a0cd0943-8d62-41e4-b3e7-29eaabec7277-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-sfq4m\" (UID: \"a0cd0943-8d62-41e4-b3e7-29eaabec7277\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sfq4m" Oct 30 16:20:37.385033 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:37.384947 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9fafe556-9653-49df-a851-100b51bb7f47-registry-tls\") pod \"image-registry-f8dddb4c-dcbgb\" (UID: \"9fafe556-9653-49df-a851-100b51bb7f47\") " pod="openshift-image-registry/image-registry-f8dddb4c-dcbgb" Oct 30 16:20:37.385033 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:37.384971 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b99d8a1f-000b-48ff-b4b0-7a62bc79c032-metrics-certs\") pod \"router-default-df85cd6ff-6pd8x\" (UID: \"b99d8a1f-000b-48ff-b4b0-7a62bc79c032\") " pod="openshift-ingress/router-default-df85cd6ff-6pd8x" Oct 30 16:20:37.385033 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:37.384994 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3e90fa4d-66a8-4689-8f41-ca090d6697a9-metrics-tls\") pod \"dns-default-qg6t6\" (UID: \"3e90fa4d-66a8-4689-8f41-ca090d6697a9\") " pod="openshift-dns/dns-default-qg6t6" Oct 30 16:20:37.385297 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:37.385177 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 16:20:37.385297 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:37.385240 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e90fa4d-66a8-4689-8f41-ca090d6697a9-metrics-tls podName:3e90fa4d-66a8-4689-8f41-ca090d6697a9 nodeName:}" failed. No retries permitted until 2026-10-30 16:20:41.385222317 +0000 UTC m=+52.685510536 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3e90fa4d-66a8-4689-8f41-ca090d6697a9-metrics-tls") pod "dns-default-qg6t6" (UID: "3e90fa4d-66a8-4689-8f41-ca090d6697a9") : secret "dns-default-metrics-tls" not found Oct 30 16:20:37.385720 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:37.385689 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 16:20:37.385720 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:37.385709 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-65d65dd574-nbb2b: secret "image-registry-tls" not found Oct 30 16:20:37.385889 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:37.385754 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f79e659b-79c9-43ee-8733-c685f60e87fb-registry-tls podName:f79e659b-79c9-43ee-8733-c685f60e87fb nodeName:}" failed. No retries permitted until 2026-10-30 16:20:41.385739121 +0000 UTC m=+52.686027333 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f79e659b-79c9-43ee-8733-c685f60e87fb-registry-tls") pod "image-registry-65d65dd574-nbb2b" (UID: "f79e659b-79c9-43ee-8733-c685f60e87fb") : secret "image-registry-tls" not found Oct 30 16:20:37.385889 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:37.385824 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b99d8a1f-000b-48ff-b4b0-7a62bc79c032-service-ca-bundle podName:b99d8a1f-000b-48ff-b4b0-7a62bc79c032 nodeName:}" failed. No retries permitted until 2026-10-30 16:20:41.385813684 +0000 UTC m=+52.686101894 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b99d8a1f-000b-48ff-b4b0-7a62bc79c032-service-ca-bundle") pod "router-default-df85cd6ff-6pd8x" (UID: "b99d8a1f-000b-48ff-b4b0-7a62bc79c032") : configmap references non-existent config key: service-ca.crt Oct 30 16:20:37.385889 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:37.385888 2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 16:20:37.386061 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:37.385921 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1042378b-9f46-4e19-88e4-2628e532e8c3-networking-console-plugin-cert podName:1042378b-9f46-4e19-88e4-2628e532e8c3 nodeName:}" failed. No retries permitted until 2026-10-30 16:20:41.385910149 +0000 UTC m=+52.686198364 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1042378b-9f46-4e19-88e4-2628e532e8c3-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-992nb" (UID: "1042378b-9f46-4e19-88e4-2628e532e8c3") : secret "networking-console-plugin-cert" not found Oct 30 16:20:37.386061 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:37.385975 2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 16:20:37.386061 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:37.386009 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0cd0943-8d62-41e4-b3e7-29eaabec7277-cluster-monitoring-operator-tls podName:a0cd0943-8d62-41e4-b3e7-29eaabec7277 nodeName:}" failed. No retries permitted until 2026-10-30 16:20:41.385998773 +0000 UTC m=+52.686286987 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a0cd0943-8d62-41e4-b3e7-29eaabec7277-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-sfq4m" (UID: "a0cd0943-8d62-41e4-b3e7-29eaabec7277") : secret "cluster-monitoring-operator-tls" not found Oct 30 16:20:37.386230 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:37.386070 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 16:20:37.386230 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:37.386081 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-f8dddb4c-dcbgb: secret "image-registry-tls" not found Oct 30 16:20:37.386230 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:37.386121 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9fafe556-9653-49df-a851-100b51bb7f47-registry-tls podName:9fafe556-9653-49df-a851-100b51bb7f47 nodeName:}" failed. No retries permitted until 2026-10-30 16:20:41.386111194 +0000 UTC m=+52.686399405 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9fafe556-9653-49df-a851-100b51bb7f47-registry-tls") pod "image-registry-f8dddb4c-dcbgb" (UID: "9fafe556-9653-49df-a851-100b51bb7f47") : secret "image-registry-tls" not found Oct 30 16:20:37.386230 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:37.386178 2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 16:20:37.386230 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:37.386207 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b99d8a1f-000b-48ff-b4b0-7a62bc79c032-metrics-certs podName:b99d8a1f-000b-48ff-b4b0-7a62bc79c032 nodeName:}" failed. No retries permitted until 2026-10-30 16:20:41.386196898 +0000 UTC m=+52.686485108 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b99d8a1f-000b-48ff-b4b0-7a62bc79c032-metrics-certs") pod "router-default-df85cd6ff-6pd8x" (UID: "b99d8a1f-000b-48ff-b4b0-7a62bc79c032") : secret "router-metrics-certs-default" not found Oct 30 16:20:37.461453 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:37.461409 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-5fw2s" event={"ID":"4d9a19d6-94d4-4490-8f93-89ca745b768f","Type":"ContainerStarted","Data":"cdfa6ed212090b654ee9b5c53360964374e1079ab527a6da9a1b018cf2aabaee"} Oct 30 16:20:38.468219 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:38.468189 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4wrf8_4f0f46f4-5cda-4d67-b6f3-47df06eb844c/console-operator/0.log" Oct 30 16:20:38.468749 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:38.468228 2580 generic.go:356] "Generic (PLEG): container finished" podID="4f0f46f4-5cda-4d67-b6f3-47df06eb844c" containerID="a422c6a0094a99fbc3b5024180e2f73c70f5438428ed7908ce5c44d62b96a90d" exitCode=255 Oct 30 16:20:38.468749 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:38.468315 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-4wrf8" event={"ID":"4f0f46f4-5cda-4d67-b6f3-47df06eb844c","Type":"ContainerDied","Data":"a422c6a0094a99fbc3b5024180e2f73c70f5438428ed7908ce5c44d62b96a90d"} Oct 30 16:20:38.468749 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:38.468704 2580 scope.go:122] "RemoveContainer" containerID="a422c6a0094a99fbc3b5024180e2f73c70f5438428ed7908ce5c44d62b96a90d" Oct 30 16:20:38.471133 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:38.471077 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jrdmg" event={"ID":"209633e5-92f7-4a7f-9d68-3d48473ed2bf","Type":"ContainerStarted","Data":"7dcb1eee8cb8e4fb65b44247f158d5e763106751ffdaf2f79781b2f7a41d68fd"} Oct 30 16:20:38.473121 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:38.473074 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-p4rk6" event={"ID":"154b2258-1b09-444b-ad80-2b194195e2cb","Type":"ContainerStarted","Data":"3a438e647b652e073cebbfa3cc04eb65dfbbd23b0bc9def2f41b113a0855b5ad"} Oct 30 16:20:38.511605 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:38.511251 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-5fw2s" podStartSLOduration=203.421826699 podStartE2EDuration="3m25.511231023s" totalImagesPullingTime="2.089404324s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:17:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:20:37.501917698 +0000 UTC m=+48.802205918" watchObservedRunningTime="2026-10-30 16:20:38.511231023 +0000 UTC m=+49.811519246" Oct 30 16:20:38.553381 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:38.553328 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-p4rk6" podStartSLOduration=201.904154875 podStartE2EDuration="3m25.553309896s" totalImagesPullingTime="3.649155021s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:17:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:20:38.552786469 +0000 UTC m=+49.853074713" watchObservedRunningTime="2026-10-30 16:20:38.553309896 +0000 UTC m=+49.853598130" Oct 30 16:20:38.616374 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:38.616309 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jrdmg" podStartSLOduration=201.774969378 podStartE2EDuration="3m25.61627373s" totalImagesPullingTime="3.841304352s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:17:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:20:38.614353164 +0000 UTC m=+49.914641383" watchObservedRunningTime="2026-10-30 16:20:38.61627373 +0000 UTC m=+49.916561950" Oct 30 16:20:38.892376 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:38.892335 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-wxvph"] Oct 30 16:20:38.897339 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:38.897317 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wxvph" Oct 30 16:20:38.912444 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:38.912419 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 16:20:38.912557 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:38.912542 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-6q4zp\"" Oct 30 16:20:38.912652 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:38.912618 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 16:20:38.920663 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:38.920592 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wxvph"] Oct 30 16:20:38.956930 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:38.956899 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-cwrtz-glm8n"] Oct 30 16:20:38.960076 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:38.960058 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-cwrtz-glm8n" Oct 30 16:20:38.983666 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:38.982496 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-cwrtz-glm8n"] Oct 30 16:20:39.008653 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:39.008617 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d6db39f8-bf92-44e8-9144-51eb3b9dee8d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wxvph\" (UID: \"d6db39f8-bf92-44e8-9144-51eb3b9dee8d\") " pod="openshift-insights/insights-runtime-extractor-wxvph" Oct 30 16:20:39.009274 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:39.009077 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c471cf39-07e7-4f4d-87d5-0c6803fd54a0-archives-path\") pod \"periodic-gathering-cwrtz-glm8n\" (UID: \"c471cf39-07e7-4f4d-87d5-0c6803fd54a0\") " pod="openshift-insights/periodic-gathering-cwrtz-glm8n" Oct 30 16:20:39.009744 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:39.009725 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c471cf39-07e7-4f4d-87d5-0c6803fd54a0-trusted-ca-bundle\") pod \"periodic-gathering-cwrtz-glm8n\" (UID: \"c471cf39-07e7-4f4d-87d5-0c6803fd54a0\") " pod="openshift-insights/periodic-gathering-cwrtz-glm8n" Oct 30 16:20:39.010666 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:39.010645 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f3d36d6-5098-43cc-8f8a-bd83d5dea5fc-metrics-certs\") pod \"network-metrics-daemon-xj6cp\" (UID: \"6f3d36d6-5098-43cc-8f8a-bd83d5dea5fc\") " pod="openshift-multus/network-metrics-daemon-xj6cp" Oct 30 16:20:39.012134 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:39.011361 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 16:20:39.013533 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:39.013376 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-2g8kz\"" Oct 30 16:20:39.014952 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:39.014937 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f3d36d6-5098-43cc-8f8a-bd83d5dea5fc-metrics-certs podName:6f3d36d6-5098-43cc-8f8a-bd83d5dea5fc nodeName:}" failed. No retries permitted until 2026-10-30 16:21:11.014909814 +0000 UTC m=+82.315198024 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6f3d36d6-5098-43cc-8f8a-bd83d5dea5fc-metrics-certs") pod "network-metrics-daemon-xj6cp" (UID: "6f3d36d6-5098-43cc-8f8a-bd83d5dea5fc") : secret "metrics-daemon-secret" not found Oct 30 16:20:39.015672 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:39.015652 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c471cf39-07e7-4f4d-87d5-0c6803fd54a0-service-ca-bundle\") pod \"periodic-gathering-cwrtz-glm8n\" (UID: \"c471cf39-07e7-4f4d-87d5-0c6803fd54a0\") " pod="openshift-insights/periodic-gathering-cwrtz-glm8n" Oct 30 16:20:39.015953 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:39.015935 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwfbm\" (UniqueName: \"kubernetes.io/projected/d6db39f8-bf92-44e8-9144-51eb3b9dee8d-kube-api-access-cwfbm\") pod \"insights-runtime-extractor-wxvph\" (UID: \"d6db39f8-bf92-44e8-9144-51eb3b9dee8d\") " pod="openshift-insights/insights-runtime-extractor-wxvph" Oct 30 16:20:39.016173 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:39.016157 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d6db39f8-bf92-44e8-9144-51eb3b9dee8d-data-volume\") pod \"insights-runtime-extractor-wxvph\" (UID: \"d6db39f8-bf92-44e8-9144-51eb3b9dee8d\") " pod="openshift-insights/insights-runtime-extractor-wxvph" Oct 30 16:20:39.016365 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:39.016349 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d6db39f8-bf92-44e8-9144-51eb3b9dee8d-crio-socket\") pod \"insights-runtime-extractor-wxvph\" (UID: \"d6db39f8-bf92-44e8-9144-51eb3b9dee8d\") " pod="openshift-insights/insights-runtime-extractor-wxvph" Oct 30 16:20:39.016569 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:39.016553 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mc2zh\" (UniqueName: \"kubernetes.io/projected/c471cf39-07e7-4f4d-87d5-0c6803fd54a0-kube-api-access-mc2zh\") pod \"periodic-gathering-cwrtz-glm8n\" (UID: \"c471cf39-07e7-4f4d-87d5-0c6803fd54a0\") " pod="openshift-insights/periodic-gathering-cwrtz-glm8n" Oct 30 16:20:39.016772 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:39.016756 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/029f4ecb-eb6b-4693-9e9e-ca61f554dd33-original-pull-secret\") pod \"global-pull-secret-syncer-ncfzj\" (UID: \"029f4ecb-eb6b-4693-9e9e-ca61f554dd33\") " pod="kube-system/global-pull-secret-syncer-ncfzj" Oct 30 16:20:39.017865 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:39.017847 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d6db39f8-bf92-44e8-9144-51eb3b9dee8d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wxvph\" (UID: \"d6db39f8-bf92-44e8-9144-51eb3b9dee8d\") " pod="openshift-insights/insights-runtime-extractor-wxvph" Oct 30 16:20:39.022621 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:39.022602 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/029f4ecb-eb6b-4693-9e9e-ca61f554dd33-original-pull-secret\") pod \"global-pull-secret-syncer-ncfzj\" (UID: \"029f4ecb-eb6b-4693-9e9e-ca61f554dd33\") " pod="kube-system/global-pull-secret-syncer-ncfzj" Oct 30 16:20:39.099455 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:39.099210 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ncfzj" Oct 30 16:20:39.119638 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:39.118675 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mc2zh\" (UniqueName: \"kubernetes.io/projected/c471cf39-07e7-4f4d-87d5-0c6803fd54a0-kube-api-access-mc2zh\") pod \"periodic-gathering-cwrtz-glm8n\" (UID: \"c471cf39-07e7-4f4d-87d5-0c6803fd54a0\") " pod="openshift-insights/periodic-gathering-cwrtz-glm8n" Oct 30 16:20:39.119638 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:39.118730 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d6db39f8-bf92-44e8-9144-51eb3b9dee8d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wxvph\" (UID: \"d6db39f8-bf92-44e8-9144-51eb3b9dee8d\") " pod="openshift-insights/insights-runtime-extractor-wxvph" Oct 30 16:20:39.119638 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:39.118767 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d6db39f8-bf92-44e8-9144-51eb3b9dee8d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wxvph\" (UID: \"d6db39f8-bf92-44e8-9144-51eb3b9dee8d\") " pod="openshift-insights/insights-runtime-extractor-wxvph" Oct 30 16:20:39.119638 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:39.118806 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c471cf39-07e7-4f4d-87d5-0c6803fd54a0-archives-path\") pod \"periodic-gathering-cwrtz-glm8n\" (UID: \"c471cf39-07e7-4f4d-87d5-0c6803fd54a0\") " pod="openshift-insights/periodic-gathering-cwrtz-glm8n" Oct 30 16:20:39.119638 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:39.118828 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c471cf39-07e7-4f4d-87d5-0c6803fd54a0-trusted-ca-bundle\") pod \"periodic-gathering-cwrtz-glm8n\" (UID: \"c471cf39-07e7-4f4d-87d5-0c6803fd54a0\") " pod="openshift-insights/periodic-gathering-cwrtz-glm8n" Oct 30 16:20:39.119638 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:39.118870 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljv94\" (UniqueName: \"kubernetes.io/projected/2d1ed7d8-37cc-4652-b225-3a715b755e48-kube-api-access-ljv94\") pod \"network-check-target-b4ljl\" (UID: \"2d1ed7d8-37cc-4652-b225-3a715b755e48\") " pod="openshift-network-diagnostics/network-check-target-b4ljl" Oct 30 16:20:39.119638 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:39.118927 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c471cf39-07e7-4f4d-87d5-0c6803fd54a0-service-ca-bundle\") pod \"periodic-gathering-cwrtz-glm8n\" (UID: \"c471cf39-07e7-4f4d-87d5-0c6803fd54a0\") " pod="openshift-insights/periodic-gathering-cwrtz-glm8n" Oct 30 16:20:39.119638 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:39.119010 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwfbm\" (UniqueName: \"kubernetes.io/projected/d6db39f8-bf92-44e8-9144-51eb3b9dee8d-kube-api-access-cwfbm\") pod \"insights-runtime-extractor-wxvph\" (UID: \"d6db39f8-bf92-44e8-9144-51eb3b9dee8d\") " pod="openshift-insights/insights-runtime-extractor-wxvph" Oct 30 16:20:39.119638 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:39.119060 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d6db39f8-bf92-44e8-9144-51eb3b9dee8d-data-volume\") pod \"insights-runtime-extractor-wxvph\" (UID: \"d6db39f8-bf92-44e8-9144-51eb3b9dee8d\") " pod="openshift-insights/insights-runtime-extractor-wxvph" Oct 30 16:20:39.119638 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:39.119082 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d6db39f8-bf92-44e8-9144-51eb3b9dee8d-crio-socket\") pod \"insights-runtime-extractor-wxvph\" (UID: \"d6db39f8-bf92-44e8-9144-51eb3b9dee8d\") " pod="openshift-insights/insights-runtime-extractor-wxvph" Oct 30 16:20:39.119638 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:39.119314 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d6db39f8-bf92-44e8-9144-51eb3b9dee8d-crio-socket\") pod \"insights-runtime-extractor-wxvph\" (UID: \"d6db39f8-bf92-44e8-9144-51eb3b9dee8d\") " pod="openshift-insights/insights-runtime-extractor-wxvph" Oct 30 16:20:39.120724 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:39.120252 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c471cf39-07e7-4f4d-87d5-0c6803fd54a0-trusted-ca-bundle\") pod \"periodic-gathering-cwrtz-glm8n\" (UID: \"c471cf39-07e7-4f4d-87d5-0c6803fd54a0\") " pod="openshift-insights/periodic-gathering-cwrtz-glm8n" Oct 30 16:20:39.120724 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:39.120308 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 16:20:39.120724 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:39.120350 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d6db39f8-bf92-44e8-9144-51eb3b9dee8d-kube-rbac-proxy-cm podName:d6db39f8-bf92-44e8-9144-51eb3b9dee8d nodeName:}" failed. No retries permitted until 2026-10-30 16:20:39.620337403 +0000 UTC m=+50.920625603 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d6db39f8-bf92-44e8-9144-51eb3b9dee8d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wxvph" (UID: "d6db39f8-bf92-44e8-9144-51eb3b9dee8d") : configmap "kube-rbac-proxy" not found Oct 30 16:20:39.120724 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:39.120535 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 16:20:39.120724 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:39.120565 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6db39f8-bf92-44e8-9144-51eb3b9dee8d-insights-runtime-extractor-tls podName:d6db39f8-bf92-44e8-9144-51eb3b9dee8d nodeName:}" failed. No retries permitted until 2026-10-30 16:20:39.620556251 +0000 UTC m=+50.920844447 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d6db39f8-bf92-44e8-9144-51eb3b9dee8d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wxvph" (UID: "d6db39f8-bf92-44e8-9144-51eb3b9dee8d") : secret "insights-runtime-extractor-tls" not found Oct 30 16:20:39.121385 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:39.120854 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c471cf39-07e7-4f4d-87d5-0c6803fd54a0-service-ca-bundle\") pod \"periodic-gathering-cwrtz-glm8n\" (UID: \"c471cf39-07e7-4f4d-87d5-0c6803fd54a0\") " pod="openshift-insights/periodic-gathering-cwrtz-glm8n" Oct 30 16:20:39.121385 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:39.120867 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c471cf39-07e7-4f4d-87d5-0c6803fd54a0-archives-path\") pod \"periodic-gathering-cwrtz-glm8n\" (UID: \"c471cf39-07e7-4f4d-87d5-0c6803fd54a0\") " pod="openshift-insights/periodic-gathering-cwrtz-glm8n" Oct 30 16:20:39.121385 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:39.121146 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d6db39f8-bf92-44e8-9144-51eb3b9dee8d-data-volume\") pod \"insights-runtime-extractor-wxvph\" (UID: \"d6db39f8-bf92-44e8-9144-51eb3b9dee8d\") " pod="openshift-insights/insights-runtime-extractor-wxvph" Oct 30 16:20:39.125122 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:39.125099 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljv94\" (UniqueName: \"kubernetes.io/projected/2d1ed7d8-37cc-4652-b225-3a715b755e48-kube-api-access-ljv94\") pod \"network-check-target-b4ljl\" (UID: \"2d1ed7d8-37cc-4652-b225-3a715b755e48\") " pod="openshift-network-diagnostics/network-check-target-b4ljl" Oct 30 16:20:39.144492 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:39.144405 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mc2zh\" (UniqueName: \"kubernetes.io/projected/c471cf39-07e7-4f4d-87d5-0c6803fd54a0-kube-api-access-mc2zh\") pod \"periodic-gathering-cwrtz-glm8n\" (UID: \"c471cf39-07e7-4f4d-87d5-0c6803fd54a0\") " pod="openshift-insights/periodic-gathering-cwrtz-glm8n" Oct 30 16:20:39.145403 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:39.145360 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwfbm\" (UniqueName: \"kubernetes.io/projected/d6db39f8-bf92-44e8-9144-51eb3b9dee8d-kube-api-access-cwfbm\") pod \"insights-runtime-extractor-wxvph\" (UID: \"d6db39f8-bf92-44e8-9144-51eb3b9dee8d\") " pod="openshift-insights/insights-runtime-extractor-wxvph" Oct 30 16:20:39.283198 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:39.283167 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-cwrtz-glm8n" Oct 30 16:20:39.352878 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:39.352852 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-c87fl"] Oct 30 16:20:39.362519 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:39.359986 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-c87fl" Oct 30 16:20:39.368753 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:39.367474 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-vgrwl"] Oct 30 16:20:39.371011 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:39.370870 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vgrwl" Oct 30 16:20:39.372695 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:39.372675 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-c87fl"] Oct 30 16:20:39.384108 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:39.384085 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 30 16:20:39.384108 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:39.384102 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-rd9lc\"" Oct 30 16:20:39.384269 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:39.384092 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 30 16:20:39.399860 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:39.399770 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-vgrwl"] Oct 30 16:20:39.406495 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:39.406471 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b4ljl" Oct 30 16:20:39.417518 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:39.412054 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ncfzj"] Oct 30 16:20:39.417717 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:20:39.417685 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod029f4ecb_eb6b_4693_9e9e_ca61f554dd33.slice/crio-4c26f010622ac6418ada166fa96c536ff64a6c2031ad2ca56fcf154a791c277c WatchSource:0}: Error finding container 4c26f010622ac6418ada166fa96c536ff64a6c2031ad2ca56fcf154a791c277c: Status 404 returned error can't find the container with id 4c26f010622ac6418ada166fa96c536ff64a6c2031ad2ca56fcf154a791c277c Oct 30 16:20:39.422647 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:39.422623 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttr7v\" (UniqueName: \"kubernetes.io/projected/341d9139-495e-4ee2-80c3-a20370039948-kube-api-access-ttr7v\") pod \"migrator-66df5d496b-c87fl\" (UID: \"341d9139-495e-4ee2-80c3-a20370039948\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-c87fl" Oct 30 16:20:39.480426 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:39.480321 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-4nn7z" event={"ID":"ef87558b-2ba2-41ce-9a0b-9e4b18663a6c","Type":"ContainerStarted","Data":"c8238ea199508356b3cbc4176329692e250298b0a3db3299e40ee4b66076624e"} Oct 30 16:20:39.483231 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:39.483141 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-ncfzj" event={"ID":"029f4ecb-eb6b-4693-9e9e-ca61f554dd33","Type":"ContainerStarted","Data":"4c26f010622ac6418ada166fa96c536ff64a6c2031ad2ca56fcf154a791c277c"} Oct 30 16:20:39.489793 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:39.489746 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-l4mhk" event={"ID":"774953d5-23d3-4db1-aec9-3b78ef81017d","Type":"ContainerStarted","Data":"30bbd2e54db70cdf52e0db565e2187478f35a4bd59e4c0bddef8c85b12ef11c3"} Oct 30 16:20:39.492542 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:39.492519 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4wrf8_4f0f46f4-5cda-4d67-b6f3-47df06eb844c/console-operator/1.log" Oct 30 16:20:39.493067 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:39.493050 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4wrf8_4f0f46f4-5cda-4d67-b6f3-47df06eb844c/console-operator/0.log" Oct 30 16:20:39.493109 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:39.493087 2580 generic.go:356] "Generic (PLEG): container finished" podID="4f0f46f4-5cda-4d67-b6f3-47df06eb844c" containerID="1a368df4208847e057012340c138c128bc78134c8e9c801e988b863ea39e70fd" exitCode=255 Oct 30 16:20:39.493890 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:39.493873 2580 scope.go:122] "RemoveContainer" containerID="1a368df4208847e057012340c138c128bc78134c8e9c801e988b863ea39e70fd" Oct 30 16:20:39.494076 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:39.494054 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-4wrf8_openshift-console-operator(4f0f46f4-5cda-4d67-b6f3-47df06eb844c)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-4wrf8" podUID="4f0f46f4-5cda-4d67-b6f3-47df06eb844c" Oct 30 16:20:39.494141 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:39.494114 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-4wrf8" event={"ID":"4f0f46f4-5cda-4d67-b6f3-47df06eb844c","Type":"ContainerDied","Data":"1a368df4208847e057012340c138c128bc78134c8e9c801e988b863ea39e70fd"} Oct 30 16:20:39.494194 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:39.494154 2580 scope.go:122] "RemoveContainer" containerID="a422c6a0094a99fbc3b5024180e2f73c70f5438428ed7908ce5c44d62b96a90d" Oct 30 16:20:39.524001 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:39.523973 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7dg7\" (UniqueName: \"kubernetes.io/projected/b54d0e33-3943-4e1c-9889-dc65864b4efb-kube-api-access-v7dg7\") pod \"migrator-66df5d496b-vgrwl\" (UID: \"b54d0e33-3943-4e1c-9889-dc65864b4efb\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vgrwl" Oct 30 16:20:39.524135 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:39.524068 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttr7v\" (UniqueName: \"kubernetes.io/projected/341d9139-495e-4ee2-80c3-a20370039948-kube-api-access-ttr7v\") pod \"migrator-66df5d496b-c87fl\" (UID: \"341d9139-495e-4ee2-80c3-a20370039948\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-c87fl" Oct 30 16:20:39.530259 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:39.530206 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-4nn7z" podStartSLOduration=201.986199913 podStartE2EDuration="3m26.53019124s" totalImagesPullingTime="4.543991327s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:17:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:20:39.528482068 +0000 UTC m=+50.828770286" watchObservedRunningTime="2026-10-30 16:20:39.53019124 +0000 UTC m=+50.830479457" Oct 30 16:20:39.552543 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:39.552512 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttr7v\" (UniqueName: \"kubernetes.io/projected/341d9139-495e-4ee2-80c3-a20370039948-kube-api-access-ttr7v\") pod \"migrator-66df5d496b-c87fl\" (UID: \"341d9139-495e-4ee2-80c3-a20370039948\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-c87fl" Oct 30 16:20:39.593285 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:39.593252 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-cwrtz-glm8n"] Oct 30 16:20:39.596677 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:20:39.596653 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc471cf39_07e7_4f4d_87d5_0c6803fd54a0.slice/crio-146ae6117475bb6dfbc6bf6e29cc5a7c142260f8d77d5a6c93946c88f12e6888 WatchSource:0}: Error finding container 146ae6117475bb6dfbc6bf6e29cc5a7c142260f8d77d5a6c93946c88f12e6888: Status 404 returned error can't find the container with id 146ae6117475bb6dfbc6bf6e29cc5a7c142260f8d77d5a6c93946c88f12e6888 Oct 30 16:20:39.625061 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:39.621798 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-l4mhk" podStartSLOduration=1.710066064 podStartE2EDuration="6.621779559s" totalImagesPullingTime="4.911713495s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:20:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:20:39.61922759 +0000 UTC m=+50.919515809" watchObservedRunningTime="2026-10-30 16:20:39.621779559 +0000 UTC m=+50.922067782" Oct 30 16:20:39.626779 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:39.625820 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d6db39f8-bf92-44e8-9144-51eb3b9dee8d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wxvph\" (UID: \"d6db39f8-bf92-44e8-9144-51eb3b9dee8d\") " pod="openshift-insights/insights-runtime-extractor-wxvph" Oct 30 16:20:39.626779 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:39.625908 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d6db39f8-bf92-44e8-9144-51eb3b9dee8d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wxvph\" (UID: \"d6db39f8-bf92-44e8-9144-51eb3b9dee8d\") " pod="openshift-insights/insights-runtime-extractor-wxvph" Oct 30 16:20:39.626779 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:39.626132 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 16:20:39.626779 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:39.626241 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6db39f8-bf92-44e8-9144-51eb3b9dee8d-insights-runtime-extractor-tls podName:d6db39f8-bf92-44e8-9144-51eb3b9dee8d nodeName:}" failed. No retries permitted until 2026-10-30 16:20:40.62618545 +0000 UTC m=+51.926473660 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d6db39f8-bf92-44e8-9144-51eb3b9dee8d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wxvph" (UID: "d6db39f8-bf92-44e8-9144-51eb3b9dee8d") : secret "insights-runtime-extractor-tls" not found Oct 30 16:20:39.626779 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:39.626702 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 16:20:39.626779 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:39.626764 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d6db39f8-bf92-44e8-9144-51eb3b9dee8d-kube-rbac-proxy-cm podName:d6db39f8-bf92-44e8-9144-51eb3b9dee8d nodeName:}" failed. No retries permitted until 2026-10-30 16:20:40.626744554 +0000 UTC m=+51.927032767 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d6db39f8-bf92-44e8-9144-51eb3b9dee8d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wxvph" (UID: "d6db39f8-bf92-44e8-9144-51eb3b9dee8d") : configmap "kube-rbac-proxy" not found Oct 30 16:20:39.628227 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:39.627735 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7dg7\" (UniqueName: \"kubernetes.io/projected/b54d0e33-3943-4e1c-9889-dc65864b4efb-kube-api-access-v7dg7\") pod \"migrator-66df5d496b-vgrwl\" (UID: \"b54d0e33-3943-4e1c-9889-dc65864b4efb\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vgrwl" Oct 30 16:20:39.656976 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:39.656953 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7dg7\" (UniqueName: \"kubernetes.io/projected/b54d0e33-3943-4e1c-9889-dc65864b4efb-kube-api-access-v7dg7\") pod \"migrator-66df5d496b-vgrwl\" (UID: \"b54d0e33-3943-4e1c-9889-dc65864b4efb\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vgrwl" Oct 30 16:20:39.685677 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:39.685656 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-c87fl" Oct 30 16:20:39.693113 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:39.693091 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vgrwl" Oct 30 16:20:39.734101 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:39.733984 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-b4ljl"] Oct 30 16:20:39.739731 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:20:39.739695 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d1ed7d8_37cc_4652_b225_3a715b755e48.slice/crio-5374ff07dd3b95bb366a83fa0629fc3c7c14d8307de2b8d53464be4431a0d3e3 WatchSource:0}: Error finding container 5374ff07dd3b95bb366a83fa0629fc3c7c14d8307de2b8d53464be4431a0d3e3: Status 404 returned error can't find the container with id 5374ff07dd3b95bb366a83fa0629fc3c7c14d8307de2b8d53464be4431a0d3e3 Oct 30 16:20:40.020408 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:40.020316 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-c87fl"] Oct 30 16:20:40.027732 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:20:40.027693 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod341d9139_495e_4ee2_80c3_a20370039948.slice/crio-86444c15018cff90ee804c28ceafd157c4eadf26a022ae47d2a28ab74dfc4f65 WatchSource:0}: Error finding container 86444c15018cff90ee804c28ceafd157c4eadf26a022ae47d2a28ab74dfc4f65: Status 404 returned error can't find the container with id 86444c15018cff90ee804c28ceafd157c4eadf26a022ae47d2a28ab74dfc4f65 Oct 30 16:20:40.032770 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:40.032682 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-vgrwl"] Oct 30 16:20:40.035604 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:20:40.035561 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb54d0e33_3943_4e1c_9889_dc65864b4efb.slice/crio-dfb8a4fcfcd0a7cc388ac0dc4ac0179ae3d500e764cb06b5274a26d9a2c7bdbe WatchSource:0}: Error finding container dfb8a4fcfcd0a7cc388ac0dc4ac0179ae3d500e764cb06b5274a26d9a2c7bdbe: Status 404 returned error can't find the container with id dfb8a4fcfcd0a7cc388ac0dc4ac0179ae3d500e764cb06b5274a26d9a2c7bdbe Oct 30 16:20:40.322256 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:40.322178 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-8hqt5"] Oct 30 16:20:40.326096 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:40.326066 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-8hqt5" Oct 30 16:20:40.342890 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:40.342844 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Oct 30 16:20:40.343403 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:40.343385 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-wncwp\"" Oct 30 16:20:40.344292 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:40.344269 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-8hqt5"] Oct 30 16:20:40.398855 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:40.398787 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Oct 30 16:20:40.399054 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:40.398947 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Oct 30 16:20:40.399054 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:40.398950 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Oct 30 16:20:40.437661 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:40.437627 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cw6q9\" (UniqueName: \"kubernetes.io/projected/b2956847-8307-4f69-8921-bb0a0ac0aaf7-kube-api-access-cw6q9\") pod \"service-ca-5fffc97bdf-8hqt5\" (UID: \"b2956847-8307-4f69-8921-bb0a0ac0aaf7\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8hqt5" Oct 30 16:20:40.437844 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:40.437671 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/b2956847-8307-4f69-8921-bb0a0ac0aaf7-signing-cabundle\") pod \"service-ca-5fffc97bdf-8hqt5\" (UID: \"b2956847-8307-4f69-8921-bb0a0ac0aaf7\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8hqt5" Oct 30 16:20:40.437844 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:40.437807 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/b2956847-8307-4f69-8921-bb0a0ac0aaf7-signing-key\") pod \"service-ca-5fffc97bdf-8hqt5\" (UID: \"b2956847-8307-4f69-8921-bb0a0ac0aaf7\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8hqt5" Oct 30 16:20:40.498524 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:40.498487 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vgrwl" event={"ID":"b54d0e33-3943-4e1c-9889-dc65864b4efb","Type":"ContainerStarted","Data":"dfb8a4fcfcd0a7cc388ac0dc4ac0179ae3d500e764cb06b5274a26d9a2c7bdbe"} Oct 30 16:20:40.500700 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:40.500638 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-nwvwc" event={"ID":"0d304765-4b62-4cba-bad2-452771d9b201","Type":"ContainerStarted","Data":"45e7a85a14c29532a18f7a629334e2559f345a893eafb17cabce1ec736b51b30"} Oct 30 16:20:40.503152 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:40.503129 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4wrf8_4f0f46f4-5cda-4d67-b6f3-47df06eb844c/console-operator/1.log" Oct 30 16:20:40.503632 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:40.503614 2580 scope.go:122] "RemoveContainer" containerID="1a368df4208847e057012340c138c128bc78134c8e9c801e988b863ea39e70fd" Oct 30 16:20:40.503863 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:40.503836 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-4wrf8_openshift-console-operator(4f0f46f4-5cda-4d67-b6f3-47df06eb844c)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-4wrf8" podUID="4f0f46f4-5cda-4d67-b6f3-47df06eb844c" Oct 30 16:20:40.506041 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:40.506015 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-b4ljl" event={"ID":"2d1ed7d8-37cc-4652-b225-3a715b755e48","Type":"ContainerStarted","Data":"6c828a0a413a01084927b9566316b19bc888c8ed8ddbefe39ba1fb056fc93839"} Oct 30 16:20:40.506126 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:40.506053 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-b4ljl" event={"ID":"2d1ed7d8-37cc-4652-b225-3a715b755e48","Type":"ContainerStarted","Data":"5374ff07dd3b95bb366a83fa0629fc3c7c14d8307de2b8d53464be4431a0d3e3"} Oct 30 16:20:40.506508 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:40.506456 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-b4ljl" Oct 30 16:20:40.509005 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:40.508979 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-c87fl" event={"ID":"341d9139-495e-4ee2-80c3-a20370039948","Type":"ContainerStarted","Data":"86444c15018cff90ee804c28ceafd157c4eadf26a022ae47d2a28ab74dfc4f65"} Oct 30 16:20:40.519189 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:40.518768 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-cwrtz-glm8n" event={"ID":"c471cf39-07e7-4f4d-87d5-0c6803fd54a0","Type":"ContainerStarted","Data":"61bd5ae8394445d96c3c4472ab6e08a058bc26e4a5313e8391fc320fc695b1a3"} Oct 30 16:20:40.519189 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:40.518809 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-cwrtz-glm8n" event={"ID":"c471cf39-07e7-4f4d-87d5-0c6803fd54a0","Type":"ContainerStarted","Data":"146ae6117475bb6dfbc6bf6e29cc5a7c142260f8d77d5a6c93946c88f12e6888"} Oct 30 16:20:40.539097 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:40.539074 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cw6q9\" (UniqueName: \"kubernetes.io/projected/b2956847-8307-4f69-8921-bb0a0ac0aaf7-kube-api-access-cw6q9\") pod \"service-ca-5fffc97bdf-8hqt5\" (UID: \"b2956847-8307-4f69-8921-bb0a0ac0aaf7\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8hqt5" Oct 30 16:20:40.539202 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:40.539106 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/b2956847-8307-4f69-8921-bb0a0ac0aaf7-signing-cabundle\") pod \"service-ca-5fffc97bdf-8hqt5\" (UID: \"b2956847-8307-4f69-8921-bb0a0ac0aaf7\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8hqt5" Oct 30 16:20:40.539202 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:40.539149 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/b2956847-8307-4f69-8921-bb0a0ac0aaf7-signing-key\") pod \"service-ca-5fffc97bdf-8hqt5\" (UID: \"b2956847-8307-4f69-8921-bb0a0ac0aaf7\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8hqt5" Oct 30 16:20:40.540310 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:40.540267 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/b2956847-8307-4f69-8921-bb0a0ac0aaf7-signing-cabundle\") pod \"service-ca-5fffc97bdf-8hqt5\" (UID: \"b2956847-8307-4f69-8921-bb0a0ac0aaf7\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8hqt5" Oct 30 16:20:40.546491 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:40.545407 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-nwvwc" podStartSLOduration=202.255081985 podStartE2EDuration="3m27.545392162s" totalImagesPullingTime="5.290310177s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:17:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:20:40.544334837 +0000 UTC m=+51.844623054" watchObservedRunningTime="2026-10-30 16:20:40.545392162 +0000 UTC m=+51.845680382" Oct 30 16:20:40.548010 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:40.547987 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/b2956847-8307-4f69-8921-bb0a0ac0aaf7-signing-key\") pod \"service-ca-5fffc97bdf-8hqt5\" (UID: \"b2956847-8307-4f69-8921-bb0a0ac0aaf7\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8hqt5" Oct 30 16:20:40.565768 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:40.565714 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cw6q9\" (UniqueName: \"kubernetes.io/projected/b2956847-8307-4f69-8921-bb0a0ac0aaf7-kube-api-access-cw6q9\") pod \"service-ca-5fffc97bdf-8hqt5\" (UID: \"b2956847-8307-4f69-8921-bb0a0ac0aaf7\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8hqt5" Oct 30 16:20:40.590270 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:40.590201 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-b4ljl" podStartSLOduration=38.59018913 podStartE2EDuration="38.59018913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:20:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:20:40.589285084 +0000 UTC m=+51.889573304" watchObservedRunningTime="2026-10-30 16:20:40.59018913 +0000 UTC m=+51.890477349" Oct 30 16:20:40.639854 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:40.639792 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-8hqt5" Oct 30 16:20:40.640358 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:40.640335 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d6db39f8-bf92-44e8-9144-51eb3b9dee8d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wxvph\" (UID: \"d6db39f8-bf92-44e8-9144-51eb3b9dee8d\") " pod="openshift-insights/insights-runtime-extractor-wxvph" Oct 30 16:20:40.640794 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:40.640775 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d6db39f8-bf92-44e8-9144-51eb3b9dee8d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wxvph\" (UID: \"d6db39f8-bf92-44e8-9144-51eb3b9dee8d\") " pod="openshift-insights/insights-runtime-extractor-wxvph" Oct 30 16:20:40.641188 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:40.641172 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 16:20:40.641234 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:40.641217 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6db39f8-bf92-44e8-9144-51eb3b9dee8d-insights-runtime-extractor-tls podName:d6db39f8-bf92-44e8-9144-51eb3b9dee8d nodeName:}" failed. No retries permitted until 2026-10-30 16:20:42.641205124 +0000 UTC m=+53.941493333 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d6db39f8-bf92-44e8-9144-51eb3b9dee8d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wxvph" (UID: "d6db39f8-bf92-44e8-9144-51eb3b9dee8d") : secret "insights-runtime-extractor-tls" not found Oct 30 16:20:40.643439 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:40.642977 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 16:20:40.643439 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:40.643029 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d6db39f8-bf92-44e8-9144-51eb3b9dee8d-kube-rbac-proxy-cm podName:d6db39f8-bf92-44e8-9144-51eb3b9dee8d nodeName:}" failed. No retries permitted until 2026-10-30 16:20:42.643013495 +0000 UTC m=+53.943301705 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d6db39f8-bf92-44e8-9144-51eb3b9dee8d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wxvph" (UID: "d6db39f8-bf92-44e8-9144-51eb3b9dee8d") : configmap "kube-rbac-proxy" not found Oct 30 16:20:40.674595 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:40.674434 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-cwrtz-glm8n" podStartSLOduration=2.6744185639999998 podStartE2EDuration="2.674418564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:20:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:20:40.673255582 +0000 UTC m=+51.973543797" watchObservedRunningTime="2026-10-30 16:20:40.674418564 +0000 UTC m=+51.974706782" Oct 30 16:20:40.960178 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:40.960139 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-8hqt5"] Oct 30 16:20:40.977547 ip-10-0-131-136 kubenswrapper[2580]: W1030 16:20:40.977519 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2956847_8307_4f69_8921_bb0a0ac0aaf7.slice/crio-19aa67ab0cc3151a6dcd73e5612f7258d560c6582c7b753f8ae9bbce691d1f74 WatchSource:0}: Error finding container 19aa67ab0cc3151a6dcd73e5612f7258d560c6582c7b753f8ae9bbce691d1f74: Status 404 returned error can't find the container with id 19aa67ab0cc3151a6dcd73e5612f7258d560c6582c7b753f8ae9bbce691d1f74 Oct 30 16:20:41.349349 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:41.349272 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/06332900-7903-4275-8e33-de06de7f6580-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-cswcz\" (UID: \"06332900-7903-4275-8e33-de06de7f6580\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-cswcz" Oct 30 16:20:41.349488 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:41.349422 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 16:20:41.349488 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:41.349477 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06332900-7903-4275-8e33-de06de7f6580-samples-operator-tls podName:06332900-7903-4275-8e33-de06de7f6580 nodeName:}" failed. No retries permitted until 2026-10-30 16:20:49.349463429 +0000 UTC m=+60.649751630 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/06332900-7903-4275-8e33-de06de7f6580-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-cswcz" (UID: "06332900-7903-4275-8e33-de06de7f6580") : secret "samples-operator-tls" not found Oct 30 16:20:41.461698 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:41.458341 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f79e659b-79c9-43ee-8733-c685f60e87fb-registry-tls\") pod \"image-registry-65d65dd574-nbb2b\" (UID: \"f79e659b-79c9-43ee-8733-c685f60e87fb\") " pod="openshift-image-registry/image-registry-65d65dd574-nbb2b" Oct 30 16:20:41.461698 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:41.458393 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b99d8a1f-000b-48ff-b4b0-7a62bc79c032-service-ca-bundle\") pod \"router-default-df85cd6ff-6pd8x\" (UID: \"b99d8a1f-000b-48ff-b4b0-7a62bc79c032\") " pod="openshift-ingress/router-default-df85cd6ff-6pd8x" Oct 30 16:20:41.461698 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:41.458438 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1042378b-9f46-4e19-88e4-2628e532e8c3-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-992nb\" (UID: \"1042378b-9f46-4e19-88e4-2628e532e8c3\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-992nb" Oct 30 16:20:41.461698 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:41.458477 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a0cd0943-8d62-41e4-b3e7-29eaabec7277-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-sfq4m\" (UID: \"a0cd0943-8d62-41e4-b3e7-29eaabec7277\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sfq4m" Oct 30 16:20:41.461698 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:41.458508 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9fafe556-9653-49df-a851-100b51bb7f47-registry-tls\") pod \"image-registry-f8dddb4c-dcbgb\" (UID: \"9fafe556-9653-49df-a851-100b51bb7f47\") " pod="openshift-image-registry/image-registry-f8dddb4c-dcbgb" Oct 30 16:20:41.461698 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:41.458542 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b99d8a1f-000b-48ff-b4b0-7a62bc79c032-metrics-certs\") pod \"router-default-df85cd6ff-6pd8x\" (UID: \"b99d8a1f-000b-48ff-b4b0-7a62bc79c032\") " pod="openshift-ingress/router-default-df85cd6ff-6pd8x" Oct 30 16:20:41.461698 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:41.458575 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3e90fa4d-66a8-4689-8f41-ca090d6697a9-metrics-tls\") pod \"dns-default-qg6t6\" (UID: \"3e90fa4d-66a8-4689-8f41-ca090d6697a9\") " pod="openshift-dns/dns-default-qg6t6" Oct 30 16:20:41.461698 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:41.458862 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 16:20:41.461698 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:41.458895 2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 16:20:41.461698 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:41.458905 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 16:20:41.461698 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:41.458925 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-65d65dd574-nbb2b: secret "image-registry-tls" not found Oct 30 16:20:41.461698 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:41.458925 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e90fa4d-66a8-4689-8f41-ca090d6697a9-metrics-tls podName:3e90fa4d-66a8-4689-8f41-ca090d6697a9 nodeName:}" failed. No retries permitted until 2026-10-30 16:20:49.458907096 +0000 UTC m=+60.759195307 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3e90fa4d-66a8-4689-8f41-ca090d6697a9-metrics-tls") pod "dns-default-qg6t6" (UID: "3e90fa4d-66a8-4689-8f41-ca090d6697a9") : secret "dns-default-metrics-tls" not found Oct 30 16:20:41.461698 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:41.458997 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1042378b-9f46-4e19-88e4-2628e532e8c3-networking-console-plugin-cert podName:1042378b-9f46-4e19-88e4-2628e532e8c3 nodeName:}" failed. No retries permitted until 2026-10-30 16:20:49.45897892 +0000 UTC m=+60.759267122 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1042378b-9f46-4e19-88e4-2628e532e8c3-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-992nb" (UID: "1042378b-9f46-4e19-88e4-2628e532e8c3") : secret "networking-console-plugin-cert" not found Oct 30 16:20:41.461698 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:41.459002 2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 16:20:41.461698 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:41.459021 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f79e659b-79c9-43ee-8733-c685f60e87fb-registry-tls podName:f79e659b-79c9-43ee-8733-c685f60e87fb nodeName:}" failed. No retries permitted until 2026-10-30 16:20:49.459003299 +0000 UTC m=+60.759291508 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f79e659b-79c9-43ee-8733-c685f60e87fb-registry-tls") pod "image-registry-65d65dd574-nbb2b" (UID: "f79e659b-79c9-43ee-8733-c685f60e87fb") : secret "image-registry-tls" not found Oct 30 16:20:41.461698 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:41.459045 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b99d8a1f-000b-48ff-b4b0-7a62bc79c032-service-ca-bundle podName:b99d8a1f-000b-48ff-b4b0-7a62bc79c032 nodeName:}" failed. No retries permitted until 2026-10-30 16:20:49.459029522 +0000 UTC m=+60.759317733 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b99d8a1f-000b-48ff-b4b0-7a62bc79c032-service-ca-bundle") pod "router-default-df85cd6ff-6pd8x" (UID: "b99d8a1f-000b-48ff-b4b0-7a62bc79c032") : configmap references non-existent config key: service-ca.crt Oct 30 16:20:41.461698 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:41.459099 2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 16:20:41.462515 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:41.459136 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b99d8a1f-000b-48ff-b4b0-7a62bc79c032-metrics-certs podName:b99d8a1f-000b-48ff-b4b0-7a62bc79c032 nodeName:}" failed. No retries permitted until 2026-10-30 16:20:49.459125256 +0000 UTC m=+60.759413484 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b99d8a1f-000b-48ff-b4b0-7a62bc79c032-metrics-certs") pod "router-default-df85cd6ff-6pd8x" (UID: "b99d8a1f-000b-48ff-b4b0-7a62bc79c032") : secret "router-metrics-certs-default" not found Oct 30 16:20:41.462515 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:41.459157 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 16:20:41.462515 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:41.459170 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-f8dddb4c-dcbgb: secret "image-registry-tls" not found Oct 30 16:20:41.462515 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:41.459218 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9fafe556-9653-49df-a851-100b51bb7f47-registry-tls podName:9fafe556-9653-49df-a851-100b51bb7f47 nodeName:}" failed. No retries permitted until 2026-10-30 16:20:49.459205182 +0000 UTC m=+60.759493392 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9fafe556-9653-49df-a851-100b51bb7f47-registry-tls") pod "image-registry-f8dddb4c-dcbgb" (UID: "9fafe556-9653-49df-a851-100b51bb7f47") : secret "image-registry-tls" not found Oct 30 16:20:41.462515 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:41.459636 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0cd0943-8d62-41e4-b3e7-29eaabec7277-cluster-monitoring-operator-tls podName:a0cd0943-8d62-41e4-b3e7-29eaabec7277 nodeName:}" failed. No retries permitted until 2026-10-30 16:20:49.459616918 +0000 UTC m=+60.759905126 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a0cd0943-8d62-41e4-b3e7-29eaabec7277-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-sfq4m" (UID: "a0cd0943-8d62-41e4-b3e7-29eaabec7277") : secret "cluster-monitoring-operator-tls" not found Oct 30 16:20:41.524327 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:41.524291 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-c87fl" event={"ID":"341d9139-495e-4ee2-80c3-a20370039948","Type":"ContainerStarted","Data":"474505c78c636b66df6bb0b1491a72b2737c3f5fe3f4cbb8df8afcb821b0cc0f"} Oct 30 16:20:41.525842 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:41.525814 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vgrwl" event={"ID":"b54d0e33-3943-4e1c-9889-dc65864b4efb","Type":"ContainerStarted","Data":"b3742a0d7753de6698e969ce87a1ca60a7d2b0a9df2e332f236cdc18409055ca"} Oct 30 16:20:41.527555 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:41.527530 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-8hqt5" event={"ID":"b2956847-8307-4f69-8921-bb0a0ac0aaf7","Type":"ContainerStarted","Data":"4d6fa9d231c364f97522e3efed25054631b16a47f41e239ac050c0fc36c2659a"} Oct 30 16:20:41.527673 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:41.527571 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-8hqt5" event={"ID":"b2956847-8307-4f69-8921-bb0a0ac0aaf7","Type":"ContainerStarted","Data":"19aa67ab0cc3151a6dcd73e5612f7258d560c6582c7b753f8ae9bbce691d1f74"} Oct 30 16:20:41.565826 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:41.565769 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-c87fl" podStartSLOduration=1.202832625 podStartE2EDuration="2.565752195s" totalImagesPullingTime="1.36291957s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:20:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:20:41.563758748 +0000 UTC m=+52.864046966" watchObservedRunningTime="2026-10-30 16:20:41.565752195 +0000 UTC m=+52.866040414" Oct 30 16:20:41.610287 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:41.610201 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vgrwl" podStartSLOduration=1.2925013 podStartE2EDuration="2.610186536s" totalImagesPullingTime="1.317685236s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:20:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:20:41.608957989 +0000 UTC m=+52.909246208" watchObservedRunningTime="2026-10-30 16:20:41.610186536 +0000 UTC m=+52.910474753" Oct 30 16:20:41.651401 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:41.651354 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-8hqt5" podStartSLOduration=1.6513385170000001 podStartE2EDuration="1.651338517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:20:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:20:41.64985823 +0000 UTC m=+52.950146445" watchObservedRunningTime="2026-10-30 16:20:41.651338517 +0000 UTC m=+52.951626740" Oct 30 16:20:42.095397 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:42.095366 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-lghnm_9ec1217e-8623-42a6-a052-adee153936a2/dns-node-resolver/0.log" Oct 30 16:20:42.673059 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:42.673012 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d6db39f8-bf92-44e8-9144-51eb3b9dee8d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wxvph\" (UID: \"d6db39f8-bf92-44e8-9144-51eb3b9dee8d\") " pod="openshift-insights/insights-runtime-extractor-wxvph" Oct 30 16:20:42.673528 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:42.673082 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d6db39f8-bf92-44e8-9144-51eb3b9dee8d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wxvph\" (UID: \"d6db39f8-bf92-44e8-9144-51eb3b9dee8d\") " pod="openshift-insights/insights-runtime-extractor-wxvph" Oct 30 16:20:42.673528 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:42.673216 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 16:20:42.673528 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:42.673275 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6db39f8-bf92-44e8-9144-51eb3b9dee8d-insights-runtime-extractor-tls podName:d6db39f8-bf92-44e8-9144-51eb3b9dee8d nodeName:}" failed. No retries permitted until 2026-10-30 16:20:46.673257811 +0000 UTC m=+57.973546020 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d6db39f8-bf92-44e8-9144-51eb3b9dee8d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wxvph" (UID: "d6db39f8-bf92-44e8-9144-51eb3b9dee8d") : secret "insights-runtime-extractor-tls" not found Oct 30 16:20:42.673528 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:42.673335 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 16:20:42.673528 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:42.673385 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d6db39f8-bf92-44e8-9144-51eb3b9dee8d-kube-rbac-proxy-cm podName:d6db39f8-bf92-44e8-9144-51eb3b9dee8d nodeName:}" failed. No retries permitted until 2026-10-30 16:20:46.6733731 +0000 UTC m=+57.973661321 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d6db39f8-bf92-44e8-9144-51eb3b9dee8d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wxvph" (UID: "d6db39f8-bf92-44e8-9144-51eb3b9dee8d") : configmap "kube-rbac-proxy" not found Oct 30 16:20:42.895985 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:42.895956 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2l5ht_d10078a0-c21c-4f18-9c5a-4915fb7c8af3/node-ca/0.log" Oct 30 16:20:43.695411 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:43.695382 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-l4mhk_774953d5-23d3-4db1-aec9-3b78ef81017d/serve-healthcheck-canary/0.log" Oct 30 16:20:43.895846 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:43.895820 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-c87fl_341d9139-495e-4ee2-80c3-a20370039948/migrator/0.log" Oct 30 16:20:43.968892 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:43.968811 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-4wrf8" Oct 30 16:20:43.968892 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:43.968886 2580 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-4wrf8" Oct 30 16:20:43.969283 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:43.969262 2580 scope.go:122] "RemoveContainer" containerID="1a368df4208847e057012340c138c128bc78134c8e9c801e988b863ea39e70fd" Oct 30 16:20:43.969503 ip-10-0-131-136 kubenswrapper[2580]: E1030 16:20:43.969477 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-4wrf8_openshift-console-operator(4f0f46f4-5cda-4d67-b6f3-47df06eb844c)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-4wrf8" podUID="4f0f46f4-5cda-4d67-b6f3-47df06eb844c" Oct 30 16:20:44.095446 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:44.095413 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-vgrwl_b54d0e33-3943-4e1c-9889-dc65864b4efb/migrator/0.log" Oct 30 16:20:44.298059 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:44.298028 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-jrdmg_209633e5-92f7-4a7f-9d68-3d48473ed2bf/kube-storage-version-migrator-operator/0.log" Oct 30 16:20:44.412500 ip-10-0-131-136 kubenswrapper[2580]: I1030 16:20:44.412448 2580 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 16:20:44.412652 ip-10-0-131-136 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 16:20:44.428083 ip-10-0-131-136 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 16:20:44.428299 ip-10-0-131-136 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 16:20:44.428522 ip-10-0-131-136 systemd[1]: kubelet.service: Consumed 3.306s CPU time. Oct 30 16:20:44.442202 ip-10-0-131-136 systemd[1]: Starting Kubernetes Kubelet... Oct 30 16:20:44.505437 ip-10-0-131-136 kubenswrapper[5052]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 16:20:44.505437 ip-10-0-131-136 kubenswrapper[5052]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 16:20:44.505437 ip-10-0-131-136 kubenswrapper[5052]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 16:20:44.505437 ip-10-0-131-136 kubenswrapper[5052]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 16:20:44.507487 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.507410 5052 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:20:44.507487 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.507487 5052 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:20:44.507487 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.507491 5052 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:20:44.507597 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.507495 5052 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:20:44.507597 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.507497 5052 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:20:44.507597 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.507500 5052 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:20:44.507597 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.507503 5052 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:20:44.507597 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.507506 5052 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:20:44.507597 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.507508 5052 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:20:44.507597 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.507511 5052 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:20:44.507597 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.507513 5052 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:20:44.507597 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.507516 5052 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:20:44.507597 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.507519 5052 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:20:44.507597 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.507521 5052 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:20:44.507597 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.507524 5052 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:20:44.507597 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.507526 5052 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:20:44.507597 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.507529 5052 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:20:44.507597 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.507531 5052 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:20:44.507597 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.507534 5052 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:20:44.507597 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.507541 5052 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:20:44.507597 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.507543 5052 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:20:44.507597 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.507547 5052 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:20:44.508056 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.507549 5052 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:20:44.508056 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.507552 5052 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:20:44.508056 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.507555 5052 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:20:44.508056 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.507557 5052 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:20:44.508056 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.507561 5052 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:20:44.508056 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.507564 5052 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:20:44.508056 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.507567 5052 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:20:44.508056 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.507569 5052 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:20:44.508056 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.507572 5052 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:20:44.508056 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.507574 5052 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:20:44.508056 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.507601 5052 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:20:44.508056 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.507606 5052 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:20:44.508056 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.507610 5052 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:20:44.508056 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.507612 5052 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:20:44.508056 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.507615 5052 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:20:44.508056 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.507618 5052 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:20:44.508056 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.507621 5052 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:20:44.508056 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.507624 5052 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:20:44.508056 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.507627 5052 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:20:44.508501 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.507629 5052 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:20:44.508501 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.507632 5052 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:20:44.508501 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.507635 5052 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:20:44.508501 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.507637 5052 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:20:44.508501 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.507640 5052 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:20:44.508501 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.507643 5052 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:20:44.508501 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.507645 5052 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:20:44.508501 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.507648 5052 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:20:44.508501 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.507650 5052 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:20:44.508501 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.507653 5052 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:20:44.508501 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.507655 5052 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:20:44.508501 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.507658 5052 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:20:44.508501 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.507662 5052 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:20:44.508501 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.507665 5052 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:20:44.508501 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.507667 5052 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:20:44.508501 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.507670 5052 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:20:44.508501 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.507674 5052 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:20:44.508501 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.507680 5052 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:20:44.508501 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.507683 5052 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:20:44.508975 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.507687 5052 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:20:44.508975 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.507690 5052 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:20:44.508975 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.507692 5052 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:20:44.508975 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.507695 5052 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:20:44.508975 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.507697 5052 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:20:44.508975 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.507700 5052 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:20:44.508975 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.507703 5052 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:20:44.508975 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.507705 5052 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:20:44.508975 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.507708 5052 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:20:44.508975 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.507710 5052 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:20:44.508975 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.507713 5052 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:20:44.508975 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.507715 5052 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:20:44.508975 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.507717 5052 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:20:44.508975 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.507720 5052 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:20:44.508975 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.507723 5052 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:20:44.508975 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.507725 5052 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:20:44.508975 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.507728 5052 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:20:44.508975 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.507730 5052 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:20:44.508975 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.507732 5052 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:20:44.508975 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.507735 5052 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:20:44.509456 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.507737 5052 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:20:44.509456 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.507740 5052 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:20:44.509456 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.507744 5052 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:20:44.509456 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.507748 5052 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:20:44.509456 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.507751 5052 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:20:44.509456 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.507755 5052 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:20:44.509456 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.507757 5052 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:20:44.509456 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.507760 5052 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:20:44.509456 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.507762 5052 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:20:44.509456 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.507765 5052 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:20:44.509456 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.507768 5052 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:20:44.509456 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.507770 5052 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:20:44.509456 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.507773 5052 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:20:44.509456 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.507776 5052 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:20:44.509456 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.507778 5052 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:20:44.509456 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.507781 5052 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:20:44.509456 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.507784 5052 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:20:44.509456 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.507786 5052 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:20:44.509456 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.507789 5052 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:20:44.509456 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.507791 5052 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:20:44.509972 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.507794 5052 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:20:44.509972 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.507796 5052 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:20:44.509972 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.507799 5052 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:20:44.509972 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.507801 5052 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:20:44.509972 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.507805 5052 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:20:44.509972 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.507808 5052 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:20:44.509972 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.508607 5052 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:20:44.509972 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.508615 5052 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:20:44.509972 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.508619 5052 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:20:44.509972 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.508621 5052 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:20:44.509972 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.508624 5052 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:20:44.509972 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.508627 5052 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:20:44.509972 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.508630 5052 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:20:44.509972 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.508632 5052 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:20:44.509972 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.508635 5052 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:20:44.509972 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.508639 5052 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:20:44.509972 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.508644 5052 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:20:44.509972 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.508648 5052 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:20:44.510410 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.508652 5052 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:20:44.510410 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.508655 5052 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:20:44.510410 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.508657 5052 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:20:44.510410 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.508660 5052 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:20:44.510410 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.508663 5052 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:20:44.510410 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.508666 5052 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:20:44.510410 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.508668 5052 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:20:44.510410 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.508671 5052 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:20:44.510410 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.508674 5052 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:20:44.510410 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.508677 5052 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:20:44.510410 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.508679 5052 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:20:44.510410 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.508682 5052 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:20:44.510410 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.508684 5052 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:20:44.510410 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.508687 5052 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:20:44.510410 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.508689 5052 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:20:44.510410 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.508692 5052 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:20:44.510410 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.508695 5052 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:20:44.510410 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.508700 5052 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:20:44.510410 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.508704 5052 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:20:44.510410 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.508707 5052 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:20:44.510901 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.508710 5052 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:20:44.510901 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.508713 5052 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:20:44.510901 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.508715 5052 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:20:44.510901 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.508718 5052 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:20:44.510901 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.508720 5052 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:20:44.510901 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.508723 5052 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:20:44.510901 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.508725 5052 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:20:44.510901 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.508728 5052 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:20:44.510901 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.508731 5052 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:20:44.510901 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.508733 5052 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:20:44.510901 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.508736 5052 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:20:44.510901 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.508739 5052 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:20:44.510901 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.508742 5052 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:20:44.510901 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.508744 5052 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:20:44.510901 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.508747 5052 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:20:44.510901 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.508753 5052 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:20:44.510901 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.508755 5052 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:20:44.510901 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.508758 5052 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:20:44.510901 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.508761 5052 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:20:44.511364 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.508763 5052 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:20:44.511364 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.508766 5052 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:20:44.511364 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.508768 5052 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:20:44.511364 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.508771 5052 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:20:44.511364 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.508774 5052 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:20:44.511364 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.508777 5052 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:20:44.511364 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.508779 5052 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:20:44.511364 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.508782 5052 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:20:44.511364 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.508784 5052 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:20:44.511364 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.508787 5052 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:20:44.511364 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.508789 5052 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:20:44.511364 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.508792 5052 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:20:44.511364 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.508794 5052 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:20:44.511364 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.508797 5052 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:20:44.511364 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.508799 5052 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:20:44.511364 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.508802 5052 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:20:44.511364 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.508805 5052 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:20:44.511364 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.508807 5052 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:20:44.511364 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.508810 5052 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:20:44.512028 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.508812 5052 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:20:44.512028 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.508815 5052 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:20:44.512028 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.508819 5052 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:20:44.512028 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.508822 5052 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:20:44.512028 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.508826 5052 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:20:44.512028 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.508829 5052 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:20:44.512028 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.508831 5052 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:20:44.512028 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.508834 5052 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:20:44.512028 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.508842 5052 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:20:44.512028 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.508847 5052 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:20:44.512028 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.508849 5052 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:20:44.512028 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.508852 5052 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:20:44.512028 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.508855 5052 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:20:44.512028 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.508857 5052 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:20:44.512028 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.508860 5052 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:20:44.512028 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.508862 5052 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:20:44.512028 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.508864 5052 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:20:44.512028 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.508867 5052 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:20:44.512028 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.508870 5052 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:20:44.512028 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.508872 5052 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:20:44.512501 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.508875 5052 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:20:44.512501 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.508877 5052 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:20:44.512501 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.508880 5052 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:20:44.512501 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.508882 5052 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:20:44.512501 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.508885 5052 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:20:44.512501 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.508887 5052 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:20:44.512501 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.508890 5052 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:20:44.512501 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.508892 5052 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:20:44.512501 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.508895 5052 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:20:44.512501 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.508898 5052 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:20:44.512501 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.508900 5052 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:20:44.512501 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.508903 5052 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:20:44.512501 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.508905 5052 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:20:44.512501 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.508908 5052 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:20:44.512501 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.508910 5052 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:20:44.512501 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.508913 5052 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:20:44.512501 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.509393 5052 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:20:44.512501 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.509398 5052 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:20:44.512501 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.509401 5052 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:20:44.512501 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.509404 5052 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:20:44.513011 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.509407 5052 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:20:44.513011 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.509412 5052 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:20:44.513011 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.509415 5052 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:20:44.513011 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.509417 5052 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:20:44.513011 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.509420 5052 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:20:44.513011 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.509423 5052 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:20:44.513011 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.509425 5052 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:20:44.513011 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.509428 5052 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:20:44.513011 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.509430 5052 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:20:44.513011 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.509433 5052 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:20:44.513011 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.509435 5052 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:20:44.513011 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.509438 5052 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:20:44.513011 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.509441 5052 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:20:44.513011 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.509443 5052 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:20:44.513011 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.509446 5052 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:20:44.513011 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.509449 5052 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:20:44.513011 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.509451 5052 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:20:44.513011 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.509454 5052 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:20:44.513011 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.509456 5052 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:20:44.513011 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.509459 5052 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:20:44.513527 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.509462 5052 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:20:44.513527 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.509465 5052 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:20:44.513527 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.509467 5052 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:20:44.513527 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.509470 5052 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:20:44.513527 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.509472 5052 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:20:44.513527 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.509475 5052 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:20:44.513527 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.509477 5052 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:20:44.513527 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.509480 5052 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:20:44.513527 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.509483 5052 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:20:44.513527 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.509485 5052 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:20:44.513527 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.509488 5052 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:20:44.513527 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.509490 5052 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:20:44.513527 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.509493 5052 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:20:44.513527 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.509497 5052 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:20:44.513527 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.509500 5052 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:20:44.513527 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.509502 5052 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:20:44.513527 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.509505 5052 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:20:44.513527 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.509508 5052 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:20:44.513527 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.509510 5052 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:20:44.513527 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.509513 5052 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:20:44.514276 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.509515 5052 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:20:44.514276 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.509518 5052 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:20:44.514276 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.509520 5052 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:20:44.514276 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.509524 5052 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:20:44.514276 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.509528 5052 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:20:44.514276 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.509532 5052 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:20:44.514276 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.509534 5052 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:20:44.514276 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.509538 5052 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:20:44.514276 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.509540 5052 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:20:44.514276 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.509543 5052 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:20:44.514276 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.509545 5052 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:20:44.514276 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.509548 5052 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:20:44.514276 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.509550 5052 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:20:44.514276 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.509553 5052 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:20:44.514276 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.509555 5052 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:20:44.514276 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.509558 5052 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:20:44.514276 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.509561 5052 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:20:44.514276 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.509563 5052 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:20:44.514276 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.509566 5052 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:20:44.515026 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.509568 5052 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:20:44.515026 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.509571 5052 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:20:44.515026 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.509573 5052 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:20:44.515026 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.509576 5052 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:20:44.515026 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.509591 5052 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:20:44.515026 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.509594 5052 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:20:44.515026 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.509598 5052 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:20:44.515026 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.509608 5052 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:20:44.515026 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.509611 5052 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:20:44.515026 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.509613 5052 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:20:44.515026 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.509616 5052 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:20:44.515026 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.509619 5052 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:20:44.515026 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.509621 5052 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:20:44.515026 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.509624 5052 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:20:44.515026 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.509627 5052 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:20:44.515026 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.509629 5052 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:20:44.515026 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.509632 5052 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:20:44.515026 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.509636 5052 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:20:44.515026 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.509640 5052 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:20:44.515511 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.509643 5052 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:20:44.515511 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.509647 5052 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:20:44.515511 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.509649 5052 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:20:44.515511 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.509652 5052 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:20:44.515511 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.509656 5052 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:20:44.515511 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.509658 5052 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:20:44.515511 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.509661 5052 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:20:44.515511 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.509663 5052 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:20:44.515511 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.509666 5052 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:20:44.515511 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.509669 5052 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:20:44.515511 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.509671 5052 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:20:44.515511 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.509674 5052 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:20:44.515511 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.509676 5052 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:20:44.515511 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.509679 5052 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:20:44.515511 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.509681 5052 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:20:44.515511 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.509684 5052 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:20:44.515511 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.509686 5052 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:20:44.515511 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.509689 5052 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:20:44.515511 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.509691 5052 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:20:44.515511 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.509695 5052 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:20:44.516007 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.509697 5052 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:20:44.516007 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.509700 5052 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:20:44.516007 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.509703 5052 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:20:44.516007 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.509705 5052 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:20:44.516007 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.509804 5052 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 16:20:44.516007 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.509819 5052 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 16:20:44.516007 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.509827 5052 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 16:20:44.516007 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.509834 5052 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 16:20:44.516007 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.509841 5052 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 16:20:44.516007 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.509844 5052 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 16:20:44.516007 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.509848 5052 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 16:20:44.516007 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.509852 5052 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 16:20:44.516007 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.509855 5052 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 16:20:44.516007 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.509858 5052 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 16:20:44.516007 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.509862 5052 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 16:20:44.516007 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.509865 5052 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 16:20:44.516007 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.509868 5052 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 16:20:44.516007 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.509871 5052 flags.go:64] FLAG: --cgroup-root="" Oct 30 16:20:44.516007 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.509874 5052 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 16:20:44.516007 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.509877 5052 flags.go:64] FLAG: --client-ca-file="" Oct 30 16:20:44.516007 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.509880 5052 flags.go:64] FLAG: --cloud-provider="external" Oct 30 16:20:44.516007 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.509883 5052 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 16:20:44.516007 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.509888 5052 flags.go:64] FLAG: --cluster-domain="" Oct 30 16:20:44.516546 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.509891 5052 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 16:20:44.516546 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.509896 5052 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 16:20:44.516546 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.509899 5052 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 16:20:44.516546 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.509903 5052 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 16:20:44.516546 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.509907 5052 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 16:20:44.516546 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.509910 5052 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 16:20:44.516546 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.509913 5052 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 16:20:44.516546 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.509916 5052 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 16:20:44.516546 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.509921 5052 flags.go:64] FLAG: --contention-profiling="false" Oct 30 16:20:44.516546 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.509924 5052 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 16:20:44.516546 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.509927 5052 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 16:20:44.516546 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.509930 5052 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 16:20:44.516546 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.509933 5052 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 16:20:44.516546 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.509937 5052 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 16:20:44.516546 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.509941 5052 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 16:20:44.516546 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.509943 5052 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 16:20:44.516546 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.509946 5052 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 16:20:44.516546 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.509949 5052 flags.go:64] FLAG: --enable-server="true" Oct 30 16:20:44.516546 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.509952 5052 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 16:20:44.516546 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.509956 5052 flags.go:64] FLAG: --event-burst="100" Oct 30 16:20:44.516546 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.509959 5052 flags.go:64] FLAG: --event-qps="50" Oct 30 16:20:44.516546 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.509962 5052 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 16:20:44.516546 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.509965 5052 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 16:20:44.516546 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.509968 5052 flags.go:64] FLAG: --eviction-hard="" Oct 30 16:20:44.517180 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.509972 5052 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 16:20:44.517180 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.509975 5052 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 16:20:44.517180 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.509978 5052 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 16:20:44.517180 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.509982 5052 flags.go:64] FLAG: --eviction-soft="" Oct 30 16:20:44.517180 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.509985 5052 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 16:20:44.517180 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.509988 5052 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 16:20:44.517180 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.509991 5052 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 16:20:44.517180 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.509994 5052 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 16:20:44.517180 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.509997 5052 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 16:20:44.517180 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.510001 5052 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 16:20:44.517180 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.510004 5052 flags.go:64] FLAG: --feature-gates="" Oct 30 16:20:44.517180 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.510008 5052 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 16:20:44.517180 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.510011 5052 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 16:20:44.517180 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.510015 5052 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 16:20:44.517180 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.510018 5052 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 16:20:44.517180 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.510021 5052 flags.go:64] FLAG: --healthz-port="10248" Oct 30 16:20:44.517180 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.510025 5052 flags.go:64] FLAG: --help="false" Oct 30 16:20:44.517180 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.510028 5052 flags.go:64] FLAG: --hostname-override="ip-10-0-131-136.ec2.internal" Oct 30 16:20:44.517180 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.510031 5052 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 16:20:44.517180 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.510034 5052 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 16:20:44.517180 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.510037 5052 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 16:20:44.517180 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.510041 5052 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 16:20:44.517180 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.510044 5052 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 16:20:44.517753 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.510047 5052 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 16:20:44.517753 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.510049 5052 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 16:20:44.517753 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.510052 5052 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 16:20:44.517753 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.510055 5052 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 16:20:44.517753 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.510058 5052 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 16:20:44.517753 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.510061 5052 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 16:20:44.517753 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.510064 5052 flags.go:64] FLAG: --kube-reserved="" Oct 30 16:20:44.517753 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.510067 5052 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 16:20:44.517753 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.510070 5052 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 16:20:44.517753 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.510073 5052 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 16:20:44.517753 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.510076 5052 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 16:20:44.517753 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.510079 5052 flags.go:64] FLAG: --lock-file="" Oct 30 16:20:44.517753 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.510081 5052 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 16:20:44.517753 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.510084 5052 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 16:20:44.517753 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.510097 5052 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 16:20:44.517753 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.510102 5052 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 16:20:44.517753 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.510105 5052 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 16:20:44.517753 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.510109 5052 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 16:20:44.517753 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.510114 5052 flags.go:64] FLAG: --logging-format="text" Oct 30 16:20:44.517753 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.510117 5052 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 16:20:44.517753 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.510120 5052 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 16:20:44.517753 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.510123 5052 flags.go:64] FLAG: --manifest-url="" Oct 30 16:20:44.517753 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.510126 5052 flags.go:64] FLAG: --manifest-url-header="" Oct 30 16:20:44.517753 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.510130 5052 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 16:20:44.517753 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.510133 5052 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 16:20:44.518423 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.510136 5052 flags.go:64] FLAG: --max-pods="110" Oct 30 16:20:44.518423 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.510139 5052 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 16:20:44.518423 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.510142 5052 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 16:20:44.518423 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.510145 5052 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 16:20:44.518423 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.510148 5052 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 16:20:44.518423 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.510152 5052 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 16:20:44.518423 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.510155 5052 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 16:20:44.518423 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.510158 5052 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 16:20:44.518423 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.510165 5052 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 16:20:44.518423 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.510168 5052 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 16:20:44.518423 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.510171 5052 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 16:20:44.518423 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.510174 5052 flags.go:64] FLAG: --pod-cidr="" Oct 30 16:20:44.518423 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.510177 5052 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 16:20:44.518423 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.510180 5052 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 16:20:44.518423 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.510183 5052 flags.go:64] FLAG: --pods-per-core="0" Oct 30 16:20:44.518423 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.510186 5052 flags.go:64] FLAG: --port="10250" Oct 30 16:20:44.518423 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.510189 5052 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 16:20:44.518423 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.510191 5052 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0237ca767909819be" Oct 30 16:20:44.518423 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.510195 5052 flags.go:64] FLAG: --qos-reserved="" Oct 30 16:20:44.518423 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.510197 5052 flags.go:64] FLAG: --read-only-port="10255" Oct 30 16:20:44.518423 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.510201 5052 flags.go:64] FLAG: --register-node="true" Oct 30 16:20:44.518423 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.510203 5052 flags.go:64] FLAG: --register-with-taints="" Oct 30 16:20:44.518423 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.510207 5052 flags.go:64] FLAG: --registry-burst="10" Oct 30 16:20:44.518423 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.510210 5052 flags.go:64] FLAG: --registry-qps="5" Oct 30 16:20:44.518423 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.510213 5052 flags.go:64] FLAG: --reserved-cpus="" Oct 30 16:20:44.518423 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.510217 5052 flags.go:64] FLAG: --reserved-memory="" Oct 30 16:20:44.519082 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.510221 5052 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 16:20:44.519082 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.510224 5052 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 16:20:44.519082 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.510227 5052 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 16:20:44.519082 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.510230 5052 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 16:20:44.519082 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.510232 5052 flags.go:64] FLAG: --runonce="false" Oct 30 16:20:44.519082 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.510235 5052 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 16:20:44.519082 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.510238 5052 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 16:20:44.519082 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.510241 5052 flags.go:64] FLAG: --seccomp-default="false" Oct 30 16:20:44.519082 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.510244 5052 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 16:20:44.519082 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.510247 5052 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 16:20:44.519082 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.510250 5052 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 16:20:44.519082 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.510253 5052 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 16:20:44.519082 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.510256 5052 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 16:20:44.519082 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.510259 5052 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 16:20:44.519082 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.510262 5052 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 16:20:44.519082 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.510265 5052 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 16:20:44.519082 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.510268 5052 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 16:20:44.519082 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.510271 5052 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 16:20:44.519082 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.510274 5052 flags.go:64] FLAG: --system-cgroups="" Oct 30 16:20:44.519082 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.510276 5052 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 16:20:44.519082 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.510282 5052 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 16:20:44.519082 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.510285 5052 flags.go:64] FLAG: --tls-cert-file="" Oct 30 16:20:44.519082 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.510287 5052 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 16:20:44.519082 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.510292 5052 flags.go:64] FLAG: --tls-min-version="" Oct 30 16:20:44.519082 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.510294 5052 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 16:20:44.519749 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.510297 5052 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 16:20:44.519749 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.510300 5052 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 16:20:44.519749 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.510303 5052 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 16:20:44.519749 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.510305 5052 flags.go:64] FLAG: --v="2" Oct 30 16:20:44.519749 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.510315 5052 flags.go:64] FLAG: --version="false" Oct 30 16:20:44.519749 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.510319 5052 flags.go:64] FLAG: --vmodule="" Oct 30 16:20:44.519749 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.510324 5052 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 16:20:44.519749 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.510327 5052 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 16:20:44.519749 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510457 5052 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:20:44.519749 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510461 5052 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:20:44.519749 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510463 5052 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:20:44.519749 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510467 5052 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:20:44.519749 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510469 5052 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:20:44.519749 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510472 5052 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:20:44.519749 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510474 5052 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:20:44.519749 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510477 5052 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:20:44.519749 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510480 5052 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:20:44.519749 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510482 5052 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:20:44.519749 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510487 5052 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:20:44.519749 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510489 5052 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:20:44.519749 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510492 5052 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:20:44.519749 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510495 5052 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:20:44.520277 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510497 5052 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:20:44.520277 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510500 5052 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:20:44.520277 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510503 5052 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:20:44.520277 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510505 5052 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:20:44.520277 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510508 5052 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:20:44.520277 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510511 5052 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:20:44.520277 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510514 5052 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:20:44.520277 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510516 5052 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:20:44.520277 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510519 5052 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:20:44.520277 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510522 5052 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:20:44.520277 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510524 5052 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:20:44.520277 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510527 5052 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:20:44.520277 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510529 5052 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:20:44.520277 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510532 5052 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:20:44.520277 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510534 5052 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:20:44.520277 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510537 5052 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:20:44.520277 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510541 5052 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:20:44.520277 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510543 5052 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:20:44.520277 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510546 5052 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:20:44.520782 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510548 5052 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:20:44.520782 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510551 5052 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:20:44.520782 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510553 5052 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:20:44.520782 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510556 5052 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:20:44.520782 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510558 5052 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:20:44.520782 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510561 5052 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:20:44.520782 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510563 5052 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:20:44.520782 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510566 5052 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:20:44.520782 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510568 5052 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:20:44.520782 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510572 5052 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:20:44.520782 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510575 5052 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:20:44.520782 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510589 5052 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:20:44.520782 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510592 5052 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:20:44.520782 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510595 5052 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:20:44.520782 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510598 5052 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:20:44.520782 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510600 5052 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:20:44.520782 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510603 5052 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:20:44.520782 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510606 5052 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:20:44.520782 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510608 5052 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:20:44.520782 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510612 5052 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:20:44.521316 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510616 5052 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:20:44.521316 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510619 5052 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:20:44.521316 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510623 5052 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:20:44.521316 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510625 5052 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:20:44.521316 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510628 5052 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:20:44.521316 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510632 5052 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:20:44.521316 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510634 5052 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:20:44.521316 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510637 5052 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:20:44.521316 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510640 5052 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:20:44.521316 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510643 5052 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:20:44.521316 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510646 5052 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:20:44.521316 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510648 5052 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:20:44.521316 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510651 5052 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:20:44.521316 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510654 5052 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:20:44.521316 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510656 5052 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:20:44.521316 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510659 5052 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:20:44.521316 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510662 5052 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:20:44.521316 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510664 5052 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:20:44.521316 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510667 5052 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:20:44.522082 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510669 5052 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:20:44.522082 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510672 5052 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:20:44.522082 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510675 5052 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:20:44.522082 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510678 5052 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:20:44.522082 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510681 5052 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:20:44.522082 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510684 5052 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:20:44.522082 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510687 5052 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:20:44.522082 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510689 5052 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:20:44.522082 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510692 5052 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:20:44.522082 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510694 5052 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:20:44.522082 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510697 5052 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:20:44.522082 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510699 5052 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:20:44.522082 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510702 5052 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:20:44.522082 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510705 5052 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:20:44.522082 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510707 5052 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:20:44.522082 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510710 5052 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:20:44.522082 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510712 5052 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:20:44.522082 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510715 5052 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:20:44.522082 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510717 5052 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:20:44.522082 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510720 5052 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:20:44.522994 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510723 5052 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:20:44.522994 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510727 5052 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:20:44.522994 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510731 5052 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:20:44.522994 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510734 5052 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:20:44.522994 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510737 5052 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:20:44.522994 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510740 5052 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:20:44.522994 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510742 5052 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:20:44.522994 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510745 5052 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:20:44.522994 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510747 5052 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:20:44.522994 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510750 5052 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:20:44.522994 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510753 5052 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:20:44.522994 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510755 5052 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:20:44.522994 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510758 5052 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:20:44.522994 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510760 5052 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:20:44.522994 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.510787 5052 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 16:20:44.523447 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510821 5052 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:20:44.523447 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510826 5052 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:20:44.523447 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510829 5052 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:20:44.523447 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510833 5052 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:20:44.523447 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510836 5052 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:20:44.523447 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510839 5052 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:20:44.523447 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510841 5052 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:20:44.523447 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510844 5052 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:20:44.523447 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510847 5052 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:20:44.523447 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510849 5052 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:20:44.523447 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510852 5052 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:20:44.523447 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510854 5052 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:20:44.523447 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510857 5052 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:20:44.523447 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510860 5052 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:20:44.523447 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510862 5052 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:20:44.523447 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510865 5052 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:20:44.523447 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510867 5052 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:20:44.523447 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510870 5052 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:20:44.523447 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510872 5052 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:20:44.523447 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510876 5052 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:20:44.524104 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510879 5052 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:20:44.524104 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510882 5052 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:20:44.524104 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510884 5052 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:20:44.524104 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510887 5052 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:20:44.524104 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510890 5052 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:20:44.524104 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510892 5052 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:20:44.524104 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510895 5052 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:20:44.524104 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510897 5052 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:20:44.524104 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510900 5052 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:20:44.524104 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510902 5052 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:20:44.524104 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510905 5052 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:20:44.524104 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510908 5052 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:20:44.524104 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510913 5052 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:20:44.524104 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510916 5052 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:20:44.524104 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510919 5052 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:20:44.524104 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510921 5052 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:20:44.524104 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510924 5052 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:20:44.524104 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510927 5052 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:20:44.524104 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510929 5052 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:20:44.524652 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510932 5052 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:20:44.524652 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510935 5052 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:20:44.524652 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510937 5052 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:20:44.524652 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510940 5052 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:20:44.524652 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510942 5052 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:20:44.524652 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510945 5052 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:20:44.524652 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510948 5052 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:20:44.524652 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510950 5052 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:20:44.524652 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510953 5052 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:20:44.524652 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510955 5052 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:20:44.524652 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510958 5052 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:20:44.524652 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510960 5052 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:20:44.524652 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510964 5052 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:20:44.524652 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510967 5052 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:20:44.524652 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510971 5052 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:20:44.524652 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510976 5052 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:20:44.524652 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510979 5052 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:20:44.524652 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510982 5052 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:20:44.524652 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510985 5052 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:20:44.524652 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510987 5052 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:20:44.525182 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510990 5052 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:20:44.525182 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510992 5052 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:20:44.525182 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510995 5052 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:20:44.525182 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.510998 5052 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:20:44.525182 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.511000 5052 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:20:44.525182 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.511004 5052 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:20:44.525182 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.511007 5052 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:20:44.525182 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.511010 5052 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:20:44.525182 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.511012 5052 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:20:44.525182 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.511015 5052 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:20:44.525182 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.511017 5052 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:20:44.525182 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.511020 5052 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:20:44.525182 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.511023 5052 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:20:44.525182 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.511026 5052 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:20:44.525182 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.511028 5052 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:20:44.525182 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.511031 5052 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:20:44.525182 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.511033 5052 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:20:44.525182 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.511036 5052 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:20:44.525182 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.511038 5052 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:20:44.525678 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.511041 5052 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:20:44.525678 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.511044 5052 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:20:44.525678 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.511047 5052 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:20:44.525678 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.511050 5052 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:20:44.525678 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.511052 5052 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:20:44.525678 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.511056 5052 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:20:44.525678 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.511058 5052 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:20:44.525678 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.511061 5052 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:20:44.525678 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.511064 5052 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:20:44.525678 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.511066 5052 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:20:44.525678 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.511069 5052 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:20:44.525678 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.511072 5052 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:20:44.525678 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.511074 5052 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:20:44.525678 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.511077 5052 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:20:44.525678 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.511079 5052 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:20:44.525678 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.511083 5052 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:20:44.525678 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.511087 5052 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:20:44.525678 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.511090 5052 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:20:44.525678 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.511095 5052 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:20:44.526161 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.511097 5052 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:20:44.526161 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.511100 5052 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:20:44.526161 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.511102 5052 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:20:44.526161 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.511105 5052 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:20:44.526161 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.511108 5052 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:20:44.526161 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.511110 5052 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:20:44.526161 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.511113 5052 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:20:44.526161 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.511117 5052 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:20:44.526161 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.511120 5052 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:20:44.526161 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.515703 5052 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 16:20:44.526161 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.515719 5052 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 16:20:44.526161 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.515738 5052 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 16:20:44.526161 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.515743 5052 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 16:20:44.526161 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.515800 5052 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:20:44.526161 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.515806 5052 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:20:44.526161 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.515809 5052 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:20:44.526161 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.515813 5052 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:20:44.526161 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.515816 5052 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:20:44.526161 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.515819 5052 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:20:44.526634 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.515821 5052 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:20:44.526634 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.515824 5052 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:20:44.526634 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.515827 5052 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:20:44.526634 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.515830 5052 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:20:44.526634 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.515833 5052 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:20:44.526634 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.515837 5052 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:20:44.526634 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.515842 5052 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:20:44.526634 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.515845 5052 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:20:44.526634 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.515848 5052 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:20:44.526634 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.515851 5052 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:20:44.526634 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.515854 5052 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:20:44.526634 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.515857 5052 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:20:44.526634 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.515859 5052 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:20:44.526634 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.515862 5052 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:20:44.526634 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.515865 5052 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:20:44.526634 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.515867 5052 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:20:44.526634 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.515870 5052 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:20:44.526634 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.515873 5052 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:20:44.526634 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.515876 5052 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:20:44.527180 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.515878 5052 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:20:44.527180 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.515885 5052 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:20:44.527180 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.515888 5052 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:20:44.527180 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.515891 5052 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:20:44.527180 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.515893 5052 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:20:44.527180 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.515896 5052 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:20:44.527180 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.515899 5052 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:20:44.527180 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.515902 5052 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:20:44.527180 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.515904 5052 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:20:44.527180 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.515907 5052 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:20:44.527180 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.515910 5052 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:20:44.527180 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.515913 5052 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:20:44.527180 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.515915 5052 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:20:44.527180 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.515918 5052 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:20:44.527180 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.515921 5052 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:20:44.527180 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.515923 5052 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:20:44.527180 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.515926 5052 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:20:44.527180 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.515930 5052 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:20:44.527180 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.515934 5052 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:20:44.527807 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.515938 5052 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:20:44.527807 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.515940 5052 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:20:44.527807 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.515944 5052 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:20:44.527807 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.515947 5052 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:20:44.527807 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.515950 5052 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:20:44.527807 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.515952 5052 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:20:44.527807 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.515955 5052 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:20:44.527807 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.515958 5052 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:20:44.527807 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.515961 5052 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:20:44.527807 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.515963 5052 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:20:44.527807 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.515966 5052 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:20:44.527807 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.515968 5052 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:20:44.527807 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.515971 5052 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:20:44.527807 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.515974 5052 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:20:44.527807 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.515976 5052 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:20:44.527807 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.515979 5052 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:20:44.527807 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.515981 5052 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:20:44.527807 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.515984 5052 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:20:44.527807 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.515986 5052 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:20:44.527807 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.515989 5052 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:20:44.528279 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.515992 5052 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:20:44.528279 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.515994 5052 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:20:44.528279 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.515997 5052 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:20:44.528279 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516000 5052 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:20:44.528279 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516002 5052 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:20:44.528279 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516004 5052 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:20:44.528279 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516007 5052 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:20:44.528279 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516010 5052 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:20:44.528279 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516012 5052 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:20:44.528279 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516015 5052 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:20:44.528279 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516018 5052 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:20:44.528279 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516022 5052 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:20:44.528279 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516025 5052 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:20:44.528279 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516028 5052 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:20:44.528279 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516031 5052 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:20:44.528279 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516034 5052 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:20:44.528279 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516036 5052 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:20:44.528279 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516039 5052 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:20:44.528279 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516042 5052 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:20:44.528279 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516044 5052 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:20:44.528808 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516047 5052 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:20:44.528808 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516049 5052 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:20:44.528808 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516052 5052 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:20:44.528808 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516055 5052 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:20:44.528808 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516057 5052 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:20:44.528808 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516060 5052 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:20:44.528808 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516062 5052 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:20:44.528808 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516065 5052 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:20:44.528808 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516067 5052 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:20:44.528808 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516070 5052 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:20:44.528808 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516072 5052 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:20:44.528808 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516075 5052 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:20:44.528808 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516077 5052 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:20:44.528808 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516080 5052 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:20:44.528808 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516082 5052 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:20:44.528808 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516085 5052 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:20:44.528808 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516087 5052 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:20:44.528808 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516090 5052 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:20:44.528808 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516092 5052 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:20:44.529267 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516095 5052 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:20:44.529267 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516098 5052 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:20:44.529267 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516100 5052 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:20:44.529267 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.516128 5052 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 16:20:44.529267 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516262 5052 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:20:44.529267 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516268 5052 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:20:44.529267 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516272 5052 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:20:44.529267 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516275 5052 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:20:44.529267 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516277 5052 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:20:44.529267 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516280 5052 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:20:44.529267 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516285 5052 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:20:44.529267 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516289 5052 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:20:44.529267 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516293 5052 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:20:44.529267 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516296 5052 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:20:44.529267 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516298 5052 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:20:44.529267 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516301 5052 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:20:44.529681 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516304 5052 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:20:44.529681 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516307 5052 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:20:44.529681 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516309 5052 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:20:44.529681 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516312 5052 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:20:44.529681 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516315 5052 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:20:44.529681 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516318 5052 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:20:44.529681 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516320 5052 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:20:44.529681 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516323 5052 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:20:44.529681 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516325 5052 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:20:44.529681 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516328 5052 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:20:44.529681 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516330 5052 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:20:44.529681 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516333 5052 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:20:44.529681 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516335 5052 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:20:44.529681 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516338 5052 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:20:44.529681 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516341 5052 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:20:44.529681 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516343 5052 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:20:44.529681 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516346 5052 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:20:44.529681 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516348 5052 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:20:44.529681 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516351 5052 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:20:44.530229 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516353 5052 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:20:44.530229 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516356 5052 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:20:44.530229 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516359 5052 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:20:44.530229 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516361 5052 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:20:44.530229 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516364 5052 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:20:44.530229 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516366 5052 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:20:44.530229 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516369 5052 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:20:44.530229 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516372 5052 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:20:44.530229 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516375 5052 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:20:44.530229 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516377 5052 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:20:44.530229 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516380 5052 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:20:44.530229 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516383 5052 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:20:44.530229 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516386 5052 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:20:44.530229 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516389 5052 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:20:44.530229 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516391 5052 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:20:44.530229 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516394 5052 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:20:44.530229 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516396 5052 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:20:44.530229 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516399 5052 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:20:44.530229 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516401 5052 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:20:44.530834 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516404 5052 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:20:44.530834 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516407 5052 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:20:44.530834 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516409 5052 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:20:44.530834 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516412 5052 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:20:44.530834 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516414 5052 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:20:44.530834 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516417 5052 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:20:44.530834 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516419 5052 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:20:44.530834 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516422 5052 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:20:44.530834 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516425 5052 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:20:44.530834 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516427 5052 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:20:44.530834 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516430 5052 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:20:44.530834 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516432 5052 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:20:44.530834 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516435 5052 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:20:44.530834 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516437 5052 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:20:44.530834 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516440 5052 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:20:44.530834 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516442 5052 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:20:44.530834 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516445 5052 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:20:44.530834 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516448 5052 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:20:44.530834 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516450 5052 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:20:44.530834 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516453 5052 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:20:44.531429 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516455 5052 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:20:44.531429 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516458 5052 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:20:44.531429 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516461 5052 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:20:44.531429 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516464 5052 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:20:44.531429 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516466 5052 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:20:44.531429 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516469 5052 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:20:44.531429 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516471 5052 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:20:44.531429 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516474 5052 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:20:44.531429 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516476 5052 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:20:44.531429 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516479 5052 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:20:44.531429 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516481 5052 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:20:44.531429 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516487 5052 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:20:44.531429 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516489 5052 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:20:44.531429 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516492 5052 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:20:44.531429 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516495 5052 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:20:44.531429 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516498 5052 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:20:44.531429 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516500 5052 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:20:44.531429 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516502 5052 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:20:44.531429 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516505 5052 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:20:44.531429 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516508 5052 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:20:44.532220 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516511 5052 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:20:44.532220 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516513 5052 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:20:44.532220 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516516 5052 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:20:44.532220 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516518 5052 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:20:44.532220 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516521 5052 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:20:44.532220 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516525 5052 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:20:44.532220 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516528 5052 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:20:44.532220 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516531 5052 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:20:44.532220 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516534 5052 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:20:44.532220 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516537 5052 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:20:44.532220 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516539 5052 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:20:44.532220 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516542 5052 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:20:44.532220 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516545 5052 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:20:44.532220 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516548 5052 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:20:44.532220 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516551 5052 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:20:44.532220 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516554 5052 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:20:44.532829 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.516595 5052 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 16:20:44.532829 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516635 5052 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:20:44.532829 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516639 5052 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:20:44.532829 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516642 5052 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:20:44.532829 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516645 5052 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:20:44.532829 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516648 5052 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:20:44.532829 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516651 5052 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:20:44.532829 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516653 5052 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:20:44.532829 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516657 5052 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:20:44.532829 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516659 5052 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:20:44.532829 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516662 5052 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:20:44.532829 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516665 5052 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:20:44.532829 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516667 5052 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:20:44.532829 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516670 5052 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:20:44.532829 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516672 5052 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:20:44.532829 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516676 5052 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:20:44.533478 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516679 5052 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:20:44.533478 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516682 5052 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:20:44.533478 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516684 5052 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:20:44.533478 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516687 5052 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:20:44.533478 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516689 5052 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:20:44.533478 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516692 5052 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:20:44.533478 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516694 5052 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:20:44.533478 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516697 5052 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:20:44.533478 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516699 5052 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:20:44.533478 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516703 5052 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:20:44.533478 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516705 5052 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:20:44.533478 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516708 5052 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:20:44.533478 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516710 5052 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:20:44.533478 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516713 5052 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:20:44.533478 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516715 5052 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:20:44.533478 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516718 5052 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:20:44.533478 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516721 5052 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:20:44.533478 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516724 5052 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:20:44.533478 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516726 5052 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:20:44.534332 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516729 5052 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:20:44.534332 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516731 5052 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:20:44.534332 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516734 5052 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:20:44.534332 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516737 5052 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:20:44.534332 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516739 5052 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:20:44.534332 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516742 5052 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:20:44.534332 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516745 5052 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:20:44.534332 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516747 5052 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:20:44.534332 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516750 5052 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:20:44.534332 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516752 5052 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:20:44.534332 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516755 5052 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:20:44.534332 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516757 5052 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:20:44.534332 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516761 5052 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:20:44.534332 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516764 5052 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:20:44.534332 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516767 5052 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:20:44.534332 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516769 5052 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:20:44.534332 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516772 5052 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:20:44.534332 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516774 5052 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:20:44.534332 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516777 5052 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:20:44.534332 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516779 5052 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:20:44.535149 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516782 5052 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:20:44.535149 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516784 5052 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:20:44.535149 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516787 5052 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:20:44.535149 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516790 5052 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:20:44.535149 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516792 5052 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:20:44.535149 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516795 5052 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:20:44.535149 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516797 5052 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:20:44.535149 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516800 5052 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:20:44.535149 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516802 5052 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:20:44.535149 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516806 5052 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:20:44.535149 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516808 5052 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:20:44.535149 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516811 5052 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:20:44.535149 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516814 5052 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:20:44.535149 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516817 5052 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:20:44.535149 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516819 5052 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:20:44.535149 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516822 5052 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:20:44.535149 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516824 5052 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:20:44.535149 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516828 5052 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:20:44.535149 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516830 5052 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:20:44.535149 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516833 5052 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:20:44.535963 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516835 5052 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:20:44.535963 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516838 5052 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:20:44.535963 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516841 5052 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:20:44.535963 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516843 5052 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:20:44.535963 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516847 5052 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:20:44.535963 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516851 5052 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:20:44.535963 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516855 5052 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:20:44.535963 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516857 5052 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:20:44.535963 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516860 5052 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:20:44.535963 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516863 5052 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:20:44.535963 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516866 5052 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:20:44.535963 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516869 5052 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:20:44.535963 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516871 5052 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:20:44.535963 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516874 5052 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:20:44.535963 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516877 5052 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:20:44.535963 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516880 5052 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:20:44.535963 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516882 5052 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:20:44.535963 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516885 5052 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:20:44.535963 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516887 5052 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:20:44.536793 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516890 5052 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:20:44.536793 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516892 5052 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:20:44.536793 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516895 5052 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:20:44.536793 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516899 5052 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:20:44.536793 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516904 5052 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:20:44.536793 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516907 5052 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:20:44.536793 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516910 5052 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:20:44.536793 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516913 5052 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:20:44.536793 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516916 5052 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:20:44.536793 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516919 5052 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:20:44.536793 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516922 5052 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:20:44.536793 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516925 5052 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:20:44.536793 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:44.516927 5052 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:20:44.536793 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.516975 5052 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 16:20:44.536793 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.517905 5052 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 16:20:44.536793 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.517954 5052 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 16:20:44.536793 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.518145 5052 server.go:1014] "Starting client certificate rotation" Oct 30 16:20:44.536793 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.518298 5052 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 16:20:44.537567 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.518348 5052 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 16:14:59 +0000 UTC" deadline="2028-04-18 16:35:35.619884317 +0000 UTC" Oct 30 16:20:44.537567 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.518384 5052 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12864h14m51.101503154s" Oct 30 16:20:44.537567 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.518669 5052 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 16:20:44.537567 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.519619 5052 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 16:20:44.537567 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.521051 5052 log.go:25] "Validated CRI v1 runtime API" Oct 30 16:20:44.537567 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.523147 5052 log.go:25] "Validated CRI v1 image API" Oct 30 16:20:44.537567 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.523957 5052 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 16:20:44.537567 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.525927 5052 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 966e0b7a-d5cb-468d-9551-a1038e613cba:/dev/nvme0n1p4 9d3cbe91-2e74-4aa6-ba44-7a5ec42b0a03:/dev/nvme0n1p3] Oct 30 16:20:44.537948 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.525947 5052 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/02210e8bbafabc424bef434e3547303fff2acff99672b3416f13da5fa795721c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/02210e8bbafabc424bef434e3547303fff2acff99672b3416f13da5fa795721c/userdata/shm major:0 minor:230 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0fbe56808be3a5df04c4b47ef6cecad39b8b2bc2bf07dceacd9baed7d4592676/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0fbe56808be3a5df04c4b47ef6cecad39b8b2bc2bf07dceacd9baed7d4592676/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/146ae6117475bb6dfbc6bf6e29cc5a7c142260f8d77d5a6c93946c88f12e6888/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/146ae6117475bb6dfbc6bf6e29cc5a7c142260f8d77d5a6c93946c88f12e6888/userdata/shm major:0 minor:316 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/19aa67ab0cc3151a6dcd73e5612f7258d560c6582c7b753f8ae9bbce691d1f74/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/19aa67ab0cc3151a6dcd73e5612f7258d560c6582c7b753f8ae9bbce691d1f74/userdata/shm major:0 minor:362 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1de6e79aacb2bda8e4dae807b79eb5116dc5be949804852582c2f1f5105e4e70/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1de6e79aacb2bda8e4dae807b79eb5116dc5be949804852582c2f1f5105e4e70/userdata/shm major:0 minor:89 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/20e0e4f0bb761cfba7ee62e5fb09dd6dace934d281fb100229b83952b32fba9a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/20e0e4f0bb761cfba7ee62e5fb09dd6dace934d281fb100229b83952b32fba9a/userdata/shm major:0 minor:85 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2e25f0b762d57dbd49faf26aa31b5c096227a38a4195247540dada265b1c1c50/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2e25f0b762d57dbd49faf26aa31b5c096227a38a4195247540dada265b1c1c50/userdata/shm major:0 minor:73 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3a0fe4e644ac14b6dbc705911600a33fa60f54ec76fdc1c3cb9a789c664f0eca/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3a0fe4e644ac14b6dbc705911600a33fa60f54ec76fdc1c3cb9a789c664f0eca/userdata/shm major:0 minor:99 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/40e9ed71cab3cd5a3993520de66e28ba52b20f5ea5c551646136cdcf0136c32b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/40e9ed71cab3cd5a3993520de66e28ba52b20f5ea5c551646136cdcf0136c32b/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/41f89ab13b121f44360e1a4bbeef89b6d52d3402131286344cb4b94f8811e2f4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/41f89ab13b121f44360e1a4bbeef89b6d52d3402131286344cb4b94f8811e2f4/userdata/shm major:0 minor:93 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4c26f010622ac6418ada166fa96c536ff64a6c2031ad2ca56fcf154a791c277c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4c26f010622ac6418ada166fa96c536ff64a6c2031ad2ca56fcf154a791c277c/userdata/shm major:0 minor:311 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4da5620bd44861a992d02fdba52cd60b17321f2c960afc3378c3307e3e4ce96a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4da5620bd44861a992d02fdba52cd60b17321f2c960afc3378c3307e3e4ce96a/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4e2f429b17f5c625d3707f009d7076d2a08fe8c8c4bfe2c2e4b0d03d3d36c00f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4e2f429b17f5c625d3707f009d7076d2a08fe8c8c4bfe2c2e4b0d03d3d36c00f/userdata/shm major:0 minor:103 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5374ff07dd3b95bb366a83fa0629fc3c7c14d8307de2b8d53464be4431a0d3e3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5374ff07dd3b95bb366a83fa0629fc3c7c14d8307de2b8d53464be4431a0d3e3/userdata/shm major:0 minor:326 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/592f76c85aefce0da12f4d2efe9bea09041835fd390ba1b2cab47cd2e618bc66/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/592f76c85aefce0da12f4d2efe9bea09041835fd390ba1b2cab47cd2e618bc66/userdata/shm major:0 minor:239 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/622b6489c48a29a9d5eae0856d7d353098c922549fe22891407212db1d417313/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/622b6489c48a29a9d5eae0856d7d353098c922549fe22891407212db1d417313/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/67f512b5ebf9ed2c68ab41c7daf01531b3f322cd10664fddaa19faa259b5e094/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/67f512b5ebf9ed2c68ab41c7daf01531b3f322cd10664fddaa19faa259b5e094/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/86444c15018cff90ee804c28ceafd157c4eadf26a022ae47d2a28ab74dfc4f65/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/86444c15018cff90ee804c28ceafd157c4eadf26a022ae47d2a28ab74dfc4f65/userdata/shm major:0 minor:341 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8dbde36737364320b9db4f9b6b3ec9cb036413547443ff1f770d9244b3824331/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8dbde36737364320b9db4f9b6b3ec9cb036413547443ff1f770d9244b3824331/userdata/shm major:0 minor:77 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a9958733293e00825c710b46bc78d037c1b12987d541df429154b651e08f3e9e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a9958733293e00825c710b46bc78d037c1b12987d541df429154b651e08f3e9e/userdata/shm major:0 minor:228 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c5f4d46d9d932ab42101bc160c22e1b28cc23fed7c0d7fccf9a8c2bcd51e905e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c5f4d46d9d932ab42101bc160c22e1b28cc23fed7c0d7fccf9a8c2bcd51e905e/userdata/shm major:0 minor:81 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cff2925bc3a36c2f6ed0ba95b40b16b4cf5bf4b6d270f52efbc14bef48eef600/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cff2925bc3a36c2f6ed0ba95b40b16b4cf5bf4b6d270f52efbc14bef48eef600/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d66366aaa32bcced3938846aed314a23cb0bb9031dc8c5c793c42a7665851493/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d66366aaa32bcced3938846aed314a23cb0bb9031dc8c5c793c42a7665851493/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dfb8a4fcfcd0a7cc388ac0dc4ac0179ae3d500e764cb06b5274a26d9a2c7bdbe/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dfb8a4fcfcd0a7cc388ac0dc4ac0179ae3d500e764cb06b5274a26d9a2c7bdbe/userdata/shm major:0 minor:347 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f8e162f8c660d3797fd06f03e23a3233b9a1b7df45142ab247c7476073fce382/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f8e162f8c660d3797fd06f03e23a3233b9a1b7df45142ab247c7476073fce382/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fd5592d8326b2be10d1373ce1a16560e5049b9a904bf066caa5d7d6dbf8dc9a6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fd5592d8326b2be10d1373ce1a16560e5049b9a904bf066caa5d7d6dbf8dc9a6/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/029f4ecb-eb6b-4693-9e9e-ca61f554dd33/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/029f4ecb-eb6b-4693-9e9e-ca61f554dd33/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:306 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/06332900-7903-4275-8e33-de06de7f6580/volumes/kubernetes.io~projected/kube-api-access-4d6v2:{mountpoint:/var/lib/kubelet/pods/06332900-7903-4275-8e33-de06de7f6580/volumes/kubernetes.io~projected/kube-api-access-4d6v2 major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0d304765-4b62-4cba-bad2-452771d9b201/volumes/kubernetes.io~projected/kube-api-access-478bs:{mountpoint:/var/lib/kubelet/pods/0d304765-4b62-4cba-bad2-452771d9b201/volumes/kubernetes.io~projected/kube-api-access-478bs major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/154b2258-1b09-444b-ad80-2b194195e2cb/volumes/kubernetes.io~projected/kube-api-access-4kgld:{mountpoint:/var/lib/kubelet/pods/154b2258-1b09-444b-ad80-2b194195e2cb/volumes/kubernetes.io~projected/kube-api-access-4kgld major:0 minor:200 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/154b2258-1b09-444b-ad80-2b194195e2cb/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/154b2258-1b09-444b-ad80-2b194195e2cb/volumes/kubernetes.io~secret/serving-cert major:0 minor:196 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/209633e5-92f7-4a7f-9d68-3d48473ed2bf/volumes/kubernetes.io~projected/kube-api-access-z65wk:{mountpoint:/var/lib/kubelet/pods/209633e5-92f7-4a7f-9d68-3d48473ed2bf/volumes/kubernetes.io~projected/kube-api-access-z65wk major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/209633e5-92f7-4a7f-9d68-3d48473ed2bf/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/209633e5-92f7-4a7f-9d68-3d48473ed2bf/volumes/kubernetes.io~secret/serving-cert major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/25c51397-40b9-432c-949d-12ba9f36532a/volumes/kubernetes.io~projected/kube-api-access-qzp5d:{mountpoint:/var/lib/kubelet/pods/25c51397-40b9-432c-949d-12ba9f36532a/volumes/kubernetes.io~projected/kube-api-access-qzp5d major:0 minor:68 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2d1ed7d8-37cc-4652-b225-3a715b755e48/volumes/kubernetes.io~projected/kube-api-access-ljv94:{mountpoint:/var/lib/kubelet/pods/2d1ed7d8-37cc-4652-b225-3a715b755e48/volumes/kubernetes.io~projected/kube-api-access-ljv94 major:0 minor:313 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/33b888e2-d349-439b-9aa1-5b519e0e08f7/volumes/kubernetes.io~projected/kube-api-access-v5sxw:{mountpoint:/var/lib/kubelet/pods/33b888e2-d349-439b-9aa1-5b519e0e08f7/volumes/kubernetes.io~projected/kube-api-access-v5sxw major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/341d9139-495e-4ee2-80c3-a20370039948/volumes/kubernetes.io~projected/kube-api-access-ttr7v:{mountpoint:/var/lib/kubelet/pods/341d9139-495e-4ee2-80c3-a20370039948/volumes/kubernetes.io~projected/kube-api-access-ttr7v major:0 minor:328 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3e90fa4d-66a8-4689-8f41-ca090d6697a9/volumes/kubernetes.io~projected/kube-api-access-pjp8r:{mountpoint:/var/lib/kubelet/pods/3e90fa4d-66a8-4689-8f41-ca090d6697a9/volumes/kubernetes.io~projected/kube-api-access-pjp8r major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4c515c55-0717-4974-9f1a-68e0a25984cd/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/4c515c55-0717-4974-9f1a-68e0a25984cd/volumes/kubernetes.io~secret/agent-certs major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4d9a19d6-94d4-4490-8f93-89ca745b768f/volumes/kubernetes.io~projected/kube-api-access-849kr:{mountpoint:/var/lib/kubelet/pods/4d9a19d6-94d4-4490-8f93-89ca745b768f/volumes/kubernetes.io~projected/kube-api-access-849kr major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4f0f46f4-5cda-4d67-b6f3-47df06eb844c/volumes/kubernetes.io~projected/kube-api-access-bnsg7:{mountpoint:/var/lib/kubelet/pods/4f0f46f4-5cda-4d67-b6f3-47df06eb844c/volumes/kubernetes.io~projected/kube-api-access-bnsg7 major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4f0f46f4-5cda-4d67-b6f3-47df06eb844c/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/4f0f46f4-5cda-4d67-b6f3-47df06eb844c/volumes/kubernetes.io~secret/serving-cert major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6f3d36d6-5098-43cc-8f8a-bd83d5dea5fc/volumes/kubernetes.io~projected/kube-api-access-v5tsm:{mountpoint:/var/lib/kubelet/pods/6f3d36d6-5098-43cc-8f8a-bd83d5dea5fc/volumes/kubernetes.io~projected/kube-api-access-v5tsm major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/749b2b57-f20e-4cdb-8d89-d2bac352f751/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/749b2b57-f20e-4cdb-8d89-d2bac352f751/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/749b2b57-f20e-4cdb-8d89-d2bac352f751/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/749b2b57-f20e-4cdb-8d89-d2bac352f751/volumes/kubernetes.io~empty-dir/tmp major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/749b2b57-f20e-4cdb-8d89-d2bac352f751/volumes/kubernetes.io~projected/kube-api-access-dfgdd:{mountpoint:/var/lib/kubelet/pods/749b2b57-f20e-4cdb-8d89-d2bac352f751/volumes/kubernetes.io~projected/kube-api-access-dfgdd major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/774953d5-23d3-4db1-aec9-3b78ef81017d/volumes/kubernetes.io~projected/kube-api-access-qc9ck:{mountpoint:/var/lib/kubelet/pods/774953d5-23d3-4db1-aec9-3b78ef81017d/volumes/kubernetes.io~projected/kube-api-access-qc9ck major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/774953d5-23d3-4db1-aec9-3b78ef81017d/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/774953d5-23d3-4db1-aec9-3b78ef81017d/volumes/kubernetes.io~secret/cert major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/90d73ca9-5d88-4467-99fa-d4e1a67f3828/volumes/kubernetes.io~projected/kube-api-access-b4hr8:{mountpoint:/var/lib/kubelet/pods/90d73ca9-5d88-4467-99fa-d4e1a67f3828/volumes/kubernetes.io~projected/kube-api-access-b4hr8 major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9ea50669-2c08-403d-8d08-f9ffffa5b896/volumes/kubernetes.io~projected/kube-api-access-zpgwx:{mountpoint:/var/lib/kubelet/pods/9ea50669-2c08-403d-8d08-f9ffffa5b896/volumes/kubernetes.io~projected/kube-api-access-zpgwx major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9ec1217e-8623-42a6-a052-adee153936a2/volumes/kubernetes.io~projected/kube-api-access-nrppt:{mountpoint:/var/lib/kubelet/pods/9ec1217e-8623-42a6-a052-adee153936a2/volumes/kubernetes.io~projected/kube-api-access-nrppt major:0 minor:72 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9fafe556-9653-49df-a851-100b51bb7f47/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/9fafe556-9653-49df-a851-100b51bb7f47/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9fafe556-9653-49df-a851-100b51bb7f47/volumes/kubernetes.io~projected/kube-api-access-httvd:{mountpoint:/var/lib/kubelet/pods/9fafe556-9653-49df-a851-100b51bb7f47/volumes/kubernetes.io~projected/kube-api-access-httvd major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9fafe556-9653-49df-a851-100b51bb7f47/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/9fafe556-9653-49df-a851-100b51bb7f47/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9fafe556-9653-49df-a851-100b51bb7f47/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/9fafe556-9653-49df-a851-100b51bb7f47/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a0cd0943-8d62-41e4-b3e7-29eaabec7277/volumes/kubernetes.io~projected/kube-api-access-t54w7:{mountpoint:/var/lib/kubelet/pods/a0cd0943-8d62-41e4-b3e7-29eaabec7277/volumes/kubernetes.io~projected/kube-api-access-t54w7 major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a6341cee-9ba8-41ce-bb75-29c9cb4e6d29/volumes/kubernetes.io~projected/kube-api-access-gnv4j:{mountpoint:/var/lib/kubelet/pods/a6341cee-9ba8-41ce-bb75-29c9cb4e6d29/volumes/kubernetes.io~projected/kube-api-access-gnv4j major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a6b95002-74f1-475d-a64a-6495733cf142/volumes/kubernetes.io~projected/kube-api-access-8zlg7:{mountpoint:/var/lib/kubelet/pods/a6b95002-74f1-475d-a64a-6495733cf142/volumes/kubernetes.io~projected/kube-api-access-8zlg7 major:0 minor:70 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b2956847-8307-4f69-8921-bb0a0ac0aaf7/volumes/kubernetes.io~projected/kube-api-access-cw6q9:{mountpoint:/var/lib/kubelet/pods/b2956847-8307-4f69-8921-bb0a0ac0aaf7/volumes/kubernetes.io~projected/kube-api-access-cw6q9 major:0 minor:361 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b2956847-8307-4f69-8921-bb0a0ac0aaf7/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/b2956847-8307-4f69-8921-bb0a0ac0aaf7/volumes/kubernetes.io~secret/signing-key major:0 minor:360 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b54d0e33-3943-4e1c-9889-dc65864b4efb/volumes/kubernetes.io~projected/kube-api-access-v7dg7:{mountpoint:/var/lib/kubelet/pods/b54d0e33-3943-4e1c-9889-dc65864b4efb/volumes/kubernetes.io~projected/kube-api-access-v7dg7 major:0 minor:338 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b99d8a1f-000b-48ff-b4b0-7a62bc79c032/volumes/kubernetes.io~projected/kube-api-access-bhcx7:{mountpoint:/var/lib/kubelet/pods/b99d8a1f-000b-48ff-b4b0-7a62bc79c032/volumes/kubernetes.io~projected/kube-api-access-bhcx7 major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b99d8a1f-000b-48ff-b4b0-7a62bc79c032/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/b99d8a1f-000b-48ff-b4b0-7a62bc79c032/volumes/kubernetes.io~secret/default-certificate major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b99d8a1f-000b-48ff-b4b0-7a62bc79c032/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/b99d8a1f-000b-48ff-b4b0-7a62bc79c032/volumes/kubernetes.io~secret/stats-auth major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c471cf39-07e7-4f4d-87d5-0c6803fd54a0/volumes/kubernetes.io~projected/kube-api-access-mc2zh:{mountpoint:/var/lib/kubelet/pods/c471cf39-07e7-4f4d-87d5-0c6803fd54a0/volumes/kubernetes.io~projected/kube-api-access-mc2zh major:0 minor:314 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d10078a0-c21c-4f18-9c5a-4915fb7c8af3/volumes/kubernetes.io~projected/kube-api-access-25jwd:{mountpoint:/var/lib/kubelet/pods/d10078a0-c21c-4f18-9c5a-4915fb7c8af3/volumes/kubernetes.io~projected/kube-api-access-25jwd major:0 minor:71 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d6db39f8-bf92-44e8-9144-51eb3b9dee8d/volumes/kubernetes.io~projected/kube-api-access-cwfbm:{mountpoint:/var/lib/kubelet/pods/d6db39f8-bf92-44e8-9144-51eb3b9dee8d/volumes/kubernetes.io~projected/kube-api-access-cwfbm major:0 minor:315 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d719cbcb-9874-4b85-8fe7-e1bf53511235/volumes/kubernetes.io~projected/kube-api-access-2mkkh:{mountpoint:/var/lib/kubelet/pods/d719cbcb-9874-4b85-8fe7-e1bf53511235/volumes/kubernetes.io~projected/kube-api-access-2mkkh major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ef87558b-2ba2-41ce-9a0b-9e4b18663a6c/volumes/kubernetes.io~projected/kube-api-access-qlqc8:{mountpoint:/var/lib/kubelet/pods/ef87558b-2ba2-41ce-9a0b-9e4b18663a6c/volumes/kubernetes.io~projected/kube-api-access-qlqc8 major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ef87558b-2ba2-41ce-9a0b-9e4b18663a6c/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/ef87558b-2ba2-41ce-9a0b-9e4b18663a6c/volumes/kubernetes.io~secret/serving-cert major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f2de1ce0-1c1e-4276-80a9-66239e13e352/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/f2de1ce0-1c1e-4276-80a9-66239e13e352/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f2de1ce0-1c1e-4276-80a9-66239e13e352/volumes/kubernetes.io~projected/kube-api-access-7d4j4:{mountpoint:/var/lib/kubelet/pods/f2de1ce0-1c1e-4276-80a9-66239e13e352/volumes/kubernetes.io~projected/kube-api-access-7d4j4 major:0 minor:69 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f2de1ce0-1c1e-4276-80a9-66239e13e352/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/f2de1ce0-1c1e-4276-80a9-66239e13e352/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f79e659b-79c9-43ee-8733-c685f60e87fb/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/f79e659b-79c9-43ee-8733-c685f60e87fb/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f79e659b-79c9-43ee-8733-c685f60e87fb/volumes/kubernetes.io~projected/kube-api-access-5qh2h:{mountpoint:/var/lib/kubelet/pods/f79e659b-79c9-43ee-8733-c685f60e87fb/volumes/kubernetes.io~projected/kube-api-access-5qh2h major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f79e659b-79c9-43ee-8733-c685f60e87fb/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/f79e659b-79c9-43ee-8733-c685f60e87fb/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f79e659b-79c9-43ee-8733-c685f60e87fb/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/f79e659b-79c9-43ee-8733-c685f60e87fb/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:213 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-101:{mountpoint:/var/lib/containers/storage/overlay/6c9e359441ee967c1102123b450930a7f0cd9287815287ff22f1fb42e8e0d3da/merged major:0 minor:101 fsType:overlay blockSize:0} overlay_0-105:{mountpoint:/var/lib/containers/storage/overlay/eb369b266eb47e162ae4dce04ada6730a441b641fa1d1cce8e4360b15015803a/merged major:0 minor:105 fsType:overlay blockSize:0} overlay_0-107:{mountpoint:/var/lib/containers/storage/overlay/7bb85aa76b40cdef0d2d9c0531bf4e83c381b35962a8715031607072f361c4b0/merged major:0 minor:107 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/2fb675c58c12b5ee84924c452609e7d32d127bcfafd2b5e4f919e8f260e006dc/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/9379c027442e2341ac2ab3797ee25a709d2696e5c084aceab680aad728434825/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/00cfc2e69637f73ac0c37d55bc3a6167a9e8e0279db546eaa3ee3f591261370b/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/c6e5da94336e4fd2808c5a8554ade06458b8b51082e3c4a95b21b181ad7283c0/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/9414d39ce5014c0afc8868803dd864fa98a68bb53d540e467cbe3421a3544ed3/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/8e9f7450e1a7d9e3a19ef386a4e6b0c15d54566a6fbdf6720db8d2af2e6d2603/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/263ac469a28b6a86c1b405c9ed7ac61bdc1c3a55c193f75057c2efa61fb0b9e5/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/0a4d4eca12f4c9495c0b9b78852ed7e9e56c725fbcc97e4ba29627087b9c26e8/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/b5f687c301ec1cf48fb367fc0bf76e59b394fcd50e59c3fc27204011c5d4c6ca/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-156:{mountpoint:/var/lib/containers/storage/overlay/0c03fb81e1b25f7745af80e4c20d99032d2c9906130b18cc64544d4932051a1e/merged major:0 minor:156 fsType:overlay blockSize:0} overlay_0-161:{mountpoint:/var/lib/containers/storage/overlay/56e345a7d451035d615b61a18a215844db995340131849693263927f979b0d0b/merged major:0 minor:161 fsType:overlay blockSize:0} overlay_0-166:{mountpoint:/var/lib/containers/storage/overlay/7fb447509f176869718f644a7d6b20c2adfc7e947b3fdc3170905fa11abad092/merged major:0 minor:166 fsType:overlay blockSize:0} overlay_0-171:{mountpoint:/var/lib/containers/storage/overlay/0f003f5e55ae6c20cec4c8d0c0ae8ae05da12fd52d90d2b5ce76f65e1ffa9d75/merged major:0 minor:171 fsType:overlay blockSize:0} overlay_0-176:{mountpoint:/var/lib/containers/storage/overlay/a2e1716751bef7ccd6d02d8fd221fd665518338516093249ff842631b83afb1d/merged major:0 minor:176 fsType:overlay blockSize:0} overlay_0-181:{mountpoint:/var/lib/containers/storage/overlay/1ca7545f7dc81f8d29fef81bb0c87cfa13ab218b6f80af2c0b9f9a154cf25573/merged major:0 minor:181 fsType:overlay blockSize:0} overlay_0-186:{mountpoint:/var/lib/containers/storage/overlay/1dbfb872c71e0ec31a4b4f2f6521fc0f19755ea44f3d1a2537d34eb9e8cf7294/merged major:0 minor:186 fsType:overlay blockSize:0} overlay_0-191:{mountpoint:/var/lib/containers/storage/overlay/18924a8ed09e3de39dc74ec08cf7977cfa7fa685892723eef488dedbe4ffe133/merged major:0 minor:191 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/a46106d82c31133066552e052fdaba0713e73c417b5d8befa809df54f85138d6/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/5af4e4f11cc9e5ec02d4f8fe93e18ce8cd5abe73129a53093a9100253dea96e8/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/a06ff98f15166890a2c055c341b37f35f4d2dd7e6f5ddb7e0b165d3d3001982a/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/e8c97b510671b3e4ad1dcb4ee3b74c043c51cfce90b55746a0cf2e757b3997b9/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/1d94d5a44d6d2200290489b74975fb6a49e84729ce2c2b0598ab4a08c6a432c7/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/9bd0e2a0281208d3be26477eb65d2ac2fdf48c1c054de374709b85b4d55d34a5/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/13bc56a67333da59024384859ee9de4c207e68eba603842cd4f837df8a5cf269/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-267:{mountpoint:/var/lib/containers/storage/overlay/8f866522197614f70dfb7551a4ea39e810c0bf14b9c7307e977778a7f9f92d24/merged major:0 minor:267 fsType:overlay blockSize:0} overlay_0-269:{mountpoint:/var/lib/containers/storage/overlay/43a05f67a2fa709ae46b04f91395bbb0c88cea56c2883bb02141e2f6339c6a9f/merged major:0 minor:269 fsType:overlay blockSize:0} overlay_0-271:{mountpoint:/var/lib/containers/storage/overlay/6fdb4bc4b4eb20e9bd0b07b132e4d69300482bbdfe4fdfedaf21da34be2dea2a/merged major:0 minor:271 fsType:overlay blockSize:0} overlay_0-273:{mountpoint:/var/lib/containers/storage/overlay/997964ab4b9dc2cfe67a45177ffddc92b7a2bddc8ddf1ab4553f7b45439ed923/merged major:0 minor:273 fsType:overlay blockSize:0} overlay_0-281:{mountpoint:/var/lib/containers/storage/overlay/4fa5d1f3bde3fe40b4f6d177a80d4992081a2c7ff4b9e3131611ca4f577f6a9d/merged major:0 minor:281 fsType:overlay blockSize:0} overlay_0-284:{mountpoint:/var/lib/containers/storage/overlay/c02aa7c5c34b898970ac82062f6e64804fb241c0fc7589a93da97ea6ada00220/merged major:0 minor:284 fsType:overlay blockSize:0} overlay_0-286:{mountpoint:/var/lib/containers/storage/overlay/3e2d5436a55f3a6d331fd103c0a9325b91378f54c9f033606e829d8ccff2d0b3/merged major:0 minor:286 fsType:overlay blockSize:0} overlay_0-288:{mountpoint:/var/lib/containers/storage/overlay/5e1e47377098d599726f700640e8a7bafae212b2ac0a1b9478b0fb198500d48b/merged major:0 minor:288 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/03b41363fe1de3206baec7829aa4b53b2a73681a8271150798cdcb33bed6581a/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-296:{mountpoint:/var/lib/containers/storage/overlay/21a6b3959bf6738b63a810ed3d08cb144d465ecf18f0837b11a17f8f9ae21fda/merged major:0 minor:296 fsType:overlay blockSize:0} overlay_0-318:{mountpoint:/var/lib/containers/storage/overlay/dba3e2c501703c8ecc07d80009776e99c4bed77c8741789bd17482dc5b0c2587/merged major:0 minor:318 fsType:overlay blockSize:0} overlay_0-320:{mountpoint:/var/lib/containers/storage/overlay/d8b07af0a4bfafc44737f8d386aa3e180f23cd1924eb18c5399b7a9fb2c7c64e/merged major:0 minor:320 fsType:overlay blockSize:0} overlay_0-329:{mountpoint:/var/lib/containers/storage/overlay/79b494d34b8cba24f63382b8faf1217315975fd3f5374e18742aae1db964de27/merged major:0 minor:329 fsType:overlay blockSize:0} overlay_0-331:{mountpoint:/var/lib/containers/storage/overlay/6f308a09ca20510cb6e7c451b2b5924d827bccdac2361403d069e3a64fa026b6/merged major:0 minor:331 fsType:overlay blockSize:0} overlay_0-333:{mountpoint:/var/lib/containers/storage/overlay/d0320ded248c13466ab97345a043f44d7cc98111be02fe9a8cd556161c3c494a/merged major:0 minor:333 fsType:overlay blockSize:0} overlay_0-340:{mountpoint:/var/lib/containers/storage/overlay/0708d0b6ca2caca6c7fd8f40039db7069ecc05ee9ff4fedf9205310c9132dc7d/merged major:0 minor:340 fsType:overlay blockSize:0} overlay_0-350:{mountpoint:/var/lib/containers/storage/overlay/52f80ac66eb0cc3dd078ad8c8e7813791cfb322cf2d01434cbdca93dfef8dc66/merged major:0 minor:350 fsType:overlay blockSize:0} overlay_0-352:{mountpoint:/var/lib/containers/storage/overlay/b507e51fb0d346f34e9c814039f56d5a61138a26d80c6b540caf226c2daeed25/merged major:0 minor:352 fsType:overlay blockSize:0} overlay_0-358:{mountpoint:/var/lib/containers/storage/overlay/6b722e27bbc89db183bfd46e0e4b5a974c32aad109918b4ae0189b5c59006bde/merged major:0 minor:358 fsType:overlay blockSize:0} overlay_0-364:{mountpoint:/var/lib/containers/storage/overlay/e62b673459dd6ef49327bde9bc8c507dcaa047727f78c9cace333e40dd493390/merged major:0 minor:364 fsType:overlay blockSize:0} overlay_0-366:{mountpoint:/var/lib/containers/storage/overlay/a0e48de5ebb1431c53ed62cec3246986957a1f3110e7294c4bf5258f6bf6f872/merged major:0 minor:366 fsType:overlay blockSize:0} overlay_0-368:{mountpoint:/var/lib/containers/storage/overlay/2b7e059d59c8b561ba9805ed59c782794db182c98eb9320957cc85724a290b53/merged major:0 minor:368 fsType:overlay blockSize:0} overlay_0-374:{mountpoint:/var/lib/containers/storage/overlay/80a72ffbe5c1955e9abf43af20482fb33520ae9193d30a07fcbbfc39fb22276c/merged major:0 minor:374 fsType:overlay blockSize:0} overlay_0-375:{mountpoint:/var/lib/containers/storage/overlay/2d0fd61b1adda9d961e739008904db7790af060aad448ed4ac5644271cca9a18/merged major:0 minor:375 fsType:overlay blockSize:0} overlay_0-45:{mountpoint:/var/lib/containers/storage/overlay/e856e8b71b9920cbe8c19c900bb056e302badedb72f97c65b7b96f338f120fca/merged major:0 minor:45 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/701d74fd8b9fb2a86796a22dd93fa4427290b4fae37e3e5f76274921b3c5507e/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/b0dc3fbae47309ab5e1887431ba68d6f643f65ee034028cb996b567f3ba7b3f4/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-57:{mountpoint:/var/lib/containers/storage/overlay/d4226e86f8f639419a1bb0de7f1eb8efaf70ec9fa4cce0f1bcfb7843c21d75ef/merged major:0 minor:57 fsType:overlay blockSize:0} overlay_0-75:{mountpoint:/var/lib/containers/storage/overlay/9693c9d4913fa66942274ffab09d3013d6e974c0399b4a24a2f633504ff35fd6/merged major:0 minor:75 fsType:overlay blockSize:0} overlay_0-79:{mountpoint:/var/lib/containers/storage/overlay/593b809e066959d512afb0043fb3a41a24a09606a0588c95337a46a95ac92809/merged major:0 minor:79 fsType:overlay blockSize:0} overlay_0-83:{mountpoint:/var/lib/containers/storage/overlay/dfa563b85eaadf98c4023e2429496a4ec41e0615fb451d8da42cdfacf442ba5a/merged major:0 minor:83 fsType:overlay blockSize:0} overlay_0-87:{mountpoint:/var/lib/containers/storage/overlay/92798f6f4cfde42321a59e0538d9f2941eda505d65a664c292e17a638182d7b7/merged major:0 minor:87 fsType:overlay blockSize:0} overlay_0-91:{mountpoint:/var/lib/containers/storage/overlay/7cdcfa1158b8d99a3677982f9d06b0426ea4a63130d62fbd16aa9678790792f9/merged major:0 minor:91 fsType:overlay blockSize:0} overlay_0-95:{mountpoint:/var/lib/containers/storage/overlay/b54475a8f7329fc31edf88892b32d05d39f364ac22f6a30367fdedeb602dfd09/merged major:0 minor:95 fsType:overlay blockSize:0} overlay_0-97:{mountpoint:/var/lib/containers/storage/overlay/b27c9bdd102ec0b696fe191111124e8424c6847a188b9f424d12fe27e4e63333/merged major:0 minor:97 fsType:overlay blockSize:0}] Oct 30 16:20:44.538321 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.534679 5052 manager.go:217] Machine: {Timestamp:2026-10-30 16:20:44.53418774 +0000 UTC m=+0.061522970 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101372 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2cb508e032bccac7092b234caaa274 SystemUUID:ec2cb508-e032-bcca-c709-2b234caaa274 BootID:e42287c7-73b0-4523-bb5c-d0439396016a Filesystems:[{Device:/var/lib/kubelet/pods/4c515c55-0717-4974-9f1a-68e0a25984cd/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:56 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-95 DeviceMajor:0 DeviceMinor:95 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f79e659b-79c9-43ee-8733-c685f60e87fb/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:213 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/19aa67ab0cc3151a6dcd73e5612f7258d560c6582c7b753f8ae9bbce691d1f74/userdata/shm DeviceMajor:0 DeviceMinor:362 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/154b2258-1b09-444b-ad80-2b194195e2cb/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:196 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/209633e5-92f7-4a7f-9d68-3d48473ed2bf/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:210 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/4d9a19d6-94d4-4490-8f93-89ca745b768f/volumes/kubernetes.io~projected/kube-api-access-849kr DeviceMajor:0 DeviceMinor:224 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-329 DeviceMajor:0 DeviceMinor:329 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/86444c15018cff90ee804c28ceafd157c4eadf26a022ae47d2a28ab74dfc4f65/userdata/shm DeviceMajor:0 DeviceMinor:341 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-375 DeviceMajor:0 DeviceMinor:375 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-156 DeviceMajor:0 DeviceMinor:156 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8dbde36737364320b9db4f9b6b3ec9cb036413547443ff1f770d9244b3824331/userdata/shm DeviceMajor:0 DeviceMinor:77 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/41f89ab13b121f44360e1a4bbeef89b6d52d3402131286344cb4b94f8811e2f4/userdata/shm DeviceMajor:0 DeviceMinor:93 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/02210e8bbafabc424bef434e3547303fff2acff99672b3416f13da5fa795721c/userdata/shm DeviceMajor:0 DeviceMinor:230 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-296 DeviceMajor:0 DeviceMinor:296 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-366 DeviceMajor:0 DeviceMinor:366 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9ec1217e-8623-42a6-a052-adee153936a2/volumes/kubernetes.io~projected/kube-api-access-nrppt DeviceMajor:0 DeviceMinor:72 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-97 DeviceMajor:0 DeviceMinor:97 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-101 DeviceMajor:0 DeviceMinor:101 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/154b2258-1b09-444b-ad80-2b194195e2cb/volumes/kubernetes.io~projected/kube-api-access-4kgld DeviceMajor:0 DeviceMinor:200 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/f79e659b-79c9-43ee-8733-c685f60e87fb/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:211 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ef87558b-2ba2-41ce-9a0b-9e4b18663a6c/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:208 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-288 DeviceMajor:0 DeviceMinor:288 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-331 DeviceMajor:0 DeviceMinor:331 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9ea50669-2c08-403d-8d08-f9ffffa5b896/volumes/kubernetes.io~projected/kube-api-access-zpgwx DeviceMajor:0 DeviceMinor:65 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/d719cbcb-9874-4b85-8fe7-e1bf53511235/volumes/kubernetes.io~projected/kube-api-access-2mkkh DeviceMajor:0 DeviceMinor:204 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/b99d8a1f-000b-48ff-b4b0-7a62bc79c032/volumes/kubernetes.io~projected/kube-api-access-bhcx7 DeviceMajor:0 DeviceMinor:216 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/592f76c85aefce0da12f4d2efe9bea09041835fd390ba1b2cab47cd2e618bc66/userdata/shm DeviceMajor:0 DeviceMinor:239 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4c26f010622ac6418ada166fa96c536ff64a6c2031ad2ca56fcf154a791c277c/userdata/shm DeviceMajor:0 DeviceMinor:311 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/b54d0e33-3943-4e1c-9889-dc65864b4efb/volumes/kubernetes.io~projected/kube-api-access-v7dg7 DeviceMajor:0 DeviceMinor:338 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/f2de1ce0-1c1e-4276-80a9-66239e13e352/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-191 DeviceMajor:0 DeviceMinor:191 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fd5592d8326b2be10d1373ce1a16560e5049b9a904bf066caa5d7d6dbf8dc9a6/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/d6db39f8-bf92-44e8-9144-51eb3b9dee8d/volumes/kubernetes.io~projected/kube-api-access-cwfbm DeviceMajor:0 DeviceMinor:315 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-352 DeviceMajor:0 DeviceMinor:352 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-57 DeviceMajor:0 DeviceMinor:57 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6f3d36d6-5098-43cc-8f8a-bd83d5dea5fc/volumes/kubernetes.io~projected/kube-api-access-v5tsm DeviceMajor:0 DeviceMinor:64 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f79e659b-79c9-43ee-8733-c685f60e87fb/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:221 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/f2de1ce0-1c1e-4276-80a9-66239e13e352/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:55 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-83 DeviceMajor:0 DeviceMinor:83 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0d304765-4b62-4cba-bad2-452771d9b201/volumes/kubernetes.io~projected/kube-api-access-478bs DeviceMajor:0 DeviceMinor:223 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-333 DeviceMajor:0 DeviceMinor:333 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9fafe556-9653-49df-a851-100b51bb7f47/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:206 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5374ff07dd3b95bb366a83fa0629fc3c7c14d8307de2b8d53464be4431a0d3e3/userdata/shm DeviceMajor:0 DeviceMinor:326 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-79 DeviceMajor:0 DeviceMinor:79 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-171 DeviceMajor:0 DeviceMinor:171 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-181 DeviceMajor:0 DeviceMinor:181 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/749b2b57-f20e-4cdb-8d89-d2bac352f751/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:59 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-271 DeviceMajor:0 DeviceMinor:271 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dfb8a4fcfcd0a7cc388ac0dc4ac0179ae3d500e764cb06b5274a26d9a2c7bdbe/userdata/shm DeviceMajor:0 DeviceMinor:347 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/25c51397-40b9-432c-949d-12ba9f36532a/volumes/kubernetes.io~projected/kube-api-access-qzp5d DeviceMajor:0 DeviceMinor:68 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/774953d5-23d3-4db1-aec9-3b78ef81017d/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:205 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/b99d8a1f-000b-48ff-b4b0-7a62bc79c032/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:207 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4da5620bd44861a992d02fdba52cd60b17321f2c960afc3378c3307e3e4ce96a/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/33b888e2-d349-439b-9aa1-5b519e0e08f7/volumes/kubernetes.io~projected/kube-api-access-v5sxw DeviceMajor:0 DeviceMinor:67 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/a6341cee-9ba8-41ce-bb75-29c9cb4e6d29/volumes/kubernetes.io~projected/kube-api-access-gnv4j DeviceMajor:0 DeviceMinor:202 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/40e9ed71cab3cd5a3993520de66e28ba52b20f5ea5c551646136cdcf0136c32b/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-45 DeviceMajor:0 DeviceMinor:45 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/90d73ca9-5d88-4467-99fa-d4e1a67f3828/volumes/kubernetes.io~projected/kube-api-access-b4hr8 DeviceMajor:0 DeviceMinor:201 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/4f0f46f4-5cda-4d67-b6f3-47df06eb844c/volumes/kubernetes.io~projected/kube-api-access-bnsg7 DeviceMajor:0 DeviceMinor:227 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a9958733293e00825c710b46bc78d037c1b12987d541df429154b651e08f3e9e/userdata/shm DeviceMajor:0 DeviceMinor:228 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-75 DeviceMajor:0 DeviceMinor:75 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-166 DeviceMajor:0 DeviceMinor:166 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f79e659b-79c9-43ee-8733-c685f60e87fb/volumes/kubernetes.io~projected/kube-api-access-5qh2h DeviceMajor:0 DeviceMinor:226 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-91 DeviceMajor:0 DeviceMinor:91 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9fafe556-9653-49df-a851-100b51bb7f47/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:225 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-269 DeviceMajor:0 DeviceMinor:269 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-374 DeviceMajor:0 DeviceMinor:374 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f2de1ce0-1c1e-4276-80a9-66239e13e352/volumes/kubernetes.io~projected/kube-api-access-7d4j4 DeviceMajor:0 DeviceMinor:69 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/146ae6117475bb6dfbc6bf6e29cc5a7c142260f8d77d5a6c93946c88f12e6888/userdata/shm DeviceMajor:0 DeviceMinor:316 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-320 DeviceMajor:0 DeviceMinor:320 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-368 DeviceMajor:0 DeviceMinor:368 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2e25f0b762d57dbd49faf26aa31b5c096227a38a4195247540dada265b1c1c50/userdata/shm DeviceMajor:0 DeviceMinor:73 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c5f4d46d9d932ab42101bc160c22e1b28cc23fed7c0d7fccf9a8c2bcd51e905e/userdata/shm DeviceMajor:0 DeviceMinor:81 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/b99d8a1f-000b-48ff-b4b0-7a62bc79c032/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:209 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/622b6489c48a29a9d5eae0856d7d353098c922549fe22891407212db1d417313/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/749b2b57-f20e-4cdb-8d89-d2bac352f751/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:60 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/749b2b57-f20e-4cdb-8d89-d2bac352f751/volumes/kubernetes.io~projected/kube-api-access-dfgdd DeviceMajor:0 DeviceMinor:66 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-186 DeviceMajor:0 DeviceMinor:186 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9fafe556-9653-49df-a851-100b51bb7f47/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:212 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-105 DeviceMajor:0 DeviceMinor:105 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/06332900-7903-4275-8e33-de06de7f6580/volumes/kubernetes.io~projected/kube-api-access-4d6v2 DeviceMajor:0 DeviceMinor:203 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/9fafe556-9653-49df-a851-100b51bb7f47/volumes/kubernetes.io~projected/kube-api-access-httvd DeviceMajor:0 DeviceMinor:219 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/3e90fa4d-66a8-4689-8f41-ca090d6697a9/volumes/kubernetes.io~projected/kube-api-access-pjp8r DeviceMajor:0 DeviceMinor:220 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-273 DeviceMajor:0 DeviceMinor:273 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/029f4ecb-eb6b-4693-9e9e-ca61f554dd33/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:306 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-318 DeviceMajor:0 DeviceMinor:318 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2d1ed7d8-37cc-4652-b225-3a715b755e48/volumes/kubernetes.io~projected/kube-api-access-ljv94 DeviceMajor:0 DeviceMinor:313 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-107 DeviceMajor:0 DeviceMinor:107 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f8e162f8c660d3797fd06f03e23a3233b9a1b7df45142ab247c7476073fce382/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/341d9139-495e-4ee2-80c3-a20370039948/volumes/kubernetes.io~projected/kube-api-access-ttr7v DeviceMajor:0 DeviceMinor:328 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-340 DeviceMajor:0 DeviceMinor:340 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-358 DeviceMajor:0 DeviceMinor:358 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-161 DeviceMajor:0 DeviceMinor:161 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4f0f46f4-5cda-4d67-b6f3-47df06eb844c/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:214 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-284 DeviceMajor:0 DeviceMinor:284 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c471cf39-07e7-4f4d-87d5-0c6803fd54a0/volumes/kubernetes.io~projected/kube-api-access-mc2zh DeviceMajor:0 DeviceMinor:314 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-87 DeviceMajor:0 DeviceMinor:87 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4e2f429b17f5c625d3707f009d7076d2a08fe8c8c4bfe2c2e4b0d03d3d36c00f/userdata/shm DeviceMajor:0 DeviceMinor:103 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/209633e5-92f7-4a7f-9d68-3d48473ed2bf/volumes/kubernetes.io~projected/kube-api-access-z65wk DeviceMajor:0 DeviceMinor:215 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/774953d5-23d3-4db1-aec9-3b78ef81017d/volumes/kubernetes.io~projected/kube-api-access-qc9ck DeviceMajor:0 DeviceMinor:218 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/67f512b5ebf9ed2c68ab41c7daf01531b3f322cd10664fddaa19faa259b5e094/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d66366aaa32bcced3938846aed314a23cb0bb9031dc8c5c793c42a7665851493/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-350 DeviceMajor:0 DeviceMinor:350 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-364 DeviceMajor:0 DeviceMinor:364 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ef87558b-2ba2-41ce-9a0b-9e4b18663a6c/volumes/kubernetes.io~projected/kube-api-access-qlqc8 DeviceMajor:0 DeviceMinor:217 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-281 DeviceMajor:0 DeviceMinor:281 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-286 DeviceMajor:0 DeviceMinor:286 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b2956847-8307-4f69-8921-bb0a0ac0aaf7/volumes/kubernetes.io~projected/kube-api-access-cw6q9 DeviceMajor:0 DeviceMinor:361 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-176 DeviceMajor:0 DeviceMinor:176 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b2956847-8307-4f69-8921-bb0a0ac0aaf7/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:360 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cff2925bc3a36c2f6ed0ba95b40b16b4cf5bf4b6d270f52efbc14bef48eef600/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/d10078a0-c21c-4f18-9c5a-4915fb7c8af3/volumes/kubernetes.io~projected/kube-api-access-25jwd DeviceMajor:0 DeviceMinor:71 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3a0fe4e644ac14b6dbc705911600a33fa60f54ec76fdc1c3cb9a789c664f0eca/userdata/shm DeviceMajor:0 DeviceMinor:99 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-267 DeviceMajor:0 DeviceMinor:267 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a6b95002-74f1-475d-a64a-6495733cf142/volumes/kubernetes.io~projected/kube-api-access-8zlg7 DeviceMajor:0 DeviceMinor:70 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/a0cd0943-8d62-41e4-b3e7-29eaabec7277/volumes/kubernetes.io~projected/kube-api-access-t54w7 DeviceMajor:0 DeviceMinor:222 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/20e0e4f0bb761cfba7ee62e5fb09dd6dace934d281fb100229b83952b32fba9a/userdata/shm DeviceMajor:0 DeviceMinor:85 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1de6e79aacb2bda8e4dae807b79eb5116dc5be949804852582c2f1f5105e4e70/userdata/shm DeviceMajor:0 DeviceMinor:89 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0fbe56808be3a5df04c4b47ef6cecad39b8b2bc2bf07dceacd9baed7d4592676/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:02210e8bbafabc4 MacAddress:82:78:59:4e:39:82 Speed:10000 Mtu:8901} {Name:0fbe56808be3a5d MacAddress:2a:ba:5c:5b:47:f5 Speed:10000 Mtu:8901} {Name:146ae6117475bb6 MacAddress:4e:e6:56:ad:10:4e Speed:10000 Mtu:8901} {Name:19aa67ab0cc3151 MacAddress:52:34:e3:8d:7e:ba Speed:10000 Mtu:8901} {Name:40e9ed71cab3cd5 MacAddress:96:54:e6:52:1a:84 Speed:10000 Mtu:8901} {Name:4c26f010622ac64 MacAddress:4e:94:31:7f:fd:2d Speed:10000 Mtu:8901} {Name:4da5620bd44861a MacAddress:96:fb:e7:d0:55:31 Speed:10000 Mtu:8901} {Name:5374ff07dd3b95b MacAddress:d6:d6:e4:20:f7:ef Speed:10000 Mtu:8901} {Name:592f76c85aefce0 MacAddress:ce:04:9b:00:a2:d7 Speed:10000 Mtu:8901} {Name:67f512b5ebf9ed2 MacAddress:96:5d:d0:30:e7:f3 Speed:10000 Mtu:8901} {Name:86444c15018cff9 MacAddress:26:a9:d6:1c:73:0f Speed:10000 Mtu:8901} {Name:a9958733293e008 MacAddress:02:91:5e:ca:bc:38 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:c9:8e:d1:67 Speed:0 Mtu:9001} {Name:br-int MacAddress:82:bf:05:6d:ba:8d Speed:0 Mtu:8901} {Name:d66366aaa32bcce MacAddress:26:33:91:df:d7:05 Speed:10000 Mtu:8901} {Name:dfb8a4fcfcd0a7c MacAddress:da:53:fe:a8:d7:1d Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:c9:8e:d1:67 Speed:0 Mtu:9001} {Name:f8e162f8c660d37 MacAddress:4a:85:cc:d3:a0:a7 Speed:10000 Mtu:8901} {Name:fd5592d8326b2be MacAddress:ae:85:23:00:ae:e0 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:62:b0:7e:d3:42:a2 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:22:c9:b7:15:fe:4c Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 16:20:44.538321 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.535044 5052 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 16:20:44.538321 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.535086 5052 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 16:20:44.538321 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.535363 5052 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 16:20:44.538321 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.535388 5052 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-136.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 16:20:44.538321 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.535507 5052 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 16:20:44.538321 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.535513 5052 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 16:20:44.538321 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.535527 5052 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 16:20:44.538321 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.535545 5052 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 16:20:44.538321 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.535640 5052 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 16:20:44.538321 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.535788 5052 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 16:20:44.538321 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.535886 5052 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 16:20:44.538321 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.535918 5052 kubelet.go:493] "Attempting to sync node with API server" Oct 30 16:20:44.538321 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.535927 5052 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 16:20:44.538321 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.535937 5052 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 16:20:44.538321 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.535946 5052 kubelet.go:405] "Adding apiserver pod source" Oct 30 16:20:44.538321 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.535953 5052 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 16:20:44.538321 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.536784 5052 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 16:20:44.538321 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.536794 5052 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 16:20:44.538321 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.537266 5052 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 16:20:44.539409 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.539396 5052 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 16:20:44.539409 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.539406 5052 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 16:20:44.539508 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.539496 5052 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 16:20:44.539711 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.539698 5052 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 16:20:44.539770 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.539727 5052 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 16:20:44.539819 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.539807 5052 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 16:20:44.539819 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.539818 5052 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 16:20:44.539906 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.539825 5052 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 16:20:44.539906 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.539830 5052 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 16:20:44.539906 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.539836 5052 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 16:20:44.539906 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.539841 5052 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 16:20:44.539906 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.539846 5052 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 16:20:44.539906 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.539852 5052 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 16:20:44.539906 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.539859 5052 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 16:20:44.539906 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.539864 5052 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 16:20:44.539906 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.539877 5052 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 16:20:44.539906 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.539887 5052 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 16:20:44.540355 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.539913 5052 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 16:20:44.540355 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.539920 5052 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 16:20:44.543300 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.543286 5052 server.go:1263] "Started kubelet" Oct 30 16:20:44.543364 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.543342 5052 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 16:20:44.543909 ip-10-0-131-136 systemd[1]: Started Kubernetes Kubelet. Oct 30 16:20:44.544022 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.543940 5052 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 16:20:44.544022 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.544000 5052 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 16:20:44.549764 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.544274 5052 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 16:20:44.549764 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.545899 5052 server.go:317] "Adding debug handlers to kubelet server" Oct 30 16:20:44.549764 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.548145 5052 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 16:20:44.549764 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.548245 5052 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 16:20:44.549764 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.548277 5052 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 16:20:44.549764 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.548290 5052 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 16:20:44.549764 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.548310 5052 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 16:15:07 +0000 UTC" deadline="2028-08-05 18:44:31.100842414 +0000 UTC" Oct 30 16:20:44.549764 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.548349 5052 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15482h23m46.552497937s" Oct 30 16:20:44.549764 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.548391 5052 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 16:20:44.549764 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:20:44.548252 5052 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-136.ec2.internal\" not found" Oct 30 16:20:44.552431 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.550654 5052 factory.go:55] Registering systemd factory Oct 30 16:20:44.552431 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.550676 5052 factory.go:223] Registration of the systemd container factory successfully Oct 30 16:20:44.553520 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.552973 5052 factory.go:153] Registering CRI-O factory Oct 30 16:20:44.553520 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.552995 5052 factory.go:223] Registration of the crio container factory successfully Oct 30 16:20:44.553520 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.553091 5052 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 16:20:44.553520 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.553118 5052 factory.go:103] Registering Raw factory Oct 30 16:20:44.553520 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.553134 5052 manager.go:1199] Started watching for new ooms in manager Oct 30 16:20:44.553797 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.553772 5052 manager.go:319] Starting recovery of all containers Oct 30 16:20:44.561608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.553956 5052 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4f0f46f4-5cda-4d67-b6f3-47df06eb844c" volumeName="kubernetes.io/configmap/4f0f46f4-5cda-4d67-b6f3-47df06eb844c-config" seLinuxMountContext="" Oct 30 16:20:44.561608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.553995 5052 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b2956847-8307-4f69-8921-bb0a0ac0aaf7" volumeName="kubernetes.io/secret/b2956847-8307-4f69-8921-bb0a0ac0aaf7-signing-key" seLinuxMountContext="" Oct 30 16:20:44.561608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.554023 5052 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b54d0e33-3943-4e1c-9889-dc65864b4efb" volumeName="kubernetes.io/projected/b54d0e33-3943-4e1c-9889-dc65864b4efb-kube-api-access-v7dg7" seLinuxMountContext="" Oct 30 16:20:44.561608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.554038 5052 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f79e659b-79c9-43ee-8733-c685f60e87fb" volumeName="kubernetes.io/empty-dir/f79e659b-79c9-43ee-8733-c685f60e87fb-ca-trust-extracted" seLinuxMountContext="" Oct 30 16:20:44.561608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.554056 5052 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9fafe556-9653-49df-a851-100b51bb7f47" volumeName="kubernetes.io/projected/9fafe556-9653-49df-a851-100b51bb7f47-kube-api-access-httvd" seLinuxMountContext="" Oct 30 16:20:44.561608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.554069 5052 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="209633e5-92f7-4a7f-9d68-3d48473ed2bf" volumeName="kubernetes.io/projected/209633e5-92f7-4a7f-9d68-3d48473ed2bf-kube-api-access-z65wk" seLinuxMountContext="" Oct 30 16:20:44.561608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.554083 5052 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="33b888e2-d349-439b-9aa1-5b519e0e08f7" volumeName="kubernetes.io/configmap/33b888e2-d349-439b-9aa1-5b519e0e08f7-multus-daemon-config" seLinuxMountContext="" Oct 30 16:20:44.561608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.554111 5052 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d9a19d6-94d4-4490-8f93-89ca745b768f" volumeName="kubernetes.io/projected/4d9a19d6-94d4-4490-8f93-89ca745b768f-kube-api-access-849kr" seLinuxMountContext="" Oct 30 16:20:44.561608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.554218 5052 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6f3d36d6-5098-43cc-8f8a-bd83d5dea5fc" volumeName="kubernetes.io/projected/6f3d36d6-5098-43cc-8f8a-bd83d5dea5fc-kube-api-access-v5tsm" seLinuxMountContext="" Oct 30 16:20:44.561608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.554247 5052 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9fafe556-9653-49df-a851-100b51bb7f47" volumeName="kubernetes.io/configmap/9fafe556-9653-49df-a851-100b51bb7f47-registry-certificates" seLinuxMountContext="" Oct 30 16:20:44.561608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.554264 5052 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d304765-4b62-4cba-bad2-452771d9b201" volumeName="kubernetes.io/projected/0d304765-4b62-4cba-bad2-452771d9b201-kube-api-access-478bs" seLinuxMountContext="" Oct 30 16:20:44.561608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.554277 5052 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="154b2258-1b09-444b-ad80-2b194195e2cb" volumeName="kubernetes.io/configmap/154b2258-1b09-444b-ad80-2b194195e2cb-trusted-ca-bundle" seLinuxMountContext="" Oct 30 16:20:44.561608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.554295 5052 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3e90fa4d-66a8-4689-8f41-ca090d6697a9" volumeName="kubernetes.io/projected/3e90fa4d-66a8-4689-8f41-ca090d6697a9-kube-api-access-pjp8r" seLinuxMountContext="" Oct 30 16:20:44.561608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.554308 5052 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="90d73ca9-5d88-4467-99fa-d4e1a67f3828" volumeName="kubernetes.io/projected/90d73ca9-5d88-4467-99fa-d4e1a67f3828-kube-api-access-b4hr8" seLinuxMountContext="" Oct 30 16:20:44.561608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.554331 5052 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b99d8a1f-000b-48ff-b4b0-7a62bc79c032" volumeName="kubernetes.io/secret/b99d8a1f-000b-48ff-b4b0-7a62bc79c032-default-certificate" seLinuxMountContext="" Oct 30 16:20:44.561608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.554345 5052 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c471cf39-07e7-4f4d-87d5-0c6803fd54a0" volumeName="kubernetes.io/configmap/c471cf39-07e7-4f4d-87d5-0c6803fd54a0-service-ca-bundle" seLinuxMountContext="" Oct 30 16:20:44.561608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.554366 5052 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d6db39f8-bf92-44e8-9144-51eb3b9dee8d" volumeName="kubernetes.io/empty-dir/d6db39f8-bf92-44e8-9144-51eb3b9dee8d-data-volume" seLinuxMountContext="" Oct 30 16:20:44.561608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.554381 5052 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef87558b-2ba2-41ce-9a0b-9e4b18663a6c" volumeName="kubernetes.io/configmap/ef87558b-2ba2-41ce-9a0b-9e4b18663a6c-config" seLinuxMountContext="" Oct 30 16:20:44.561608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.554395 5052 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="33b888e2-d349-439b-9aa1-5b519e0e08f7" volumeName="kubernetes.io/projected/33b888e2-d349-439b-9aa1-5b519e0e08f7-kube-api-access-v5sxw" seLinuxMountContext="" Oct 30 16:20:44.561608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.554415 5052 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3e90fa4d-66a8-4689-8f41-ca090d6697a9" volumeName="kubernetes.io/configmap/3e90fa4d-66a8-4689-8f41-ca090d6697a9-config-volume" seLinuxMountContext="" Oct 30 16:20:44.561608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.554432 5052 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3e90fa4d-66a8-4689-8f41-ca090d6697a9" volumeName="kubernetes.io/empty-dir/3e90fa4d-66a8-4689-8f41-ca090d6697a9-tmp-dir" seLinuxMountContext="" Oct 30 16:20:44.561608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.554452 5052 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="749b2b57-f20e-4cdb-8d89-d2bac352f751" volumeName="kubernetes.io/projected/749b2b57-f20e-4cdb-8d89-d2bac352f751-kube-api-access-dfgdd" seLinuxMountContext="" Oct 30 16:20:44.561608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.554466 5052 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9ea50669-2c08-403d-8d08-f9ffffa5b896" volumeName="kubernetes.io/projected/9ea50669-2c08-403d-8d08-f9ffffa5b896-kube-api-access-zpgwx" seLinuxMountContext="" Oct 30 16:20:44.561608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.554526 5052 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9ec1217e-8623-42a6-a052-adee153936a2" volumeName="kubernetes.io/projected/9ec1217e-8623-42a6-a052-adee153936a2-kube-api-access-nrppt" seLinuxMountContext="" Oct 30 16:20:44.561608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.554545 5052 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b2956847-8307-4f69-8921-bb0a0ac0aaf7" volumeName="kubernetes.io/configmap/b2956847-8307-4f69-8921-bb0a0ac0aaf7-signing-cabundle" seLinuxMountContext="" Oct 30 16:20:44.561608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.554560 5052 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f79e659b-79c9-43ee-8733-c685f60e87fb" volumeName="kubernetes.io/configmap/f79e659b-79c9-43ee-8733-c685f60e87fb-trusted-ca" seLinuxMountContext="" Oct 30 16:20:44.561608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.554611 5052 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="209633e5-92f7-4a7f-9d68-3d48473ed2bf" volumeName="kubernetes.io/secret/209633e5-92f7-4a7f-9d68-3d48473ed2bf-serving-cert" seLinuxMountContext="" Oct 30 16:20:44.561608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.554629 5052 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a0cd0943-8d62-41e4-b3e7-29eaabec7277" volumeName="kubernetes.io/projected/a0cd0943-8d62-41e4-b3e7-29eaabec7277-kube-api-access-t54w7" seLinuxMountContext="" Oct 30 16:20:44.561608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.554670 5052 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef87558b-2ba2-41ce-9a0b-9e4b18663a6c" volumeName="kubernetes.io/projected/ef87558b-2ba2-41ce-9a0b-9e4b18663a6c-kube-api-access-qlqc8" seLinuxMountContext="" Oct 30 16:20:44.561608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.554683 5052 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f2de1ce0-1c1e-4276-80a9-66239e13e352" volumeName="kubernetes.io/projected/f2de1ce0-1c1e-4276-80a9-66239e13e352-kube-api-access-7d4j4" seLinuxMountContext="" Oct 30 16:20:44.561608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.554700 5052 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="341d9139-495e-4ee2-80c3-a20370039948" volumeName="kubernetes.io/projected/341d9139-495e-4ee2-80c3-a20370039948-kube-api-access-ttr7v" seLinuxMountContext="" Oct 30 16:20:44.561608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.554712 5052 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d10078a0-c21c-4f18-9c5a-4915fb7c8af3" volumeName="kubernetes.io/projected/d10078a0-c21c-4f18-9c5a-4915fb7c8af3-kube-api-access-25jwd" seLinuxMountContext="" Oct 30 16:20:44.561608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.554730 5052 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f79e659b-79c9-43ee-8733-c685f60e87fb" volumeName="kubernetes.io/configmap/f79e659b-79c9-43ee-8733-c685f60e87fb-registry-certificates" seLinuxMountContext="" Oct 30 16:20:44.561608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.554743 5052 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="33b888e2-d349-439b-9aa1-5b519e0e08f7" volumeName="kubernetes.io/configmap/33b888e2-d349-439b-9aa1-5b519e0e08f7-cni-binary-copy" seLinuxMountContext="" Oct 30 16:20:44.561608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.554755 5052 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9ea50669-2c08-403d-8d08-f9ffffa5b896" volumeName="kubernetes.io/configmap/9ea50669-2c08-403d-8d08-f9ffffa5b896-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 16:20:44.561608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.554772 5052 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9fafe556-9653-49df-a851-100b51bb7f47" volumeName="kubernetes.io/empty-dir/9fafe556-9653-49df-a851-100b51bb7f47-ca-trust-extracted" seLinuxMountContext="" Oct 30 16:20:44.561608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.554785 5052 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a6b95002-74f1-475d-a64a-6495733cf142" volumeName="kubernetes.io/projected/a6b95002-74f1-475d-a64a-6495733cf142-kube-api-access-8zlg7" seLinuxMountContext="" Oct 30 16:20:44.561608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.554802 5052 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b99d8a1f-000b-48ff-b4b0-7a62bc79c032" volumeName="kubernetes.io/projected/b99d8a1f-000b-48ff-b4b0-7a62bc79c032-kube-api-access-bhcx7" seLinuxMountContext="" Oct 30 16:20:44.561608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.554817 5052 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c471cf39-07e7-4f4d-87d5-0c6803fd54a0" volumeName="kubernetes.io/configmap/c471cf39-07e7-4f4d-87d5-0c6803fd54a0-trusted-ca-bundle" seLinuxMountContext="" Oct 30 16:20:44.561608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.554846 5052 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d6db39f8-bf92-44e8-9144-51eb3b9dee8d" volumeName="kubernetes.io/projected/d6db39f8-bf92-44e8-9144-51eb3b9dee8d-kube-api-access-cwfbm" seLinuxMountContext="" Oct 30 16:20:44.561608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.554859 5052 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f79e659b-79c9-43ee-8733-c685f60e87fb" volumeName="kubernetes.io/projected/f79e659b-79c9-43ee-8733-c685f60e87fb-bound-sa-token" seLinuxMountContext="" Oct 30 16:20:44.561608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.554872 5052 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4c515c55-0717-4974-9f1a-68e0a25984cd" volumeName="kubernetes.io/configmap/4c515c55-0717-4974-9f1a-68e0a25984cd-konnectivity-ca" seLinuxMountContext="" Oct 30 16:20:44.561608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.554889 5052 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4c515c55-0717-4974-9f1a-68e0a25984cd" volumeName="kubernetes.io/secret/4c515c55-0717-4974-9f1a-68e0a25984cd-agent-certs" seLinuxMountContext="" Oct 30 16:20:44.561608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.554901 5052 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9ec1217e-8623-42a6-a052-adee153936a2" volumeName="kubernetes.io/empty-dir/9ec1217e-8623-42a6-a052-adee153936a2-tmp-dir" seLinuxMountContext="" Oct 30 16:20:44.561608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.554920 5052 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a0cd0943-8d62-41e4-b3e7-29eaabec7277" volumeName="kubernetes.io/configmap/a0cd0943-8d62-41e4-b3e7-29eaabec7277-telemetry-config" seLinuxMountContext="" Oct 30 16:20:44.561608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.554937 5052 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a6341cee-9ba8-41ce-bb75-29c9cb4e6d29" volumeName="kubernetes.io/projected/a6341cee-9ba8-41ce-bb75-29c9cb4e6d29-kube-api-access-gnv4j" seLinuxMountContext="" Oct 30 16:20:44.561608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.554951 5052 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9fafe556-9653-49df-a851-100b51bb7f47" volumeName="kubernetes.io/secret/9fafe556-9653-49df-a851-100b51bb7f47-installation-pull-secrets" seLinuxMountContext="" Oct 30 16:20:44.561608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.554970 5052 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="209633e5-92f7-4a7f-9d68-3d48473ed2bf" volumeName="kubernetes.io/configmap/209633e5-92f7-4a7f-9d68-3d48473ed2bf-config" seLinuxMountContext="" Oct 30 16:20:44.561608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.554982 5052 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="029f4ecb-eb6b-4693-9e9e-ca61f554dd33" volumeName="kubernetes.io/secret/029f4ecb-eb6b-4693-9e9e-ca61f554dd33-original-pull-secret" seLinuxMountContext="" Oct 30 16:20:44.561608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.555000 5052 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="154b2258-1b09-444b-ad80-2b194195e2cb" volumeName="kubernetes.io/configmap/154b2258-1b09-444b-ad80-2b194195e2cb-service-ca-bundle" seLinuxMountContext="" Oct 30 16:20:44.561608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.555014 5052 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="154b2258-1b09-444b-ad80-2b194195e2cb" volumeName="kubernetes.io/secret/154b2258-1b09-444b-ad80-2b194195e2cb-serving-cert" seLinuxMountContext="" Oct 30 16:20:44.561608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.555026 5052 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2d1ed7d8-37cc-4652-b225-3a715b755e48" volumeName="kubernetes.io/projected/2d1ed7d8-37cc-4652-b225-3a715b755e48-kube-api-access-ljv94" seLinuxMountContext="" Oct 30 16:20:44.561608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.555044 5052 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4f0f46f4-5cda-4d67-b6f3-47df06eb844c" volumeName="kubernetes.io/configmap/4f0f46f4-5cda-4d67-b6f3-47df06eb844c-trusted-ca" seLinuxMountContext="" Oct 30 16:20:44.561608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.555056 5052 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b2956847-8307-4f69-8921-bb0a0ac0aaf7" volumeName="kubernetes.io/projected/b2956847-8307-4f69-8921-bb0a0ac0aaf7-kube-api-access-cw6q9" seLinuxMountContext="" Oct 30 16:20:44.561608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.555074 5052 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9fafe556-9653-49df-a851-100b51bb7f47" volumeName="kubernetes.io/configmap/9fafe556-9653-49df-a851-100b51bb7f47-trusted-ca" seLinuxMountContext="" Oct 30 16:20:44.561608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.556225 5052 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="06332900-7903-4275-8e33-de06de7f6580" volumeName="kubernetes.io/configmap/06332900-7903-4275-8e33-de06de7f6580-samples-operator-config" seLinuxMountContext="" Oct 30 16:20:44.561608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.556271 5052 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="06332900-7903-4275-8e33-de06de7f6580" volumeName="kubernetes.io/projected/06332900-7903-4275-8e33-de06de7f6580-kube-api-access-4d6v2" seLinuxMountContext="" Oct 30 16:20:44.561608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.556286 5052 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="749b2b57-f20e-4cdb-8d89-d2bac352f751" volumeName="kubernetes.io/empty-dir/749b2b57-f20e-4cdb-8d89-d2bac352f751-tmp" seLinuxMountContext="" Oct 30 16:20:44.561608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.556302 5052 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="774953d5-23d3-4db1-aec9-3b78ef81017d" volumeName="kubernetes.io/secret/774953d5-23d3-4db1-aec9-3b78ef81017d-cert" seLinuxMountContext="" Oct 30 16:20:44.561608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.556314 5052 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9ea50669-2c08-403d-8d08-f9ffffa5b896" volumeName="kubernetes.io/configmap/9ea50669-2c08-403d-8d08-f9ffffa5b896-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 16:20:44.561608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.556326 5052 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b99d8a1f-000b-48ff-b4b0-7a62bc79c032" volumeName="kubernetes.io/secret/b99d8a1f-000b-48ff-b4b0-7a62bc79c032-stats-auth" seLinuxMountContext="" Oct 30 16:20:44.561608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.556340 5052 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c471cf39-07e7-4f4d-87d5-0c6803fd54a0" volumeName="kubernetes.io/projected/c471cf39-07e7-4f4d-87d5-0c6803fd54a0-kube-api-access-mc2zh" seLinuxMountContext="" Oct 30 16:20:44.561608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.556352 5052 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25c51397-40b9-432c-949d-12ba9f36532a" volumeName="kubernetes.io/configmap/25c51397-40b9-432c-949d-12ba9f36532a-iptables-alerter-script" seLinuxMountContext="" Oct 30 16:20:44.561608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.556364 5052 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f2de1ce0-1c1e-4276-80a9-66239e13e352" volumeName="kubernetes.io/configmap/f2de1ce0-1c1e-4276-80a9-66239e13e352-env-overrides" seLinuxMountContext="" Oct 30 16:20:44.561608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.556376 5052 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f2de1ce0-1c1e-4276-80a9-66239e13e352" volumeName="kubernetes.io/configmap/f2de1ce0-1c1e-4276-80a9-66239e13e352-ovnkube-script-lib" seLinuxMountContext="" Oct 30 16:20:44.561608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.556389 5052 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f79e659b-79c9-43ee-8733-c685f60e87fb" volumeName="kubernetes.io/projected/f79e659b-79c9-43ee-8733-c685f60e87fb-kube-api-access-5qh2h" seLinuxMountContext="" Oct 30 16:20:44.561608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.556400 5052 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f79e659b-79c9-43ee-8733-c685f60e87fb" volumeName="kubernetes.io/secret/f79e659b-79c9-43ee-8733-c685f60e87fb-image-registry-private-configuration" seLinuxMountContext="" Oct 30 16:20:44.561608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.556413 5052 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9fafe556-9653-49df-a851-100b51bb7f47" volumeName="kubernetes.io/projected/9fafe556-9653-49df-a851-100b51bb7f47-bound-sa-token" seLinuxMountContext="" Oct 30 16:20:44.561608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.556426 5052 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="154b2258-1b09-444b-ad80-2b194195e2cb" volumeName="kubernetes.io/empty-dir/154b2258-1b09-444b-ad80-2b194195e2cb-snapshots" seLinuxMountContext="" Oct 30 16:20:44.561608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.556438 5052 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4f0f46f4-5cda-4d67-b6f3-47df06eb844c" volumeName="kubernetes.io/projected/4f0f46f4-5cda-4d67-b6f3-47df06eb844c-kube-api-access-bnsg7" seLinuxMountContext="" Oct 30 16:20:44.561608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.556451 5052 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="774953d5-23d3-4db1-aec9-3b78ef81017d" volumeName="kubernetes.io/projected/774953d5-23d3-4db1-aec9-3b78ef81017d-kube-api-access-qc9ck" seLinuxMountContext="" Oct 30 16:20:44.561608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.556464 5052 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef87558b-2ba2-41ce-9a0b-9e4b18663a6c" volumeName="kubernetes.io/secret/ef87558b-2ba2-41ce-9a0b-9e4b18663a6c-serving-cert" seLinuxMountContext="" Oct 30 16:20:44.561608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.556475 5052 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f2de1ce0-1c1e-4276-80a9-66239e13e352" volumeName="kubernetes.io/configmap/f2de1ce0-1c1e-4276-80a9-66239e13e352-ovnkube-config" seLinuxMountContext="" Oct 30 16:20:44.561608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.556487 5052 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c471cf39-07e7-4f4d-87d5-0c6803fd54a0" volumeName="kubernetes.io/empty-dir/c471cf39-07e7-4f4d-87d5-0c6803fd54a0-archives-path" seLinuxMountContext="" Oct 30 16:20:44.561608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.556499 5052 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="154b2258-1b09-444b-ad80-2b194195e2cb" volumeName="kubernetes.io/empty-dir/154b2258-1b09-444b-ad80-2b194195e2cb-tmp" seLinuxMountContext="" Oct 30 16:20:44.561608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.556512 5052 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4f0f46f4-5cda-4d67-b6f3-47df06eb844c" volumeName="kubernetes.io/secret/4f0f46f4-5cda-4d67-b6f3-47df06eb844c-serving-cert" seLinuxMountContext="" Oct 30 16:20:44.561608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.556527 5052 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="749b2b57-f20e-4cdb-8d89-d2bac352f751" volumeName="kubernetes.io/empty-dir/749b2b57-f20e-4cdb-8d89-d2bac352f751-etc-tuned" seLinuxMountContext="" Oct 30 16:20:44.561608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.556538 5052 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9fafe556-9653-49df-a851-100b51bb7f47" volumeName="kubernetes.io/secret/9fafe556-9653-49df-a851-100b51bb7f47-image-registry-private-configuration" seLinuxMountContext="" Oct 30 16:20:44.561608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.556550 5052 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d10078a0-c21c-4f18-9c5a-4915fb7c8af3" volumeName="kubernetes.io/configmap/d10078a0-c21c-4f18-9c5a-4915fb7c8af3-serviceca" seLinuxMountContext="" Oct 30 16:20:44.561608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.556561 5052 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f2de1ce0-1c1e-4276-80a9-66239e13e352" volumeName="kubernetes.io/secret/f2de1ce0-1c1e-4276-80a9-66239e13e352-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 16:20:44.561608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.556573 5052 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f79e659b-79c9-43ee-8733-c685f60e87fb" volumeName="kubernetes.io/secret/f79e659b-79c9-43ee-8733-c685f60e87fb-installation-pull-secrets" seLinuxMountContext="" Oct 30 16:20:44.561608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.556602 5052 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1042378b-9f46-4e19-88e4-2628e532e8c3" volumeName="kubernetes.io/configmap/1042378b-9f46-4e19-88e4-2628e532e8c3-nginx-conf" seLinuxMountContext="" Oct 30 16:20:44.561608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.556616 5052 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d719cbcb-9874-4b85-8fe7-e1bf53511235" volumeName="kubernetes.io/projected/d719cbcb-9874-4b85-8fe7-e1bf53511235-kube-api-access-2mkkh" seLinuxMountContext="" Oct 30 16:20:44.561608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.556627 5052 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9ea50669-2c08-403d-8d08-f9ffffa5b896" volumeName="kubernetes.io/configmap/9ea50669-2c08-403d-8d08-f9ffffa5b896-cni-binary-copy" seLinuxMountContext="" Oct 30 16:20:44.561608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.556640 5052 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="154b2258-1b09-444b-ad80-2b194195e2cb" volumeName="kubernetes.io/projected/154b2258-1b09-444b-ad80-2b194195e2cb-kube-api-access-4kgld" seLinuxMountContext="" Oct 30 16:20:44.561608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.556653 5052 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25c51397-40b9-432c-949d-12ba9f36532a" volumeName="kubernetes.io/projected/25c51397-40b9-432c-949d-12ba9f36532a-kube-api-access-qzp5d" seLinuxMountContext="" Oct 30 16:20:44.561608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.556665 5052 reconstruct.go:97] "Volume reconstruction finished" Oct 30 16:20:44.561608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.556673 5052 reconciler.go:29] "Reconciler: start to sync state" Oct 30 16:20:44.578151 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.578083 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4wrf8_4f0f46f4-5cda-4d67-b6f3-47df06eb844c/console-operator/1.log" Oct 30 16:20:44.582152 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.578657 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4wrf8_4f0f46f4-5cda-4d67-b6f3-47df06eb844c/console-operator/1.log" Oct 30 16:20:44.587459 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.587436 5052 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 16:20:44.589316 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.589297 5052 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 16:20:44.589316 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.589316 5052 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 16:20:44.589497 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.589336 5052 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 16:20:44.589497 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:20:44.589376 5052 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 16:20:44.591432 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.591406 5052 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 16:20:44.611008 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.610974 5052 generic.go:356] "Generic (PLEG): container finished" podID="9ea50669-2c08-403d-8d08-f9ffffa5b896" containerID="6b9a56d52e98361c9aef362ebf341a8dbf87b064a9aa3c93539874bdc1c1c792" exitCode=0 Oct 30 16:20:44.611008 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.611007 5052 generic.go:356] "Generic (PLEG): container finished" podID="9ea50669-2c08-403d-8d08-f9ffffa5b896" containerID="ea7603f6eb8322434165482e2da9cee993244aba808b6258fb700f4ecfe0cb45" exitCode=0 Oct 30 16:20:44.611008 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.611015 5052 generic.go:356] "Generic (PLEG): container finished" podID="9ea50669-2c08-403d-8d08-f9ffffa5b896" containerID="8737afe289702ed6701d13e3de20816e1549986e2c852c5f88ab20a38c49f52e" exitCode=0 Oct 30 16:20:44.611239 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.611023 5052 generic.go:356] "Generic (PLEG): container finished" podID="9ea50669-2c08-403d-8d08-f9ffffa5b896" containerID="4cc1581bac68fafdd21e5a05433eb540aadd5de0eb78eff5e9c58a771b98c2b9" exitCode=0 Oct 30 16:20:44.611239 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.611030 5052 generic.go:356] "Generic (PLEG): container finished" podID="9ea50669-2c08-403d-8d08-f9ffffa5b896" containerID="0b0a760f70d4716ef6980b4da47d24ffb414f98d4954fe814e8b4264a8ebe822" exitCode=0 Oct 30 16:20:44.611239 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.611037 5052 generic.go:356] "Generic (PLEG): container finished" podID="9ea50669-2c08-403d-8d08-f9ffffa5b896" containerID="400482828bf07e07d191072e688d0b20b11a611875a527e690e216b28be6b46f" exitCode=0 Oct 30 16:20:44.618474 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.618445 5052 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 16:20:44.618633 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.618614 5052 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 16:20:44.619353 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.619283 5052 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 16:20:44.620016 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.619998 5052 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 16:20:44.656048 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.656022 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4wrf8_4f0f46f4-5cda-4d67-b6f3-47df06eb844c/console-operator/1.log" Oct 30 16:20:44.656192 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.656051 5052 generic.go:356] "Generic (PLEG): container finished" podID="4f0f46f4-5cda-4d67-b6f3-47df06eb844c" containerID="1a368df4208847e057012340c138c128bc78134c8e9c801e988b863ea39e70fd" exitCode=255 Oct 30 16:20:44.657957 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.657932 5052 generic.go:356] "Generic (PLEG): container finished" podID="9ce644f9016009147990b020805ff06d" containerID="e62b2d0f673552954712774014cc22bed99f8d3f3e61d4488511b83f0aeffd54" exitCode=0 Oct 30 16:20:44.659073 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.659057 5052 manager.go:324] Recovery completed Oct 30 16:20:44.678619 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.678598 5052 cpu_manager.go:226] "Starting" policy="none" Oct 30 16:20:44.678619 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.678614 5052 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 16:20:44.678752 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.678634 5052 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 16:20:44.678796 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.678774 5052 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 16:20:44.678826 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.678798 5052 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 16:20:44.678826 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.678819 5052 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 16:20:44.678881 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.678826 5052 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 16:20:44.678881 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.678835 5052 policy_none.go:50] "Start" Oct 30 16:20:44.678881 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.678846 5052 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 16:20:44.678881 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.678860 5052 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 16:20:44.680051 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.680039 5052 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 16:20:44.680085 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.680053 5052 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 16:20:44.680085 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.680061 5052 policy_none.go:44] "Start" Oct 30 16:20:44.685452 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.685424 5052 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 16:20:44.689428 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:20:44.689411 5052 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Oct 30 16:20:44.691074 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.691061 5052 manager.go:341] "Starting Device Plugin manager" Oct 30 16:20:44.691524 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:20:44.691090 5052 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 16:20:44.691617 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.691528 5052 server.go:87] "Starting device plugin registration server" Oct 30 16:20:44.691832 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.691819 5052 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 16:20:44.691893 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.691834 5052 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 16:20:44.691940 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.691902 5052 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 16:20:44.692026 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.691992 5052 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 16:20:44.692080 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.692035 5052 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 16:20:44.692080 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.692042 5052 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 16:20:44.692186 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.692108 5052 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T16:20:44.692024866Z","UUID":"b496f21f-f457-4b7a-b1f0-b6a976565b2c","Handler":null,"Name":"","Endpoint":""} Oct 30 16:20:44.693312 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:20:44.693291 5052 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 16:20:44.693384 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.693335 5052 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 16:20:44.693384 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.693355 5052 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 16:20:44.792476 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.792413 5052 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 16:20:44.793456 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.793438 5052 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-136.ec2.internal" event="NodeHasSufficientMemory" Oct 30 16:20:44.793533 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.793469 5052 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-136.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 16:20:44.793533 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.793483 5052 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-136.ec2.internal" event="NodeHasSufficientPID" Oct 30 16:20:44.793628 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.793535 5052 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-131-136.ec2.internal" Oct 30 16:20:44.834994 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.834971 5052 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-131-136.ec2.internal" Oct 30 16:20:44.835089 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.835036 5052 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-131-136.ec2.internal" Oct 30 16:20:44.890550 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.890496 5052 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-136.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-136.ec2.internal"] Oct 30 16:20:44.891150 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.891135 5052 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-136.ec2.internal" Oct 30 16:20:44.891198 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.891151 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-136.ec2.internal" event={"ID":"3de5cf7923bfc1c5b7b74a0700d7b3fa","Type":"ContainerStarted","Data":"853edd892ee7f9eea56e6e455c769971da6f0c48189481b115f4b81e636a22da"} Oct 30 16:20:44.891198 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.891192 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-136.ec2.internal" event={"ID":"3de5cf7923bfc1c5b7b74a0700d7b3fa","Type":"ContainerStarted","Data":"cff2925bc3a36c2f6ed0ba95b40b16b4cf5bf4b6d270f52efbc14bef48eef600"} Oct 30 16:20:44.891255 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.891214 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-136.ec2.internal" event={"ID":"9ce644f9016009147990b020805ff06d","Type":"ContainerStarted","Data":"b0f31a88ac74561bf5fb58c82d0cbdad3966b5c838f996c996dde6b60acabcc5"} Oct 30 16:20:44.891255 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.891227 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-136.ec2.internal" event={"ID":"9ce644f9016009147990b020805ff06d","Type":"ContainerDied","Data":"e62b2d0f673552954712774014cc22bed99f8d3f3e61d4488511b83f0aeffd54"} Oct 30 16:20:44.891255 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.891239 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-136.ec2.internal" event={"ID":"9ce644f9016009147990b020805ff06d","Type":"ContainerStarted","Data":"622b6489c48a29a9d5eae0856d7d353098c922549fe22891407212db1d417313"} Oct 30 16:20:44.891366 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.891356 5052 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-136.ec2.internal" Oct 30 16:20:44.926417 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.926400 5052 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 16:20:44.926468 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:20:44.926437 5052 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-131-136.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-136.ec2.internal" Oct 30 16:20:44.927123 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:44.927110 5052 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 16:20:44.927156 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:20:44.927142 5052 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-131-136.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-136.ec2.internal" Oct 30 16:20:45.019368 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.019330 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9ce644f9016009147990b020805ff06d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-136.ec2.internal\" (UID: \"9ce644f9016009147990b020805ff06d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-136.ec2.internal" Oct 30 16:20:45.019491 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.019391 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9ce644f9016009147990b020805ff06d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-136.ec2.internal\" (UID: \"9ce644f9016009147990b020805ff06d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-136.ec2.internal" Oct 30 16:20:45.019491 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.019423 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3de5cf7923bfc1c5b7b74a0700d7b3fa-config\") pod \"kube-apiserver-proxy-ip-10-0-131-136.ec2.internal\" (UID: \"3de5cf7923bfc1c5b7b74a0700d7b3fa\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-136.ec2.internal" Oct 30 16:20:45.119769 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.119704 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9ce644f9016009147990b020805ff06d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-136.ec2.internal\" (UID: \"9ce644f9016009147990b020805ff06d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-136.ec2.internal" Oct 30 16:20:45.119769 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.119743 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9ce644f9016009147990b020805ff06d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-136.ec2.internal\" (UID: \"9ce644f9016009147990b020805ff06d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-136.ec2.internal" Oct 30 16:20:45.119911 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.119807 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9ce644f9016009147990b020805ff06d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-136.ec2.internal\" (UID: \"9ce644f9016009147990b020805ff06d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-136.ec2.internal" Oct 30 16:20:45.119911 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.119868 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9ce644f9016009147990b020805ff06d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-136.ec2.internal\" (UID: \"9ce644f9016009147990b020805ff06d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-136.ec2.internal" Oct 30 16:20:45.119911 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.119898 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3de5cf7923bfc1c5b7b74a0700d7b3fa-config\") pod \"kube-apiserver-proxy-ip-10-0-131-136.ec2.internal\" (UID: \"3de5cf7923bfc1c5b7b74a0700d7b3fa\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-136.ec2.internal" Oct 30 16:20:45.120001 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.119934 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3de5cf7923bfc1c5b7b74a0700d7b3fa-config\") pod \"kube-apiserver-proxy-ip-10-0-131-136.ec2.internal\" (UID: \"3de5cf7923bfc1c5b7b74a0700d7b3fa\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-136.ec2.internal" Oct 30 16:20:45.536497 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.536465 5052 apiserver.go:52] "Watching apiserver" Oct 30 16:20:45.596453 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.596425 5052 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 16:20:45.597178 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.597153 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g69ll","openshift-network-diagnostics/network-check-target-b4ljl","openshift-network-operator/iptables-alerter-d9kr8","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-cswcz","openshift-console-operator/console-operator-7cd56b8dbc-4wrf8","openshift-image-registry/image-registry-f8dddb4c-dcbgb","openshift-kube-storage-version-migrator/migrator-66df5d496b-c87fl","openshift-monitoring/cluster-monitoring-operator-884576c57-sfq4m","openshift-service-ca/service-ca-5fffc97bdf-8hqt5","openshift-multus/network-metrics-daemon-xj6cp","openshift-multus/multus-additional-cni-plugins-x9ddt","openshift-network-diagnostics/network-check-source-6f78d59f99-nwvwc","kube-system/kas-connection-checker-685895c7c9-fhzbv","openshift-multus/multus-7f2mt","openshift-insights/periodic-gathering-cwrtz-glm8n","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-136.ec2.internal","openshift-network-console/networking-console-plugin-86c478bb88-992nb","kube-system/kas-connection-checker-685895c7c9-4dfsk","kube-system/global-pull-secret-syncer-ncfzj","openshift-insights/insights-runtime-extractor-wxvph","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jrdmg","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-5fw2s","openshift-insights/insights-operator-7c4986d449-p4rk6","kube-system/kube-apiserver-proxy-ip-10-0-131-136.ec2.internal","openshift-image-registry/node-ca-2l5ht","openshift-cluster-node-tuning-operator/tuned-c8x4k","openshift-dns/node-resolver-lghnm","openshift-ovn-kubernetes/ovnkube-node-2krz6","openshift-service-ca-operator/service-ca-operator-544d749c8c-4nn7z","kube-system/kas-connection-checker-685895c7c9-cpxbb","openshift-ingress-canary/ingress-canary-l4mhk","kube-system/konnectivity-agent-sg9tg","openshift-ingress/router-default-df85cd6ff-6pd8x","openshift-image-registry/image-registry-65d65dd574-nbb2b","openshift-kube-storage-version-migrator/migrator-66df5d496b-vgrwl","openshift-dns/dns-default-qg6t6"] Oct 30 16:20:45.599114 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.599095 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-f8dddb4c-dcbgb" Oct 30 16:20:45.599485 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.599470 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sfq4m" Oct 30 16:20:45.601007 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.600991 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-cswcz" Oct 30 16:20:45.604324 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.601833 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-df85cd6ff-6pd8x" Oct 30 16:20:45.604324 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.602995 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-992nb" Oct 30 16:20:45.607525 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.607508 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xj6cp" Oct 30 16:20:45.610123 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.610104 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-65d65dd574-nbb2b" Oct 30 16:20:45.611122 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.611106 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qg6t6" Oct 30 16:20:45.611624 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.611608 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wxvph" Oct 30 16:20:45.615551 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.615533 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 30 16:20:45.615707 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.615694 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 30 16:20:45.615943 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.615922 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 30 16:20:45.616008 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.615958 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 30 16:20:45.616008 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.615972 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 16:20:45.616008 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.615982 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 30 16:20:45.616153 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.616028 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 30 16:20:45.616153 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.616064 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 16:20:45.616153 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.616088 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 16:20:45.616405 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.616380 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 30 16:20:45.616768 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.616748 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 16:20:45.616840 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.616777 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 30 16:20:45.616926 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.616903 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 30 16:20:45.617175 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.617161 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 16:20:45.617355 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.617336 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 30 16:20:45.617556 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.617543 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 30 16:20:45.617768 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.617756 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 30 16:20:45.630202 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.630182 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 16:20:45.630266 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.630201 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 16:20:45.633598 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.633537 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 30 16:20:45.633837 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.633820 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 16:20:45.634153 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.634095 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 30 16:20:45.634153 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.634128 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 16:20:45.634287 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.634268 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 16:20:45.634563 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.634545 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-fpzw7\"" Oct 30 16:20:45.634684 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.634664 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 16:20:45.634944 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.634923 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 30 16:20:45.635336 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.635315 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 16:20:45.635665 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.635647 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-m64dw\"" Oct 30 16:20:45.635778 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.635762 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 30 16:20:45.635988 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.635966 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 16:20:45.636076 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.636040 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 16:20:45.636259 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.636226 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 16:20:45.636259 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.636238 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 16:20:45.636539 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.636520 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 16:20:45.636637 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.636567 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 30 16:20:45.636637 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.636614 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 16:20:45.636637 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.636630 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 16:20:45.636783 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.636754 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Oct 30 16:20:45.636838 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.636823 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Oct 30 16:20:45.636888 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.636864 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-l28xz\"" Oct 30 16:20:45.636947 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.636931 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 16:20:45.637011 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.636995 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 30 16:20:45.637057 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.637011 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 16:20:45.637100 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.637053 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 16:20:45.637189 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.637172 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 30 16:20:45.637307 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.637290 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 16:20:45.637392 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.637377 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 30 16:20:45.637450 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.637307 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 30 16:20:45.637573 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.637546 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 16:20:45.637573 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.637562 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 16:20:45.637756 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.637602 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 16:20:45.637756 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.637641 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 16:20:45.637756 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.637654 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 16:20:45.637756 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.637659 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-jrplp\"" Oct 30 16:20:45.637756 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.636228 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 30 16:20:45.637756 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.637682 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 16:20:45.637756 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.636533 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-n7jsz\"" Oct 30 16:20:45.637756 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.637702 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 16:20:45.638112 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.637810 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 30 16:20:45.638112 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.637299 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 30 16:20:45.638112 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.637862 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-hxhjs\"" Oct 30 16:20:45.638112 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.637664 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Oct 30 16:20:45.638112 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.637934 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 16:20:45.638112 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.637974 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 16:20:45.638351 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.638161 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-5kvnn\"" Oct 30 16:20:45.638351 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.638282 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-gfrzf\"" Oct 30 16:20:45.638784 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.638767 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-s8kck\"" Oct 30 16:20:45.638852 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.638838 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Oct 30 16:20:45.638922 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.638906 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 16:20:45.638967 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.638953 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 16:20:45.639016 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.638984 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 16:20:45.639057 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.639023 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 30 16:20:45.639108 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.639058 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 16:20:45.639198 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.639183 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 30 16:20:45.639243 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.639199 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 16:20:45.639291 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.639268 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 30 16:20:45.639387 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.639370 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 16:20:45.639387 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.639380 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 16:20:45.639486 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.639470 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-lngbh\"" Oct 30 16:20:45.639533 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.639494 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 30 16:20:45.639533 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.639524 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 16:20:45.639634 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.639573 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 16:20:45.639694 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.639637 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 16:20:45.639757 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.639692 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 16:20:45.639757 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.639737 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-wncwp\"" Oct 30 16:20:45.639757 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.639750 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 30 16:20:45.639931 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.639845 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 30 16:20:45.639931 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.639865 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-2g8kz\"" Oct 30 16:20:45.640009 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.639980 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 16:20:45.640009 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.639986 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-46f47\"" Oct 30 16:20:45.640009 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.639988 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 30 16:20:45.640124 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.640052 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 16:20:45.640124 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.640074 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 16:20:45.640124 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.640110 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-6q4zp\"" Oct 30 16:20:45.644303 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.644284 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 16:20:45.644425 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.644403 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 16:20:45.644511 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.644460 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 16:20:45.644511 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.644488 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 30 16:20:45.644511 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.644488 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-d8rcz\"" Oct 30 16:20:45.644666 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.644469 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 16:20:45.644666 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.644526 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-m9dmv\"" Oct 30 16:20:45.644666 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.644595 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-pmlj2\"" Oct 30 16:20:45.644666 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.644652 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-sg8tz\"" Oct 30 16:20:45.651177 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.651163 5052 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 16:20:45.661822 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.661807 5052 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-136.ec2.internal" Oct 30 16:20:45.661962 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.661949 5052 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-136.ec2.internal" Oct 30 16:20:45.693076 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.693060 5052 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 16:20:45.693133 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:20:45.693095 5052 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-131-136.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-136.ec2.internal" Oct 30 16:20:45.693214 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.693201 5052 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 16:20:45.693250 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:20:45.693235 5052 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-131-136.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-136.ec2.internal" Oct 30 16:20:45.717123 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.717105 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-rd9lc\"" Oct 30 16:20:45.724373 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.724355 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f79e659b-79c9-43ee-8733-c685f60e87fb-installation-pull-secrets\") pod \"image-registry-65d65dd574-nbb2b\" (UID: \"f79e659b-79c9-43ee-8733-c685f60e87fb\") " pod="openshift-image-registry/image-registry-65d65dd574-nbb2b" Oct 30 16:20:45.724428 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.724381 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a6b95002-74f1-475d-a64a-6495733cf142-socket-dir\") pod \"aws-ebs-csi-driver-node-g69ll\" (UID: \"a6b95002-74f1-475d-a64a-6495733cf142\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g69ll" Oct 30 16:20:45.724428 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.724405 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpgwx\" (UniqueName: \"kubernetes.io/projected/9ea50669-2c08-403d-8d08-f9ffffa5b896-kube-api-access-zpgwx\") pod \"multus-additional-cni-plugins-x9ddt\" (UID: \"9ea50669-2c08-403d-8d08-f9ffffa5b896\") " pod="openshift-multus/multus-additional-cni-plugins-x9ddt" Oct 30 16:20:45.724428 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.724421 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3e90fa4d-66a8-4689-8f41-ca090d6697a9-config-volume\") pod \"dns-default-qg6t6\" (UID: \"3e90fa4d-66a8-4689-8f41-ca090d6697a9\") " pod="openshift-dns/dns-default-qg6t6" Oct 30 16:20:45.724577 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.724454 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t54w7\" (UniqueName: \"kubernetes.io/projected/a0cd0943-8d62-41e4-b3e7-29eaabec7277-kube-api-access-t54w7\") pod \"cluster-monitoring-operator-884576c57-sfq4m\" (UID: \"a0cd0943-8d62-41e4-b3e7-29eaabec7277\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sfq4m" Oct 30 16:20:45.724577 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.724498 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/33b888e2-d349-439b-9aa1-5b519e0e08f7-host-var-lib-cni-bin\") pod \"multus-7f2mt\" (UID: \"33b888e2-d349-439b-9aa1-5b519e0e08f7\") " pod="openshift-multus/multus-7f2mt" Oct 30 16:20:45.724577 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.724549 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mkkh\" (UniqueName: \"kubernetes.io/projected/d719cbcb-9874-4b85-8fe7-e1bf53511235-kube-api-access-2mkkh\") pod \"kas-connection-checker-685895c7c9-4dfsk\" (UID: \"d719cbcb-9874-4b85-8fe7-e1bf53511235\") " pod="kube-system/kas-connection-checker-685895c7c9-4dfsk" Oct 30 16:20:45.724749 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.724602 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-478bs\" (UniqueName: \"kubernetes.io/projected/0d304765-4b62-4cba-bad2-452771d9b201-kube-api-access-478bs\") pod \"network-check-source-6f78d59f99-nwvwc\" (UID: \"0d304765-4b62-4cba-bad2-452771d9b201\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-nwvwc" Oct 30 16:20:45.724749 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.724630 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9fafe556-9653-49df-a851-100b51bb7f47-image-registry-private-configuration\") pod \"image-registry-f8dddb4c-dcbgb\" (UID: \"9fafe556-9653-49df-a851-100b51bb7f47\") " pod="openshift-image-registry/image-registry-f8dddb4c-dcbgb" Oct 30 16:20:45.724749 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.724657 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrppt\" (UniqueName: \"kubernetes.io/projected/9ec1217e-8623-42a6-a052-adee153936a2-kube-api-access-nrppt\") pod \"node-resolver-lghnm\" (UID: \"9ec1217e-8623-42a6-a052-adee153936a2\") " pod="openshift-dns/node-resolver-lghnm" Oct 30 16:20:45.724749 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.724687 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/154b2258-1b09-444b-ad80-2b194195e2cb-service-ca-bundle\") pod \"insights-operator-7c4986d449-p4rk6\" (UID: \"154b2258-1b09-444b-ad80-2b194195e2cb\") " pod="openshift-insights/insights-operator-7c4986d449-p4rk6" Oct 30 16:20:45.724749 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.724693 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f79e659b-79c9-43ee-8733-c685f60e87fb-installation-pull-secrets\") pod \"image-registry-65d65dd574-nbb2b\" (UID: \"f79e659b-79c9-43ee-8733-c685f60e87fb\") " pod="openshift-image-registry/image-registry-65d65dd574-nbb2b" Oct 30 16:20:45.724749 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.724713 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9fafe556-9653-49df-a851-100b51bb7f47-registry-tls\") pod \"image-registry-f8dddb4c-dcbgb\" (UID: \"9fafe556-9653-49df-a851-100b51bb7f47\") " pod="openshift-image-registry/image-registry-f8dddb4c-dcbgb" Oct 30 16:20:45.724749 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.724741 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzp5d\" (UniqueName: \"kubernetes.io/projected/25c51397-40b9-432c-949d-12ba9f36532a-kube-api-access-qzp5d\") pod \"iptables-alerter-d9kr8\" (UID: \"25c51397-40b9-432c-949d-12ba9f36532a\") " pod="openshift-network-operator/iptables-alerter-d9kr8" Oct 30 16:20:45.725072 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.724779 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3e90fa4d-66a8-4689-8f41-ca090d6697a9-config-volume\") pod \"dns-default-qg6t6\" (UID: \"3e90fa4d-66a8-4689-8f41-ca090d6697a9\") " pod="openshift-dns/dns-default-qg6t6" Oct 30 16:20:45.725072 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.724794 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f79e659b-79c9-43ee-8733-c685f60e87fb-registry-certificates\") pod \"image-registry-65d65dd574-nbb2b\" (UID: \"f79e659b-79c9-43ee-8733-c685f60e87fb\") " pod="openshift-image-registry/image-registry-65d65dd574-nbb2b" Oct 30 16:20:45.725072 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.724827 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f2de1ce0-1c1e-4276-80a9-66239e13e352-host-cni-bin\") pod \"ovnkube-node-2krz6\" (UID: \"f2de1ce0-1c1e-4276-80a9-66239e13e352\") " pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:45.725072 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.724854 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f2de1ce0-1c1e-4276-80a9-66239e13e352-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2krz6\" (UID: \"f2de1ce0-1c1e-4276-80a9-66239e13e352\") " pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:45.725072 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.724894 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d6db39f8-bf92-44e8-9144-51eb3b9dee8d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wxvph\" (UID: \"d6db39f8-bf92-44e8-9144-51eb3b9dee8d\") " pod="openshift-insights/insights-runtime-extractor-wxvph" Oct 30 16:20:45.725072 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.724922 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/06332900-7903-4275-8e33-de06de7f6580-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-cswcz\" (UID: \"06332900-7903-4275-8e33-de06de7f6580\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-cswcz" Oct 30 16:20:45.725072 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.724961 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ef87558b-2ba2-41ce-9a0b-9e4b18663a6c-serving-cert\") pod \"service-ca-operator-544d749c8c-4nn7z\" (UID: \"ef87558b-2ba2-41ce-9a0b-9e4b18663a6c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-4nn7z" Oct 30 16:20:45.725072 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.725000 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/33b888e2-d349-439b-9aa1-5b519e0e08f7-multus-cni-dir\") pod \"multus-7f2mt\" (UID: \"33b888e2-d349-439b-9aa1-5b519e0e08f7\") " pod="openshift-multus/multus-7f2mt" Oct 30 16:20:45.725072 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.725014 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f79e659b-79c9-43ee-8733-c685f60e87fb-registry-certificates\") pod \"image-registry-65d65dd574-nbb2b\" (UID: \"f79e659b-79c9-43ee-8733-c685f60e87fb\") " pod="openshift-image-registry/image-registry-65d65dd574-nbb2b" Oct 30 16:20:45.725072 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.725061 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/33b888e2-d349-439b-9aa1-5b519e0e08f7-host-run-k8s-cni-cncf-io\") pod \"multus-7f2mt\" (UID: \"33b888e2-d349-439b-9aa1-5b519e0e08f7\") " pod="openshift-multus/multus-7f2mt" Oct 30 16:20:45.725510 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.725084 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9fafe556-9653-49df-a851-100b51bb7f47-image-registry-private-configuration\") pod \"image-registry-f8dddb4c-dcbgb\" (UID: \"9fafe556-9653-49df-a851-100b51bb7f47\") " pod="openshift-image-registry/image-registry-f8dddb4c-dcbgb" Oct 30 16:20:45.725510 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.725089 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/209633e5-92f7-4a7f-9d68-3d48473ed2bf-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jrdmg\" (UID: \"209633e5-92f7-4a7f-9d68-3d48473ed2bf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jrdmg" Oct 30 16:20:45.725510 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.725132 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f2de1ce0-1c1e-4276-80a9-66239e13e352-host-run-ovn-kubernetes\") pod \"ovnkube-node-2krz6\" (UID: \"f2de1ce0-1c1e-4276-80a9-66239e13e352\") " pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:45.725510 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.725197 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4hr8\" (UniqueName: \"kubernetes.io/projected/90d73ca9-5d88-4467-99fa-d4e1a67f3828-kube-api-access-b4hr8\") pod \"kas-connection-checker-685895c7c9-cpxbb\" (UID: \"90d73ca9-5d88-4467-99fa-d4e1a67f3828\") " pod="kube-system/kas-connection-checker-685895c7c9-cpxbb" Oct 30 16:20:45.725510 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.725234 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/33b888e2-d349-439b-9aa1-5b519e0e08f7-hostroot\") pod \"multus-7f2mt\" (UID: \"33b888e2-d349-439b-9aa1-5b519e0e08f7\") " pod="openshift-multus/multus-7f2mt" Oct 30 16:20:45.725510 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.725259 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ef87558b-2ba2-41ce-9a0b-9e4b18663a6c-serving-cert\") pod \"service-ca-operator-544d749c8c-4nn7z\" (UID: \"ef87558b-2ba2-41ce-9a0b-9e4b18663a6c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-4nn7z" Oct 30 16:20:45.725510 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.725366 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/749b2b57-f20e-4cdb-8d89-d2bac352f751-etc-systemd\") pod \"tuned-c8x4k\" (UID: \"749b2b57-f20e-4cdb-8d89-d2bac352f751\") " pod="openshift-cluster-node-tuning-operator/tuned-c8x4k" Oct 30 16:20:45.725510 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.725400 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttr7v\" (UniqueName: \"kubernetes.io/projected/341d9139-495e-4ee2-80c3-a20370039948-kube-api-access-ttr7v\") pod \"migrator-66df5d496b-c87fl\" (UID: \"341d9139-495e-4ee2-80c3-a20370039948\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-c87fl" Oct 30 16:20:45.725510 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.725407 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/154b2258-1b09-444b-ad80-2b194195e2cb-service-ca-bundle\") pod \"insights-operator-7c4986d449-p4rk6\" (UID: \"154b2258-1b09-444b-ad80-2b194195e2cb\") " pod="openshift-insights/insights-operator-7c4986d449-p4rk6" Oct 30 16:20:45.725510 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.725393 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/209633e5-92f7-4a7f-9d68-3d48473ed2bf-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jrdmg\" (UID: \"209633e5-92f7-4a7f-9d68-3d48473ed2bf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jrdmg" Oct 30 16:20:45.725510 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.725438 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d10078a0-c21c-4f18-9c5a-4915fb7c8af3-serviceca\") pod \"node-ca-2l5ht\" (UID: \"d10078a0-c21c-4f18-9c5a-4915fb7c8af3\") " pod="openshift-image-registry/node-ca-2l5ht" Oct 30 16:20:45.726015 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.725596 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9ea50669-2c08-403d-8d08-f9ffffa5b896-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-x9ddt\" (UID: \"9ea50669-2c08-403d-8d08-f9ffffa5b896\") " pod="openshift-multus/multus-additional-cni-plugins-x9ddt" Oct 30 16:20:45.726015 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.725645 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e90fa4d-66a8-4689-8f41-ca090d6697a9-tmp-dir\") pod \"dns-default-qg6t6\" (UID: \"3e90fa4d-66a8-4689-8f41-ca090d6697a9\") " pod="openshift-dns/dns-default-qg6t6" Oct 30 16:20:45.726015 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.725672 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/749b2b57-f20e-4cdb-8d89-d2bac352f751-tmp\") pod \"tuned-c8x4k\" (UID: \"749b2b57-f20e-4cdb-8d89-d2bac352f751\") " pod="openshift-cluster-node-tuning-operator/tuned-c8x4k" Oct 30 16:20:45.726015 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.725700 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/154b2258-1b09-444b-ad80-2b194195e2cb-tmp\") pod \"insights-operator-7c4986d449-p4rk6\" (UID: \"154b2258-1b09-444b-ad80-2b194195e2cb\") " pod="openshift-insights/insights-operator-7c4986d449-p4rk6" Oct 30 16:20:45.726015 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.725726 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/154b2258-1b09-444b-ad80-2b194195e2cb-serving-cert\") pod \"insights-operator-7c4986d449-p4rk6\" (UID: \"154b2258-1b09-444b-ad80-2b194195e2cb\") " pod="openshift-insights/insights-operator-7c4986d449-p4rk6" Oct 30 16:20:45.726015 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.725732 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e90fa4d-66a8-4689-8f41-ca090d6697a9-tmp-dir\") pod \"dns-default-qg6t6\" (UID: \"3e90fa4d-66a8-4689-8f41-ca090d6697a9\") " pod="openshift-dns/dns-default-qg6t6" Oct 30 16:20:45.726015 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.725750 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4f0f46f4-5cda-4d67-b6f3-47df06eb844c-serving-cert\") pod \"console-operator-7cd56b8dbc-4wrf8\" (UID: \"4f0f46f4-5cda-4d67-b6f3-47df06eb844c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-4wrf8" Oct 30 16:20:45.726015 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.725760 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d10078a0-c21c-4f18-9c5a-4915fb7c8af3-serviceca\") pod \"node-ca-2l5ht\" (UID: \"d10078a0-c21c-4f18-9c5a-4915fb7c8af3\") " pod="openshift-image-registry/node-ca-2l5ht" Oct 30 16:20:45.726015 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.725777 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f79e659b-79c9-43ee-8733-c685f60e87fb-ca-trust-extracted\") pod \"image-registry-65d65dd574-nbb2b\" (UID: \"f79e659b-79c9-43ee-8733-c685f60e87fb\") " pod="openshift-image-registry/image-registry-65d65dd574-nbb2b" Oct 30 16:20:45.726015 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.725818 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/154b2258-1b09-444b-ad80-2b194195e2cb-tmp\") pod \"insights-operator-7c4986d449-p4rk6\" (UID: \"154b2258-1b09-444b-ad80-2b194195e2cb\") " pod="openshift-insights/insights-operator-7c4986d449-p4rk6" Oct 30 16:20:45.726015 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.725822 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/749b2b57-f20e-4cdb-8d89-d2bac352f751-tmp\") pod \"tuned-c8x4k\" (UID: \"749b2b57-f20e-4cdb-8d89-d2bac352f751\") " pod="openshift-cluster-node-tuning-operator/tuned-c8x4k" Oct 30 16:20:45.726015 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.725860 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9fafe556-9653-49df-a851-100b51bb7f47-installation-pull-secrets\") pod \"image-registry-f8dddb4c-dcbgb\" (UID: \"9fafe556-9653-49df-a851-100b51bb7f47\") " pod="openshift-image-registry/image-registry-f8dddb4c-dcbgb" Oct 30 16:20:45.726015 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.725865 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f79e659b-79c9-43ee-8733-c685f60e87fb-ca-trust-extracted\") pod \"image-registry-65d65dd574-nbb2b\" (UID: \"f79e659b-79c9-43ee-8733-c685f60e87fb\") " pod="openshift-image-registry/image-registry-65d65dd574-nbb2b" Oct 30 16:20:45.726015 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.725869 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9ea50669-2c08-403d-8d08-f9ffffa5b896-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-x9ddt\" (UID: \"9ea50669-2c08-403d-8d08-f9ffffa5b896\") " pod="openshift-multus/multus-additional-cni-plugins-x9ddt" Oct 30 16:20:45.726015 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.725883 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mc2zh\" (UniqueName: \"kubernetes.io/projected/c471cf39-07e7-4f4d-87d5-0c6803fd54a0-kube-api-access-mc2zh\") pod \"periodic-gathering-cwrtz-glm8n\" (UID: \"c471cf39-07e7-4f4d-87d5-0c6803fd54a0\") " pod="openshift-insights/periodic-gathering-cwrtz-glm8n" Oct 30 16:20:45.726015 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.725905 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4f0f46f4-5cda-4d67-b6f3-47df06eb844c-serving-cert\") pod \"console-operator-7cd56b8dbc-4wrf8\" (UID: \"4f0f46f4-5cda-4d67-b6f3-47df06eb844c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-4wrf8" Oct 30 16:20:45.726015 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.725924 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1042378b-9f46-4e19-88e4-2628e532e8c3-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-992nb\" (UID: \"1042378b-9f46-4e19-88e4-2628e532e8c3\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-992nb" Oct 30 16:20:45.726015 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.725944 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f79e659b-79c9-43ee-8733-c685f60e87fb-image-registry-private-configuration\") pod \"image-registry-65d65dd574-nbb2b\" (UID: \"f79e659b-79c9-43ee-8733-c685f60e87fb\") " pod="openshift-image-registry/image-registry-65d65dd574-nbb2b" Oct 30 16:20:45.726015 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.725968 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a6b95002-74f1-475d-a64a-6495733cf142-etc-selinux\") pod \"aws-ebs-csi-driver-node-g69ll\" (UID: \"a6b95002-74f1-475d-a64a-6495733cf142\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g69ll" Oct 30 16:20:45.726015 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.725994 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4d6v2\" (UniqueName: \"kubernetes.io/projected/06332900-7903-4275-8e33-de06de7f6580-kube-api-access-4d6v2\") pod \"cluster-samples-operator-6ddb6c8b94-cswcz\" (UID: \"06332900-7903-4275-8e33-de06de7f6580\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-cswcz" Oct 30 16:20:45.726966 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.726101 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9fafe556-9653-49df-a851-100b51bb7f47-installation-pull-secrets\") pod \"image-registry-f8dddb4c-dcbgb\" (UID: \"9fafe556-9653-49df-a851-100b51bb7f47\") " pod="openshift-image-registry/image-registry-f8dddb4c-dcbgb" Oct 30 16:20:45.726966 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.726116 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5tsm\" (UniqueName: \"kubernetes.io/projected/6f3d36d6-5098-43cc-8f8a-bd83d5dea5fc-kube-api-access-v5tsm\") pod \"network-metrics-daemon-xj6cp\" (UID: \"6f3d36d6-5098-43cc-8f8a-bd83d5dea5fc\") " pod="openshift-multus/network-metrics-daemon-xj6cp" Oct 30 16:20:45.726966 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.726162 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qc9ck\" (UniqueName: \"kubernetes.io/projected/774953d5-23d3-4db1-aec9-3b78ef81017d-kube-api-access-qc9ck\") pod \"ingress-canary-l4mhk\" (UID: \"774953d5-23d3-4db1-aec9-3b78ef81017d\") " pod="openshift-ingress-canary/ingress-canary-l4mhk" Oct 30 16:20:45.726966 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.726188 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c471cf39-07e7-4f4d-87d5-0c6803fd54a0-archives-path\") pod \"periodic-gathering-cwrtz-glm8n\" (UID: \"c471cf39-07e7-4f4d-87d5-0c6803fd54a0\") " pod="openshift-insights/periodic-gathering-cwrtz-glm8n" Oct 30 16:20:45.726966 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.726189 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f79e659b-79c9-43ee-8733-c685f60e87fb-image-registry-private-configuration\") pod \"image-registry-65d65dd574-nbb2b\" (UID: \"f79e659b-79c9-43ee-8733-c685f60e87fb\") " pod="openshift-image-registry/image-registry-65d65dd574-nbb2b" Oct 30 16:20:45.726966 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.726194 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/154b2258-1b09-444b-ad80-2b194195e2cb-serving-cert\") pod \"insights-operator-7c4986d449-p4rk6\" (UID: \"154b2258-1b09-444b-ad80-2b194195e2cb\") " pod="openshift-insights/insights-operator-7c4986d449-p4rk6" Oct 30 16:20:45.726966 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.726247 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c471cf39-07e7-4f4d-87d5-0c6803fd54a0-archives-path\") pod \"periodic-gathering-cwrtz-glm8n\" (UID: \"c471cf39-07e7-4f4d-87d5-0c6803fd54a0\") " pod="openshift-insights/periodic-gathering-cwrtz-glm8n" Oct 30 16:20:45.726966 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.726311 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1042378b-9f46-4e19-88e4-2628e532e8c3-nginx-conf\") pod \"networking-console-plugin-86c478bb88-992nb\" (UID: \"1042378b-9f46-4e19-88e4-2628e532e8c3\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-992nb" Oct 30 16:20:45.726966 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.726334 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/25c51397-40b9-432c-949d-12ba9f36532a-host-slash\") pod \"iptables-alerter-d9kr8\" (UID: \"25c51397-40b9-432c-949d-12ba9f36532a\") " pod="openshift-network-operator/iptables-alerter-d9kr8" Oct 30 16:20:45.726966 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.726350 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7d4j4\" (UniqueName: \"kubernetes.io/projected/f2de1ce0-1c1e-4276-80a9-66239e13e352-kube-api-access-7d4j4\") pod \"ovnkube-node-2krz6\" (UID: \"f2de1ce0-1c1e-4276-80a9-66239e13e352\") " pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:45.726966 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.726366 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cw6q9\" (UniqueName: \"kubernetes.io/projected/b2956847-8307-4f69-8921-bb0a0ac0aaf7-kube-api-access-cw6q9\") pod \"service-ca-5fffc97bdf-8hqt5\" (UID: \"b2956847-8307-4f69-8921-bb0a0ac0aaf7\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8hqt5" Oct 30 16:20:45.726966 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.726410 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/25c51397-40b9-432c-949d-12ba9f36532a-iptables-alerter-script\") pod \"iptables-alerter-d9kr8\" (UID: \"25c51397-40b9-432c-949d-12ba9f36532a\") " pod="openshift-network-operator/iptables-alerter-d9kr8" Oct 30 16:20:45.726966 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.726431 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a6b95002-74f1-475d-a64a-6495733cf142-registration-dir\") pod \"aws-ebs-csi-driver-node-g69ll\" (UID: \"a6b95002-74f1-475d-a64a-6495733cf142\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g69ll" Oct 30 16:20:45.726966 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.726448 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/749b2b57-f20e-4cdb-8d89-d2bac352f751-var-lib-kubelet\") pod \"tuned-c8x4k\" (UID: \"749b2b57-f20e-4cdb-8d89-d2bac352f751\") " pod="openshift-cluster-node-tuning-operator/tuned-c8x4k" Oct 30 16:20:45.726966 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.726460 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1042378b-9f46-4e19-88e4-2628e532e8c3-nginx-conf\") pod \"networking-console-plugin-86c478bb88-992nb\" (UID: \"1042378b-9f46-4e19-88e4-2628e532e8c3\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-992nb" Oct 30 16:20:45.726966 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.726464 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfgdd\" (UniqueName: \"kubernetes.io/projected/749b2b57-f20e-4cdb-8d89-d2bac352f751-kube-api-access-dfgdd\") pod \"tuned-c8x4k\" (UID: \"749b2b57-f20e-4cdb-8d89-d2bac352f751\") " pod="openshift-cluster-node-tuning-operator/tuned-c8x4k" Oct 30 16:20:45.726966 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.726502 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4c515c55-0717-4974-9f1a-68e0a25984cd-agent-certs\") pod \"konnectivity-agent-sg9tg\" (UID: \"4c515c55-0717-4974-9f1a-68e0a25984cd\") " pod="kube-system/konnectivity-agent-sg9tg" Oct 30 16:20:45.726966 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.726543 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f2de1ce0-1c1e-4276-80a9-66239e13e352-etc-openvswitch\") pod \"ovnkube-node-2krz6\" (UID: \"f2de1ce0-1c1e-4276-80a9-66239e13e352\") " pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:45.726966 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.726630 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/749b2b57-f20e-4cdb-8d89-d2bac352f751-etc-sysctl-conf\") pod \"tuned-c8x4k\" (UID: \"749b2b57-f20e-4cdb-8d89-d2bac352f751\") " pod="openshift-cluster-node-tuning-operator/tuned-c8x4k" Oct 30 16:20:45.726966 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.726663 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d6db39f8-bf92-44e8-9144-51eb3b9dee8d-data-volume\") pod \"insights-runtime-extractor-wxvph\" (UID: \"d6db39f8-bf92-44e8-9144-51eb3b9dee8d\") " pod="openshift-insights/insights-runtime-extractor-wxvph" Oct 30 16:20:45.726966 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.726730 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/25c51397-40b9-432c-949d-12ba9f36532a-iptables-alerter-script\") pod \"iptables-alerter-d9kr8\" (UID: \"25c51397-40b9-432c-949d-12ba9f36532a\") " pod="openshift-network-operator/iptables-alerter-d9kr8" Oct 30 16:20:45.726966 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.726739 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c471cf39-07e7-4f4d-87d5-0c6803fd54a0-service-ca-bundle\") pod \"periodic-gathering-cwrtz-glm8n\" (UID: \"c471cf39-07e7-4f4d-87d5-0c6803fd54a0\") " pod="openshift-insights/periodic-gathering-cwrtz-glm8n" Oct 30 16:20:45.726966 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.726772 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c471cf39-07e7-4f4d-87d5-0c6803fd54a0-trusted-ca-bundle\") pod \"periodic-gathering-cwrtz-glm8n\" (UID: \"c471cf39-07e7-4f4d-87d5-0c6803fd54a0\") " pod="openshift-insights/periodic-gathering-cwrtz-glm8n" Oct 30 16:20:45.726966 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.726801 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/33b888e2-d349-439b-9aa1-5b519e0e08f7-host-run-netns\") pod \"multus-7f2mt\" (UID: \"33b888e2-d349-439b-9aa1-5b519e0e08f7\") " pod="openshift-multus/multus-7f2mt" Oct 30 16:20:45.726966 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.726813 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d6db39f8-bf92-44e8-9144-51eb3b9dee8d-data-volume\") pod \"insights-runtime-extractor-wxvph\" (UID: \"d6db39f8-bf92-44e8-9144-51eb3b9dee8d\") " pod="openshift-insights/insights-runtime-extractor-wxvph" Oct 30 16:20:45.726966 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.726813 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4c515c55-0717-4974-9f1a-68e0a25984cd-agent-certs\") pod \"konnectivity-agent-sg9tg\" (UID: \"4c515c55-0717-4974-9f1a-68e0a25984cd\") " pod="kube-system/konnectivity-agent-sg9tg" Oct 30 16:20:45.726966 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.726826 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f2de1ce0-1c1e-4276-80a9-66239e13e352-var-lib-openvswitch\") pod \"ovnkube-node-2krz6\" (UID: \"f2de1ce0-1c1e-4276-80a9-66239e13e352\") " pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:45.726966 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.726878 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljv94\" (UniqueName: \"kubernetes.io/projected/2d1ed7d8-37cc-4652-b225-3a715b755e48-kube-api-access-ljv94\") pod \"network-check-target-b4ljl\" (UID: \"2d1ed7d8-37cc-4652-b225-3a715b755e48\") " pod="openshift-network-diagnostics/network-check-target-b4ljl" Oct 30 16:20:45.726966 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.726911 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjp8r\" (UniqueName: \"kubernetes.io/projected/3e90fa4d-66a8-4689-8f41-ca090d6697a9-kube-api-access-pjp8r\") pod \"dns-default-qg6t6\" (UID: \"3e90fa4d-66a8-4689-8f41-ca090d6697a9\") " pod="openshift-dns/dns-default-qg6t6" Oct 30 16:20:45.726966 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.726938 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f3d36d6-5098-43cc-8f8a-bd83d5dea5fc-metrics-certs\") pod \"network-metrics-daemon-xj6cp\" (UID: \"6f3d36d6-5098-43cc-8f8a-bd83d5dea5fc\") " pod="openshift-multus/network-metrics-daemon-xj6cp" Oct 30 16:20:45.726966 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.726962 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/749b2b57-f20e-4cdb-8d89-d2bac352f751-etc-modprobe-d\") pod \"tuned-c8x4k\" (UID: \"749b2b57-f20e-4cdb-8d89-d2bac352f751\") " pod="openshift-cluster-node-tuning-operator/tuned-c8x4k" Oct 30 16:20:45.726966 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.726986 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/33b888e2-d349-439b-9aa1-5b519e0e08f7-cni-binary-copy\") pod \"multus-7f2mt\" (UID: \"33b888e2-d349-439b-9aa1-5b519e0e08f7\") " pod="openshift-multus/multus-7f2mt" Oct 30 16:20:45.728287 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.727012 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/33b888e2-d349-439b-9aa1-5b519e0e08f7-multus-conf-dir\") pod \"multus-7f2mt\" (UID: \"33b888e2-d349-439b-9aa1-5b519e0e08f7\") " pod="openshift-multus/multus-7f2mt" Oct 30 16:20:45.728287 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.727038 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/33b888e2-d349-439b-9aa1-5b519e0e08f7-host-run-multus-certs\") pod \"multus-7f2mt\" (UID: \"33b888e2-d349-439b-9aa1-5b519e0e08f7\") " pod="openshift-multus/multus-7f2mt" Oct 30 16:20:45.728287 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.727063 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c471cf39-07e7-4f4d-87d5-0c6803fd54a0-trusted-ca-bundle\") pod \"periodic-gathering-cwrtz-glm8n\" (UID: \"c471cf39-07e7-4f4d-87d5-0c6803fd54a0\") " pod="openshift-insights/periodic-gathering-cwrtz-glm8n" Oct 30 16:20:45.728287 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.727087 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zlg7\" (UniqueName: \"kubernetes.io/projected/a6b95002-74f1-475d-a64a-6495733cf142-kube-api-access-8zlg7\") pod \"aws-ebs-csi-driver-node-g69ll\" (UID: \"a6b95002-74f1-475d-a64a-6495733cf142\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g69ll" Oct 30 16:20:45.728287 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.727179 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f2de1ce0-1c1e-4276-80a9-66239e13e352-host-run-netns\") pod \"ovnkube-node-2krz6\" (UID: \"f2de1ce0-1c1e-4276-80a9-66239e13e352\") " pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:45.728287 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.727188 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c471cf39-07e7-4f4d-87d5-0c6803fd54a0-service-ca-bundle\") pod \"periodic-gathering-cwrtz-glm8n\" (UID: \"c471cf39-07e7-4f4d-87d5-0c6803fd54a0\") " pod="openshift-insights/periodic-gathering-cwrtz-glm8n" Oct 30 16:20:45.728287 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.727223 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f2de1ce0-1c1e-4276-80a9-66239e13e352-run-systemd\") pod \"ovnkube-node-2krz6\" (UID: \"f2de1ce0-1c1e-4276-80a9-66239e13e352\") " pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:45.728287 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.727263 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ec1217e-8623-42a6-a052-adee153936a2-tmp-dir\") pod \"node-resolver-lghnm\" (UID: \"9ec1217e-8623-42a6-a052-adee153936a2\") " pod="openshift-dns/node-resolver-lghnm" Oct 30 16:20:45.728287 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.727267 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/33b888e2-d349-439b-9aa1-5b519e0e08f7-cni-binary-copy\") pod \"multus-7f2mt\" (UID: \"33b888e2-d349-439b-9aa1-5b519e0e08f7\") " pod="openshift-multus/multus-7f2mt" Oct 30 16:20:45.728287 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.727312 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f79e659b-79c9-43ee-8733-c685f60e87fb-trusted-ca\") pod \"image-registry-65d65dd574-nbb2b\" (UID: \"f79e659b-79c9-43ee-8733-c685f60e87fb\") " pod="openshift-image-registry/image-registry-65d65dd574-nbb2b" Oct 30 16:20:45.728287 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.727319 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ec1217e-8623-42a6-a052-adee153936a2-tmp-dir\") pod \"node-resolver-lghnm\" (UID: \"9ec1217e-8623-42a6-a052-adee153936a2\") " pod="openshift-dns/node-resolver-lghnm" Oct 30 16:20:45.728287 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.727357 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/33b888e2-d349-439b-9aa1-5b519e0e08f7-etc-kubernetes\") pod \"multus-7f2mt\" (UID: \"33b888e2-d349-439b-9aa1-5b519e0e08f7\") " pod="openshift-multus/multus-7f2mt" Oct 30 16:20:45.728287 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.727376 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7dg7\" (UniqueName: \"kubernetes.io/projected/b54d0e33-3943-4e1c-9889-dc65864b4efb-kube-api-access-v7dg7\") pod \"migrator-66df5d496b-vgrwl\" (UID: \"b54d0e33-3943-4e1c-9889-dc65864b4efb\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vgrwl" Oct 30 16:20:45.728287 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.727520 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f2de1ce0-1c1e-4276-80a9-66239e13e352-ovnkube-script-lib\") pod \"ovnkube-node-2krz6\" (UID: \"f2de1ce0-1c1e-4276-80a9-66239e13e352\") " pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:45.728287 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.727561 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4f0f46f4-5cda-4d67-b6f3-47df06eb844c-config\") pod \"console-operator-7cd56b8dbc-4wrf8\" (UID: \"4f0f46f4-5cda-4d67-b6f3-47df06eb844c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-4wrf8" Oct 30 16:20:45.728287 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.727626 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/b99d8a1f-000b-48ff-b4b0-7a62bc79c032-default-certificate\") pod \"router-default-df85cd6ff-6pd8x\" (UID: \"b99d8a1f-000b-48ff-b4b0-7a62bc79c032\") " pod="openshift-ingress/router-default-df85cd6ff-6pd8x" Oct 30 16:20:45.728287 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.727681 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/33b888e2-d349-439b-9aa1-5b519e0e08f7-multus-daemon-config\") pod \"multus-7f2mt\" (UID: \"33b888e2-d349-439b-9aa1-5b519e0e08f7\") " pod="openshift-multus/multus-7f2mt" Oct 30 16:20:45.728287 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.727758 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4f0f46f4-5cda-4d67-b6f3-47df06eb844c-config\") pod \"console-operator-7cd56b8dbc-4wrf8\" (UID: \"4f0f46f4-5cda-4d67-b6f3-47df06eb844c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-4wrf8" Oct 30 16:20:45.728287 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.727803 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f2de1ce0-1c1e-4276-80a9-66239e13e352-ovnkube-script-lib\") pod \"ovnkube-node-2krz6\" (UID: \"f2de1ce0-1c1e-4276-80a9-66239e13e352\") " pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:45.728287 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.727803 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-849kr\" (UniqueName: \"kubernetes.io/projected/4d9a19d6-94d4-4490-8f93-89ca745b768f-kube-api-access-849kr\") pod \"volume-data-source-validator-58c9f5f46d-5fw2s\" (UID: \"4d9a19d6-94d4-4490-8f93-89ca745b768f\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-5fw2s" Oct 30 16:20:45.728287 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.727878 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/b99d8a1f-000b-48ff-b4b0-7a62bc79c032-default-certificate\") pod \"router-default-df85cd6ff-6pd8x\" (UID: \"b99d8a1f-000b-48ff-b4b0-7a62bc79c032\") " pod="openshift-ingress/router-default-df85cd6ff-6pd8x" Oct 30 16:20:45.728287 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.727847 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/029f4ecb-eb6b-4693-9e9e-ca61f554dd33-kubelet-config\") pod \"global-pull-secret-syncer-ncfzj\" (UID: \"029f4ecb-eb6b-4693-9e9e-ca61f554dd33\") " pod="kube-system/global-pull-secret-syncer-ncfzj" Oct 30 16:20:45.728287 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.727903 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/33b888e2-d349-439b-9aa1-5b519e0e08f7-multus-daemon-config\") pod \"multus-7f2mt\" (UID: \"33b888e2-d349-439b-9aa1-5b519e0e08f7\") " pod="openshift-multus/multus-7f2mt" Oct 30 16:20:45.728287 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.727920 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f2de1ce0-1c1e-4276-80a9-66239e13e352-systemd-units\") pod \"ovnkube-node-2krz6\" (UID: \"f2de1ce0-1c1e-4276-80a9-66239e13e352\") " pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:45.728287 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.727941 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f2de1ce0-1c1e-4276-80a9-66239e13e352-host-cni-netd\") pod \"ovnkube-node-2krz6\" (UID: \"f2de1ce0-1c1e-4276-80a9-66239e13e352\") " pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:45.728287 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.727958 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwfbm\" (UniqueName: \"kubernetes.io/projected/d6db39f8-bf92-44e8-9144-51eb3b9dee8d-kube-api-access-cwfbm\") pod \"insights-runtime-extractor-wxvph\" (UID: \"d6db39f8-bf92-44e8-9144-51eb3b9dee8d\") " pod="openshift-insights/insights-runtime-extractor-wxvph" Oct 30 16:20:45.728287 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.728020 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlqc8\" (UniqueName: \"kubernetes.io/projected/ef87558b-2ba2-41ce-9a0b-9e4b18663a6c-kube-api-access-qlqc8\") pod \"service-ca-operator-544d749c8c-4nn7z\" (UID: \"ef87558b-2ba2-41ce-9a0b-9e4b18663a6c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-4nn7z" Oct 30 16:20:45.728287 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.728051 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/33b888e2-d349-439b-9aa1-5b519e0e08f7-system-cni-dir\") pod \"multus-7f2mt\" (UID: \"33b888e2-d349-439b-9aa1-5b519e0e08f7\") " pod="openshift-multus/multus-7f2mt" Oct 30 16:20:45.728287 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.728147 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z65wk\" (UniqueName: \"kubernetes.io/projected/209633e5-92f7-4a7f-9d68-3d48473ed2bf-kube-api-access-z65wk\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jrdmg\" (UID: \"209633e5-92f7-4a7f-9d68-3d48473ed2bf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jrdmg" Oct 30 16:20:45.728287 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.728168 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9ea50669-2c08-403d-8d08-f9ffffa5b896-cni-binary-copy\") pod \"multus-additional-cni-plugins-x9ddt\" (UID: \"9ea50669-2c08-403d-8d08-f9ffffa5b896\") " pod="openshift-multus/multus-additional-cni-plugins-x9ddt" Oct 30 16:20:45.728287 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.728184 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/029f4ecb-eb6b-4693-9e9e-ca61f554dd33-original-pull-secret\") pod \"global-pull-secret-syncer-ncfzj\" (UID: \"029f4ecb-eb6b-4693-9e9e-ca61f554dd33\") " pod="kube-system/global-pull-secret-syncer-ncfzj" Oct 30 16:20:45.728287 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.728167 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f79e659b-79c9-43ee-8733-c685f60e87fb-trusted-ca\") pod \"image-registry-65d65dd574-nbb2b\" (UID: \"f79e659b-79c9-43ee-8733-c685f60e87fb\") " pod="openshift-image-registry/image-registry-65d65dd574-nbb2b" Oct 30 16:20:45.728287 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.728205 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f2de1ce0-1c1e-4276-80a9-66239e13e352-run-openvswitch\") pod \"ovnkube-node-2krz6\" (UID: \"f2de1ce0-1c1e-4276-80a9-66239e13e352\") " pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:45.728287 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.728237 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f2de1ce0-1c1e-4276-80a9-66239e13e352-node-log\") pod \"ovnkube-node-2krz6\" (UID: \"f2de1ce0-1c1e-4276-80a9-66239e13e352\") " pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:45.729602 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.728346 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4f0f46f4-5cda-4d67-b6f3-47df06eb844c-trusted-ca\") pod \"console-operator-7cd56b8dbc-4wrf8\" (UID: \"4f0f46f4-5cda-4d67-b6f3-47df06eb844c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-4wrf8" Oct 30 16:20:45.729602 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.728347 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9ea50669-2c08-403d-8d08-f9ffffa5b896-cni-binary-copy\") pod \"multus-additional-cni-plugins-x9ddt\" (UID: \"9ea50669-2c08-403d-8d08-f9ffffa5b896\") " pod="openshift-multus/multus-additional-cni-plugins-x9ddt" Oct 30 16:20:45.729602 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.728375 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/b99d8a1f-000b-48ff-b4b0-7a62bc79c032-stats-auth\") pod \"router-default-df85cd6ff-6pd8x\" (UID: \"b99d8a1f-000b-48ff-b4b0-7a62bc79c032\") " pod="openshift-ingress/router-default-df85cd6ff-6pd8x" Oct 30 16:20:45.729602 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.728396 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/749b2b57-f20e-4cdb-8d89-d2bac352f751-sys\") pod \"tuned-c8x4k\" (UID: \"749b2b57-f20e-4cdb-8d89-d2bac352f751\") " pod="openshift-cluster-node-tuning-operator/tuned-c8x4k" Oct 30 16:20:45.729602 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.728417 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f2de1ce0-1c1e-4276-80a9-66239e13e352-env-overrides\") pod \"ovnkube-node-2krz6\" (UID: \"f2de1ce0-1c1e-4276-80a9-66239e13e352\") " pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:45.729602 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.728435 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9fafe556-9653-49df-a851-100b51bb7f47-registry-certificates\") pod \"image-registry-f8dddb4c-dcbgb\" (UID: \"9fafe556-9653-49df-a851-100b51bb7f47\") " pod="openshift-image-registry/image-registry-f8dddb4c-dcbgb" Oct 30 16:20:45.729602 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.728447 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/029f4ecb-eb6b-4693-9e9e-ca61f554dd33-original-pull-secret\") pod \"global-pull-secret-syncer-ncfzj\" (UID: \"029f4ecb-eb6b-4693-9e9e-ca61f554dd33\") " pod="kube-system/global-pull-secret-syncer-ncfzj" Oct 30 16:20:45.729602 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.728451 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25jwd\" (UniqueName: \"kubernetes.io/projected/d10078a0-c21c-4f18-9c5a-4915fb7c8af3-kube-api-access-25jwd\") pod \"node-ca-2l5ht\" (UID: \"d10078a0-c21c-4f18-9c5a-4915fb7c8af3\") " pod="openshift-image-registry/node-ca-2l5ht" Oct 30 16:20:45.729602 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.728499 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/749b2b57-f20e-4cdb-8d89-d2bac352f751-etc-kubernetes\") pod \"tuned-c8x4k\" (UID: \"749b2b57-f20e-4cdb-8d89-d2bac352f751\") " pod="openshift-cluster-node-tuning-operator/tuned-c8x4k" Oct 30 16:20:45.729602 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.728536 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d6db39f8-bf92-44e8-9144-51eb3b9dee8d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wxvph\" (UID: \"d6db39f8-bf92-44e8-9144-51eb3b9dee8d\") " pod="openshift-insights/insights-runtime-extractor-wxvph" Oct 30 16:20:45.729602 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.728571 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/b99d8a1f-000b-48ff-b4b0-7a62bc79c032-stats-auth\") pod \"router-default-df85cd6ff-6pd8x\" (UID: \"b99d8a1f-000b-48ff-b4b0-7a62bc79c032\") " pod="openshift-ingress/router-default-df85cd6ff-6pd8x" Oct 30 16:20:45.729602 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.728596 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f2de1ce0-1c1e-4276-80a9-66239e13e352-env-overrides\") pod \"ovnkube-node-2krz6\" (UID: \"f2de1ce0-1c1e-4276-80a9-66239e13e352\") " pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:45.729602 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.728614 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4f0f46f4-5cda-4d67-b6f3-47df06eb844c-trusted-ca\") pod \"console-operator-7cd56b8dbc-4wrf8\" (UID: \"4f0f46f4-5cda-4d67-b6f3-47df06eb844c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-4wrf8" Oct 30 16:20:45.729602 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.728627 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/33b888e2-d349-439b-9aa1-5b519e0e08f7-multus-socket-dir-parent\") pod \"multus-7f2mt\" (UID: \"33b888e2-d349-439b-9aa1-5b519e0e08f7\") " pod="openshift-multus/multus-7f2mt" Oct 30 16:20:45.729602 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.728620 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9fafe556-9653-49df-a851-100b51bb7f47-registry-certificates\") pod \"image-registry-f8dddb4c-dcbgb\" (UID: \"9fafe556-9653-49df-a851-100b51bb7f47\") " pod="openshift-image-registry/image-registry-f8dddb4c-dcbgb" Oct 30 16:20:45.729602 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.728663 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f2de1ce0-1c1e-4276-80a9-66239e13e352-run-ovn\") pod \"ovnkube-node-2krz6\" (UID: \"f2de1ce0-1c1e-4276-80a9-66239e13e352\") " pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:45.729602 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.728697 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a6b95002-74f1-475d-a64a-6495733cf142-device-dir\") pod \"aws-ebs-csi-driver-node-g69ll\" (UID: \"a6b95002-74f1-475d-a64a-6495733cf142\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g69ll" Oct 30 16:20:45.729602 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.728719 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/749b2b57-f20e-4cdb-8d89-d2bac352f751-host\") pod \"tuned-c8x4k\" (UID: \"749b2b57-f20e-4cdb-8d89-d2bac352f751\") " pod="openshift-cluster-node-tuning-operator/tuned-c8x4k" Oct 30 16:20:45.729602 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.728736 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4c515c55-0717-4974-9f1a-68e0a25984cd-konnectivity-ca\") pod \"konnectivity-agent-sg9tg\" (UID: \"4c515c55-0717-4974-9f1a-68e0a25984cd\") " pod="kube-system/konnectivity-agent-sg9tg" Oct 30 16:20:45.729602 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.728779 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f2de1ce0-1c1e-4276-80a9-66239e13e352-ovn-node-metrics-cert\") pod \"ovnkube-node-2krz6\" (UID: \"f2de1ce0-1c1e-4276-80a9-66239e13e352\") " pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:45.729602 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.728795 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/33b888e2-d349-439b-9aa1-5b519e0e08f7-cnibin\") pod \"multus-7f2mt\" (UID: \"33b888e2-d349-439b-9aa1-5b519e0e08f7\") " pod="openshift-multus/multus-7f2mt" Oct 30 16:20:45.729602 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.728869 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f2de1ce0-1c1e-4276-80a9-66239e13e352-log-socket\") pod \"ovnkube-node-2krz6\" (UID: \"f2de1ce0-1c1e-4276-80a9-66239e13e352\") " pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:45.729602 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.728923 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4c515c55-0717-4974-9f1a-68e0a25984cd-konnectivity-ca\") pod \"konnectivity-agent-sg9tg\" (UID: \"4c515c55-0717-4974-9f1a-68e0a25984cd\") " pod="kube-system/konnectivity-agent-sg9tg" Oct 30 16:20:45.729602 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.728948 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/b2956847-8307-4f69-8921-bb0a0ac0aaf7-signing-cabundle\") pod \"service-ca-5fffc97bdf-8hqt5\" (UID: \"b2956847-8307-4f69-8921-bb0a0ac0aaf7\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8hqt5" Oct 30 16:20:45.729602 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.729009 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f2de1ce0-1c1e-4276-80a9-66239e13e352-ovn-node-metrics-cert\") pod \"ovnkube-node-2krz6\" (UID: \"f2de1ce0-1c1e-4276-80a9-66239e13e352\") " pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:45.729602 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.729056 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9fafe556-9653-49df-a851-100b51bb7f47-ca-trust-extracted\") pod \"image-registry-f8dddb4c-dcbgb\" (UID: \"9fafe556-9653-49df-a851-100b51bb7f47\") " pod="openshift-image-registry/image-registry-f8dddb4c-dcbgb" Oct 30 16:20:45.729602 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.729074 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/b2956847-8307-4f69-8921-bb0a0ac0aaf7-signing-cabundle\") pod \"service-ca-5fffc97bdf-8hqt5\" (UID: \"b2956847-8307-4f69-8921-bb0a0ac0aaf7\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8hqt5" Oct 30 16:20:45.729602 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.729088 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnsg7\" (UniqueName: \"kubernetes.io/projected/4f0f46f4-5cda-4d67-b6f3-47df06eb844c-kube-api-access-bnsg7\") pod \"console-operator-7cd56b8dbc-4wrf8\" (UID: \"4f0f46f4-5cda-4d67-b6f3-47df06eb844c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-4wrf8" Oct 30 16:20:45.729602 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.729114 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9fafe556-9653-49df-a851-100b51bb7f47-ca-trust-extracted\") pod \"image-registry-f8dddb4c-dcbgb\" (UID: \"9fafe556-9653-49df-a851-100b51bb7f47\") " pod="openshift-image-registry/image-registry-f8dddb4c-dcbgb" Oct 30 16:20:45.729602 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.729140 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f79e659b-79c9-43ee-8733-c685f60e87fb-registry-tls\") pod \"image-registry-65d65dd574-nbb2b\" (UID: \"f79e659b-79c9-43ee-8733-c685f60e87fb\") " pod="openshift-image-registry/image-registry-65d65dd574-nbb2b" Oct 30 16:20:45.729602 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.729155 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/749b2b57-f20e-4cdb-8d89-d2bac352f751-etc-sysconfig\") pod \"tuned-c8x4k\" (UID: \"749b2b57-f20e-4cdb-8d89-d2bac352f751\") " pod="openshift-cluster-node-tuning-operator/tuned-c8x4k" Oct 30 16:20:45.729602 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.729174 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d6db39f8-bf92-44e8-9144-51eb3b9dee8d-crio-socket\") pod \"insights-runtime-extractor-wxvph\" (UID: \"d6db39f8-bf92-44e8-9144-51eb3b9dee8d\") " pod="openshift-insights/insights-runtime-extractor-wxvph" Oct 30 16:20:45.729602 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.729282 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a0cd0943-8d62-41e4-b3e7-29eaabec7277-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-sfq4m\" (UID: \"a0cd0943-8d62-41e4-b3e7-29eaabec7277\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sfq4m" Oct 30 16:20:45.729602 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.729330 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhcx7\" (UniqueName: \"kubernetes.io/projected/b99d8a1f-000b-48ff-b4b0-7a62bc79c032-kube-api-access-bhcx7\") pod \"router-default-df85cd6ff-6pd8x\" (UID: \"b99d8a1f-000b-48ff-b4b0-7a62bc79c032\") " pod="openshift-ingress/router-default-df85cd6ff-6pd8x" Oct 30 16:20:45.729602 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.729353 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a6b95002-74f1-475d-a64a-6495733cf142-sys-fs\") pod \"aws-ebs-csi-driver-node-g69ll\" (UID: \"a6b95002-74f1-475d-a64a-6495733cf142\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g69ll" Oct 30 16:20:45.729602 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.729406 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9ea50669-2c08-403d-8d08-f9ffffa5b896-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-x9ddt\" (UID: \"9ea50669-2c08-403d-8d08-f9ffffa5b896\") " pod="openshift-multus/multus-additional-cni-plugins-x9ddt" Oct 30 16:20:45.729602 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.729456 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/029f4ecb-eb6b-4693-9e9e-ca61f554dd33-dbus\") pod \"global-pull-secret-syncer-ncfzj\" (UID: \"029f4ecb-eb6b-4693-9e9e-ca61f554dd33\") " pod="kube-system/global-pull-secret-syncer-ncfzj" Oct 30 16:20:45.729602 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.729485 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a0cd0943-8d62-41e4-b3e7-29eaabec7277-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-sfq4m\" (UID: \"a0cd0943-8d62-41e4-b3e7-29eaabec7277\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sfq4m" Oct 30 16:20:45.729602 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.729521 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9fafe556-9653-49df-a851-100b51bb7f47-trusted-ca\") pod \"image-registry-f8dddb4c-dcbgb\" (UID: \"9fafe556-9653-49df-a851-100b51bb7f47\") " pod="openshift-image-registry/image-registry-f8dddb4c-dcbgb" Oct 30 16:20:45.729602 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.729565 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/774953d5-23d3-4db1-aec9-3b78ef81017d-cert\") pod \"ingress-canary-l4mhk\" (UID: \"774953d5-23d3-4db1-aec9-3b78ef81017d\") " pod="openshift-ingress-canary/ingress-canary-l4mhk" Oct 30 16:20:45.729602 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.729604 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d10078a0-c21c-4f18-9c5a-4915fb7c8af3-host\") pod \"node-ca-2l5ht\" (UID: \"d10078a0-c21c-4f18-9c5a-4915fb7c8af3\") " pod="openshift-image-registry/node-ca-2l5ht" Oct 30 16:20:45.729602 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.729622 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f79e659b-79c9-43ee-8733-c685f60e87fb-bound-sa-token\") pod \"image-registry-65d65dd574-nbb2b\" (UID: \"f79e659b-79c9-43ee-8733-c685f60e87fb\") " pod="openshift-image-registry/image-registry-65d65dd574-nbb2b" Oct 30 16:20:45.731150 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.729641 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnv4j\" (UniqueName: \"kubernetes.io/projected/a6341cee-9ba8-41ce-bb75-29c9cb4e6d29-kube-api-access-gnv4j\") pod \"kas-connection-checker-685895c7c9-fhzbv\" (UID: \"a6341cee-9ba8-41ce-bb75-29c9cb4e6d29\") " pod="kube-system/kas-connection-checker-685895c7c9-fhzbv" Oct 30 16:20:45.731150 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.729652 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9ea50669-2c08-403d-8d08-f9ffffa5b896-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-x9ddt\" (UID: \"9ea50669-2c08-403d-8d08-f9ffffa5b896\") " pod="openshift-multus/multus-additional-cni-plugins-x9ddt" Oct 30 16:20:45.731150 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.729664 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9ea50669-2c08-403d-8d08-f9ffffa5b896-tuning-conf-dir\") pod \"multus-additional-cni-plugins-x9ddt\" (UID: \"9ea50669-2c08-403d-8d08-f9ffffa5b896\") " pod="openshift-multus/multus-additional-cni-plugins-x9ddt" Oct 30 16:20:45.731150 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.729719 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/749b2b57-f20e-4cdb-8d89-d2bac352f751-run\") pod \"tuned-c8x4k\" (UID: \"749b2b57-f20e-4cdb-8d89-d2bac352f751\") " pod="openshift-cluster-node-tuning-operator/tuned-c8x4k" Oct 30 16:20:45.731150 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.729755 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a0cd0943-8d62-41e4-b3e7-29eaabec7277-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-sfq4m\" (UID: \"a0cd0943-8d62-41e4-b3e7-29eaabec7277\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sfq4m" Oct 30 16:20:45.731150 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.729769 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/209633e5-92f7-4a7f-9d68-3d48473ed2bf-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jrdmg\" (UID: \"209633e5-92f7-4a7f-9d68-3d48473ed2bf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jrdmg" Oct 30 16:20:45.731150 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.729799 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/774953d5-23d3-4db1-aec9-3b78ef81017d-cert\") pod \"ingress-canary-l4mhk\" (UID: \"774953d5-23d3-4db1-aec9-3b78ef81017d\") " pod="openshift-ingress-canary/ingress-canary-l4mhk" Oct 30 16:20:45.731150 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.729834 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9fafe556-9653-49df-a851-100b51bb7f47-trusted-ca\") pod \"image-registry-f8dddb4c-dcbgb\" (UID: \"9fafe556-9653-49df-a851-100b51bb7f47\") " pod="openshift-image-registry/image-registry-f8dddb4c-dcbgb" Oct 30 16:20:45.731150 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.729800 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/b2956847-8307-4f69-8921-bb0a0ac0aaf7-signing-key\") pod \"service-ca-5fffc97bdf-8hqt5\" (UID: \"b2956847-8307-4f69-8921-bb0a0ac0aaf7\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8hqt5" Oct 30 16:20:45.731150 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.729933 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/154b2258-1b09-444b-ad80-2b194195e2cb-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-p4rk6\" (UID: \"154b2258-1b09-444b-ad80-2b194195e2cb\") " pod="openshift-insights/insights-operator-7c4986d449-p4rk6" Oct 30 16:20:45.731150 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.729980 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/b2956847-8307-4f69-8921-bb0a0ac0aaf7-signing-key\") pod \"service-ca-5fffc97bdf-8hqt5\" (UID: \"b2956847-8307-4f69-8921-bb0a0ac0aaf7\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8hqt5" Oct 30 16:20:45.731150 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.729945 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/209633e5-92f7-4a7f-9d68-3d48473ed2bf-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jrdmg\" (UID: \"209633e5-92f7-4a7f-9d68-3d48473ed2bf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jrdmg" Oct 30 16:20:45.731150 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.730002 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a6b95002-74f1-475d-a64a-6495733cf142-kubelet-dir\") pod \"aws-ebs-csi-driver-node-g69ll\" (UID: \"a6b95002-74f1-475d-a64a-6495733cf142\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g69ll" Oct 30 16:20:45.731150 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.730025 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/749b2b57-f20e-4cdb-8d89-d2bac352f751-lib-modules\") pod \"tuned-c8x4k\" (UID: \"749b2b57-f20e-4cdb-8d89-d2bac352f751\") " pod="openshift-cluster-node-tuning-operator/tuned-c8x4k" Oct 30 16:20:45.731150 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.730086 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9ec1217e-8623-42a6-a052-adee153936a2-hosts-file\") pod \"node-resolver-lghnm\" (UID: \"9ec1217e-8623-42a6-a052-adee153936a2\") " pod="openshift-dns/node-resolver-lghnm" Oct 30 16:20:45.731150 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.730144 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-httvd\" (UniqueName: \"kubernetes.io/projected/9fafe556-9653-49df-a851-100b51bb7f47-kube-api-access-httvd\") pod \"image-registry-f8dddb4c-dcbgb\" (UID: \"9fafe556-9653-49df-a851-100b51bb7f47\") " pod="openshift-image-registry/image-registry-f8dddb4c-dcbgb" Oct 30 16:20:45.731150 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.730167 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5sxw\" (UniqueName: \"kubernetes.io/projected/33b888e2-d349-439b-9aa1-5b519e0e08f7-kube-api-access-v5sxw\") pod \"multus-7f2mt\" (UID: \"33b888e2-d349-439b-9aa1-5b519e0e08f7\") " pod="openshift-multus/multus-7f2mt" Oct 30 16:20:45.731150 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.730183 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3e90fa4d-66a8-4689-8f41-ca090d6697a9-metrics-tls\") pod \"dns-default-qg6t6\" (UID: \"3e90fa4d-66a8-4689-8f41-ca090d6697a9\") " pod="openshift-dns/dns-default-qg6t6" Oct 30 16:20:45.731150 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.730198 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/749b2b57-f20e-4cdb-8d89-d2bac352f751-etc-sysctl-d\") pod \"tuned-c8x4k\" (UID: \"749b2b57-f20e-4cdb-8d89-d2bac352f751\") " pod="openshift-cluster-node-tuning-operator/tuned-c8x4k" Oct 30 16:20:45.731150 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.730212 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/154b2258-1b09-444b-ad80-2b194195e2cb-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-p4rk6\" (UID: \"154b2258-1b09-444b-ad80-2b194195e2cb\") " pod="openshift-insights/insights-operator-7c4986d449-p4rk6" Oct 30 16:20:45.731150 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.730221 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/749b2b57-f20e-4cdb-8d89-d2bac352f751-etc-tuned\") pod \"tuned-c8x4k\" (UID: \"749b2b57-f20e-4cdb-8d89-d2bac352f751\") " pod="openshift-cluster-node-tuning-operator/tuned-c8x4k" Oct 30 16:20:45.731150 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.730259 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f2de1ce0-1c1e-4276-80a9-66239e13e352-host-kubelet\") pod \"ovnkube-node-2krz6\" (UID: \"f2de1ce0-1c1e-4276-80a9-66239e13e352\") " pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:45.731150 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.730303 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/33b888e2-d349-439b-9aa1-5b519e0e08f7-host-var-lib-kubelet\") pod \"multus-7f2mt\" (UID: \"33b888e2-d349-439b-9aa1-5b519e0e08f7\") " pod="openshift-multus/multus-7f2mt" Oct 30 16:20:45.731150 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.730331 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/749b2b57-f20e-4cdb-8d89-d2bac352f751-etc-tuned\") pod \"tuned-c8x4k\" (UID: \"749b2b57-f20e-4cdb-8d89-d2bac352f751\") " pod="openshift-cluster-node-tuning-operator/tuned-c8x4k" Oct 30 16:20:45.731150 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.730344 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9ea50669-2c08-403d-8d08-f9ffffa5b896-system-cni-dir\") pod \"multus-additional-cni-plugins-x9ddt\" (UID: \"9ea50669-2c08-403d-8d08-f9ffffa5b896\") " pod="openshift-multus/multus-additional-cni-plugins-x9ddt" Oct 30 16:20:45.731150 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.730379 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f2de1ce0-1c1e-4276-80a9-66239e13e352-host-slash\") pod \"ovnkube-node-2krz6\" (UID: \"f2de1ce0-1c1e-4276-80a9-66239e13e352\") " pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:45.731150 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.730410 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f2de1ce0-1c1e-4276-80a9-66239e13e352-ovnkube-config\") pod \"ovnkube-node-2krz6\" (UID: \"f2de1ce0-1c1e-4276-80a9-66239e13e352\") " pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:45.731150 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.730436 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/06332900-7903-4275-8e33-de06de7f6580-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-cswcz\" (UID: \"06332900-7903-4275-8e33-de06de7f6580\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-cswcz" Oct 30 16:20:45.731150 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.730530 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b99d8a1f-000b-48ff-b4b0-7a62bc79c032-service-ca-bundle\") pod \"router-default-df85cd6ff-6pd8x\" (UID: \"b99d8a1f-000b-48ff-b4b0-7a62bc79c032\") " pod="openshift-ingress/router-default-df85cd6ff-6pd8x" Oct 30 16:20:45.731150 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.730626 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b99d8a1f-000b-48ff-b4b0-7a62bc79c032-metrics-certs\") pod \"router-default-df85cd6ff-6pd8x\" (UID: \"b99d8a1f-000b-48ff-b4b0-7a62bc79c032\") " pod="openshift-ingress/router-default-df85cd6ff-6pd8x" Oct 30 16:20:45.731150 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.730631 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f2de1ce0-1c1e-4276-80a9-66239e13e352-ovnkube-config\") pod \"ovnkube-node-2krz6\" (UID: \"f2de1ce0-1c1e-4276-80a9-66239e13e352\") " pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:45.731150 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.730670 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/154b2258-1b09-444b-ad80-2b194195e2cb-snapshots\") pod \"insights-operator-7c4986d449-p4rk6\" (UID: \"154b2258-1b09-444b-ad80-2b194195e2cb\") " pod="openshift-insights/insights-operator-7c4986d449-p4rk6" Oct 30 16:20:45.731150 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.730681 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/06332900-7903-4275-8e33-de06de7f6580-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-cswcz\" (UID: \"06332900-7903-4275-8e33-de06de7f6580\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-cswcz" Oct 30 16:20:45.731150 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.730717 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kgld\" (UniqueName: \"kubernetes.io/projected/154b2258-1b09-444b-ad80-2b194195e2cb-kube-api-access-4kgld\") pod \"insights-operator-7c4986d449-p4rk6\" (UID: \"154b2258-1b09-444b-ad80-2b194195e2cb\") " pod="openshift-insights/insights-operator-7c4986d449-p4rk6" Oct 30 16:20:45.731150 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.730759 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qh2h\" (UniqueName: \"kubernetes.io/projected/f79e659b-79c9-43ee-8733-c685f60e87fb-kube-api-access-5qh2h\") pod \"image-registry-65d65dd574-nbb2b\" (UID: \"f79e659b-79c9-43ee-8733-c685f60e87fb\") " pod="openshift-image-registry/image-registry-65d65dd574-nbb2b" Oct 30 16:20:45.731150 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.730791 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/154b2258-1b09-444b-ad80-2b194195e2cb-snapshots\") pod \"insights-operator-7c4986d449-p4rk6\" (UID: \"154b2258-1b09-444b-ad80-2b194195e2cb\") " pod="openshift-insights/insights-operator-7c4986d449-p4rk6" Oct 30 16:20:45.731150 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.730830 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/33b888e2-d349-439b-9aa1-5b519e0e08f7-host-var-lib-cni-multus\") pod \"multus-7f2mt\" (UID: \"33b888e2-d349-439b-9aa1-5b519e0e08f7\") " pod="openshift-multus/multus-7f2mt" Oct 30 16:20:45.731150 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.730865 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9ea50669-2c08-403d-8d08-f9ffffa5b896-cnibin\") pod \"multus-additional-cni-plugins-x9ddt\" (UID: \"9ea50669-2c08-403d-8d08-f9ffffa5b896\") " pod="openshift-multus/multus-additional-cni-plugins-x9ddt" Oct 30 16:20:45.731150 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.730911 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9fafe556-9653-49df-a851-100b51bb7f47-bound-sa-token\") pod \"image-registry-f8dddb4c-dcbgb\" (UID: \"9fafe556-9653-49df-a851-100b51bb7f47\") " pod="openshift-image-registry/image-registry-f8dddb4c-dcbgb" Oct 30 16:20:45.731150 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.730936 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ef87558b-2ba2-41ce-9a0b-9e4b18663a6c-config\") pod \"service-ca-operator-544d749c8c-4nn7z\" (UID: \"ef87558b-2ba2-41ce-9a0b-9e4b18663a6c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-4nn7z" Oct 30 16:20:45.731150 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.731090 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ef87558b-2ba2-41ce-9a0b-9e4b18663a6c-config\") pod \"service-ca-operator-544d749c8c-4nn7z\" (UID: \"ef87558b-2ba2-41ce-9a0b-9e4b18663a6c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-4nn7z" Oct 30 16:20:45.752262 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.752241 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzp5d\" (UniqueName: \"kubernetes.io/projected/25c51397-40b9-432c-949d-12ba9f36532a-kube-api-access-qzp5d\") pod \"iptables-alerter-d9kr8\" (UID: \"25c51397-40b9-432c-949d-12ba9f36532a\") " pod="openshift-network-operator/iptables-alerter-d9kr8" Oct 30 16:20:45.752376 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.752348 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttr7v\" (UniqueName: \"kubernetes.io/projected/341d9139-495e-4ee2-80c3-a20370039948-kube-api-access-ttr7v\") pod \"migrator-66df5d496b-c87fl\" (UID: \"341d9139-495e-4ee2-80c3-a20370039948\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-c87fl" Oct 30 16:20:45.752816 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.752779 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrppt\" (UniqueName: \"kubernetes.io/projected/9ec1217e-8623-42a6-a052-adee153936a2-kube-api-access-nrppt\") pod \"node-resolver-lghnm\" (UID: \"9ec1217e-8623-42a6-a052-adee153936a2\") " pod="openshift-dns/node-resolver-lghnm" Oct 30 16:20:45.752933 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.752900 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5tsm\" (UniqueName: \"kubernetes.io/projected/6f3d36d6-5098-43cc-8f8a-bd83d5dea5fc-kube-api-access-v5tsm\") pod \"network-metrics-daemon-xj6cp\" (UID: \"6f3d36d6-5098-43cc-8f8a-bd83d5dea5fc\") " pod="openshift-multus/network-metrics-daemon-xj6cp" Oct 30 16:20:45.752933 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.752917 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4hr8\" (UniqueName: \"kubernetes.io/projected/90d73ca9-5d88-4467-99fa-d4e1a67f3828-kube-api-access-b4hr8\") pod \"kas-connection-checker-685895c7c9-cpxbb\" (UID: \"90d73ca9-5d88-4467-99fa-d4e1a67f3828\") " pod="kube-system/kas-connection-checker-685895c7c9-cpxbb" Oct 30 16:20:45.753092 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.753074 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4d6v2\" (UniqueName: \"kubernetes.io/projected/06332900-7903-4275-8e33-de06de7f6580-kube-api-access-4d6v2\") pod \"cluster-samples-operator-6ddb6c8b94-cswcz\" (UID: \"06332900-7903-4275-8e33-de06de7f6580\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-cswcz" Oct 30 16:20:45.753154 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.753123 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cw6q9\" (UniqueName: \"kubernetes.io/projected/b2956847-8307-4f69-8921-bb0a0ac0aaf7-kube-api-access-cw6q9\") pod \"service-ca-5fffc97bdf-8hqt5\" (UID: \"b2956847-8307-4f69-8921-bb0a0ac0aaf7\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8hqt5" Oct 30 16:20:45.753280 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.753260 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mc2zh\" (UniqueName: \"kubernetes.io/projected/c471cf39-07e7-4f4d-87d5-0c6803fd54a0-kube-api-access-mc2zh\") pod \"periodic-gathering-cwrtz-glm8n\" (UID: \"c471cf39-07e7-4f4d-87d5-0c6803fd54a0\") " pod="openshift-insights/periodic-gathering-cwrtz-glm8n" Oct 30 16:20:45.753382 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.753369 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7d4j4\" (UniqueName: \"kubernetes.io/projected/f2de1ce0-1c1e-4276-80a9-66239e13e352-kube-api-access-7d4j4\") pod \"ovnkube-node-2krz6\" (UID: \"f2de1ce0-1c1e-4276-80a9-66239e13e352\") " pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:45.753984 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.753969 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t54w7\" (UniqueName: \"kubernetes.io/projected/a0cd0943-8d62-41e4-b3e7-29eaabec7277-kube-api-access-t54w7\") pod \"cluster-monitoring-operator-884576c57-sfq4m\" (UID: \"a0cd0943-8d62-41e4-b3e7-29eaabec7277\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sfq4m" Oct 30 16:20:45.757473 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.757451 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlqc8\" (UniqueName: \"kubernetes.io/projected/ef87558b-2ba2-41ce-9a0b-9e4b18663a6c-kube-api-access-qlqc8\") pod \"service-ca-operator-544d749c8c-4nn7z\" (UID: \"ef87558b-2ba2-41ce-9a0b-9e4b18663a6c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-4nn7z" Oct 30 16:20:45.757897 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.757875 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwfbm\" (UniqueName: \"kubernetes.io/projected/d6db39f8-bf92-44e8-9144-51eb3b9dee8d-kube-api-access-cwfbm\") pod \"insights-runtime-extractor-wxvph\" (UID: \"d6db39f8-bf92-44e8-9144-51eb3b9dee8d\") " pod="openshift-insights/insights-runtime-extractor-wxvph" Oct 30 16:20:45.758022 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.758005 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-849kr\" (UniqueName: \"kubernetes.io/projected/4d9a19d6-94d4-4490-8f93-89ca745b768f-kube-api-access-849kr\") pod \"volume-data-source-validator-58c9f5f46d-5fw2s\" (UID: \"4d9a19d6-94d4-4490-8f93-89ca745b768f\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-5fw2s" Oct 30 16:20:45.758100 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.758065 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qc9ck\" (UniqueName: \"kubernetes.io/projected/774953d5-23d3-4db1-aec9-3b78ef81017d-kube-api-access-qc9ck\") pod \"ingress-canary-l4mhk\" (UID: \"774953d5-23d3-4db1-aec9-3b78ef81017d\") " pod="openshift-ingress-canary/ingress-canary-l4mhk" Oct 30 16:20:45.758320 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.758302 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfgdd\" (UniqueName: \"kubernetes.io/projected/749b2b57-f20e-4cdb-8d89-d2bac352f751-kube-api-access-dfgdd\") pod \"tuned-c8x4k\" (UID: \"749b2b57-f20e-4cdb-8d89-d2bac352f751\") " pod="openshift-cluster-node-tuning-operator/tuned-c8x4k" Oct 30 16:20:45.758369 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.758324 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpgwx\" (UniqueName: \"kubernetes.io/projected/9ea50669-2c08-403d-8d08-f9ffffa5b896-kube-api-access-zpgwx\") pod \"multus-additional-cni-plugins-x9ddt\" (UID: \"9ea50669-2c08-403d-8d08-f9ffffa5b896\") " pod="openshift-multus/multus-additional-cni-plugins-x9ddt" Oct 30 16:20:45.758529 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.758509 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mkkh\" (UniqueName: \"kubernetes.io/projected/d719cbcb-9874-4b85-8fe7-e1bf53511235-kube-api-access-2mkkh\") pod \"kas-connection-checker-685895c7c9-4dfsk\" (UID: \"d719cbcb-9874-4b85-8fe7-e1bf53511235\") " pod="kube-system/kas-connection-checker-685895c7c9-4dfsk" Oct 30 16:20:45.758735 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.758719 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnv4j\" (UniqueName: \"kubernetes.io/projected/a6341cee-9ba8-41ce-bb75-29c9cb4e6d29-kube-api-access-gnv4j\") pod \"kas-connection-checker-685895c7c9-fhzbv\" (UID: \"a6341cee-9ba8-41ce-bb75-29c9cb4e6d29\") " pod="kube-system/kas-connection-checker-685895c7c9-fhzbv" Oct 30 16:20:45.758795 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.758781 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljv94\" (UniqueName: \"kubernetes.io/projected/2d1ed7d8-37cc-4652-b225-3a715b755e48-kube-api-access-ljv94\") pod \"network-check-target-b4ljl\" (UID: \"2d1ed7d8-37cc-4652-b225-3a715b755e48\") " pod="openshift-network-diagnostics/network-check-target-b4ljl" Oct 30 16:20:45.759017 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.759002 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-httvd\" (UniqueName: \"kubernetes.io/projected/9fafe556-9653-49df-a851-100b51bb7f47-kube-api-access-httvd\") pod \"image-registry-f8dddb4c-dcbgb\" (UID: \"9fafe556-9653-49df-a851-100b51bb7f47\") " pod="openshift-image-registry/image-registry-f8dddb4c-dcbgb" Oct 30 16:20:45.759142 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.759129 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f79e659b-79c9-43ee-8733-c685f60e87fb-bound-sa-token\") pod \"image-registry-65d65dd574-nbb2b\" (UID: \"f79e659b-79c9-43ee-8733-c685f60e87fb\") " pod="openshift-image-registry/image-registry-65d65dd574-nbb2b" Oct 30 16:20:45.763345 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.763311 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhcx7\" (UniqueName: \"kubernetes.io/projected/b99d8a1f-000b-48ff-b4b0-7a62bc79c032-kube-api-access-bhcx7\") pod \"router-default-df85cd6ff-6pd8x\" (UID: \"b99d8a1f-000b-48ff-b4b0-7a62bc79c032\") " pod="openshift-ingress/router-default-df85cd6ff-6pd8x" Oct 30 16:20:45.763345 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.763326 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qh2h\" (UniqueName: \"kubernetes.io/projected/f79e659b-79c9-43ee-8733-c685f60e87fb-kube-api-access-5qh2h\") pod \"image-registry-65d65dd574-nbb2b\" (UID: \"f79e659b-79c9-43ee-8733-c685f60e87fb\") " pod="openshift-image-registry/image-registry-65d65dd574-nbb2b" Oct 30 16:20:45.763661 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.763314 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7dg7\" (UniqueName: \"kubernetes.io/projected/b54d0e33-3943-4e1c-9889-dc65864b4efb-kube-api-access-v7dg7\") pod \"migrator-66df5d496b-vgrwl\" (UID: \"b54d0e33-3943-4e1c-9889-dc65864b4efb\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vgrwl" Oct 30 16:20:45.767726 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.763901 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kgld\" (UniqueName: \"kubernetes.io/projected/154b2258-1b09-444b-ad80-2b194195e2cb-kube-api-access-4kgld\") pod \"insights-operator-7c4986d449-p4rk6\" (UID: \"154b2258-1b09-444b-ad80-2b194195e2cb\") " pod="openshift-insights/insights-operator-7c4986d449-p4rk6" Oct 30 16:20:45.767726 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.763906 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25jwd\" (UniqueName: \"kubernetes.io/projected/d10078a0-c21c-4f18-9c5a-4915fb7c8af3-kube-api-access-25jwd\") pod \"node-ca-2l5ht\" (UID: \"d10078a0-c21c-4f18-9c5a-4915fb7c8af3\") " pod="openshift-image-registry/node-ca-2l5ht" Oct 30 16:20:45.767726 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.764025 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnsg7\" (UniqueName: \"kubernetes.io/projected/4f0f46f4-5cda-4d67-b6f3-47df06eb844c-kube-api-access-bnsg7\") pod \"console-operator-7cd56b8dbc-4wrf8\" (UID: \"4f0f46f4-5cda-4d67-b6f3-47df06eb844c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-4wrf8" Oct 30 16:20:45.767726 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.764283 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9fafe556-9653-49df-a851-100b51bb7f47-bound-sa-token\") pod \"image-registry-f8dddb4c-dcbgb\" (UID: \"9fafe556-9653-49df-a851-100b51bb7f47\") " pod="openshift-image-registry/image-registry-f8dddb4c-dcbgb" Oct 30 16:20:45.767726 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.764352 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjp8r\" (UniqueName: \"kubernetes.io/projected/3e90fa4d-66a8-4689-8f41-ca090d6697a9-kube-api-access-pjp8r\") pod \"dns-default-qg6t6\" (UID: \"3e90fa4d-66a8-4689-8f41-ca090d6697a9\") " pod="openshift-dns/dns-default-qg6t6" Oct 30 16:20:45.767726 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.764371 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zlg7\" (UniqueName: \"kubernetes.io/projected/a6b95002-74f1-475d-a64a-6495733cf142-kube-api-access-8zlg7\") pod \"aws-ebs-csi-driver-node-g69ll\" (UID: \"a6b95002-74f1-475d-a64a-6495733cf142\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g69ll" Oct 30 16:20:45.767726 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.764511 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z65wk\" (UniqueName: \"kubernetes.io/projected/209633e5-92f7-4a7f-9d68-3d48473ed2bf-kube-api-access-z65wk\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jrdmg\" (UID: \"209633e5-92f7-4a7f-9d68-3d48473ed2bf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jrdmg" Oct 30 16:20:45.767726 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.764822 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5sxw\" (UniqueName: \"kubernetes.io/projected/33b888e2-d349-439b-9aa1-5b519e0e08f7-kube-api-access-v5sxw\") pod \"multus-7f2mt\" (UID: \"33b888e2-d349-439b-9aa1-5b519e0e08f7\") " pod="openshift-multus/multus-7f2mt" Oct 30 16:20:45.767726 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.764988 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-478bs\" (UniqueName: \"kubernetes.io/projected/0d304765-4b62-4cba-bad2-452771d9b201-kube-api-access-478bs\") pod \"network-check-source-6f78d59f99-nwvwc\" (UID: \"0d304765-4b62-4cba-bad2-452771d9b201\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-nwvwc" Oct 30 16:20:45.832130 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.832075 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d6db39f8-bf92-44e8-9144-51eb3b9dee8d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wxvph\" (UID: \"d6db39f8-bf92-44e8-9144-51eb3b9dee8d\") " pod="openshift-insights/insights-runtime-extractor-wxvph" Oct 30 16:20:45.832130 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.832100 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/06332900-7903-4275-8e33-de06de7f6580-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-cswcz\" (UID: \"06332900-7903-4275-8e33-de06de7f6580\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-cswcz" Oct 30 16:20:45.832130 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.832118 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/33b888e2-d349-439b-9aa1-5b519e0e08f7-multus-cni-dir\") pod \"multus-7f2mt\" (UID: \"33b888e2-d349-439b-9aa1-5b519e0e08f7\") " pod="openshift-multus/multus-7f2mt" Oct 30 16:20:45.832318 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.832134 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/33b888e2-d349-439b-9aa1-5b519e0e08f7-host-run-k8s-cni-cncf-io\") pod \"multus-7f2mt\" (UID: \"33b888e2-d349-439b-9aa1-5b519e0e08f7\") " pod="openshift-multus/multus-7f2mt" Oct 30 16:20:45.832318 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.832157 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f2de1ce0-1c1e-4276-80a9-66239e13e352-host-run-ovn-kubernetes\") pod \"ovnkube-node-2krz6\" (UID: \"f2de1ce0-1c1e-4276-80a9-66239e13e352\") " pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:45.832318 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.832181 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/33b888e2-d349-439b-9aa1-5b519e0e08f7-hostroot\") pod \"multus-7f2mt\" (UID: \"33b888e2-d349-439b-9aa1-5b519e0e08f7\") " pod="openshift-multus/multus-7f2mt" Oct 30 16:20:45.832318 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.832202 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/749b2b57-f20e-4cdb-8d89-d2bac352f751-etc-systemd\") pod \"tuned-c8x4k\" (UID: \"749b2b57-f20e-4cdb-8d89-d2bac352f751\") " pod="openshift-cluster-node-tuning-operator/tuned-c8x4k" Oct 30 16:20:45.832318 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.832227 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/33b888e2-d349-439b-9aa1-5b519e0e08f7-host-run-k8s-cni-cncf-io\") pod \"multus-7f2mt\" (UID: \"33b888e2-d349-439b-9aa1-5b519e0e08f7\") " pod="openshift-multus/multus-7f2mt" Oct 30 16:20:45.832318 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.832232 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1042378b-9f46-4e19-88e4-2628e532e8c3-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-992nb\" (UID: \"1042378b-9f46-4e19-88e4-2628e532e8c3\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-992nb" Oct 30 16:20:45.832318 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.832251 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/33b888e2-d349-439b-9aa1-5b519e0e08f7-multus-cni-dir\") pod \"multus-7f2mt\" (UID: \"33b888e2-d349-439b-9aa1-5b519e0e08f7\") " pod="openshift-multus/multus-7f2mt" Oct 30 16:20:45.832318 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:20:45.832253 5052 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 16:20:45.832318 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.832276 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a6b95002-74f1-475d-a64a-6495733cf142-etc-selinux\") pod \"aws-ebs-csi-driver-node-g69ll\" (UID: \"a6b95002-74f1-475d-a64a-6495733cf142\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g69ll" Oct 30 16:20:45.832318 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.832278 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/33b888e2-d349-439b-9aa1-5b519e0e08f7-hostroot\") pod \"multus-7f2mt\" (UID: \"33b888e2-d349-439b-9aa1-5b519e0e08f7\") " pod="openshift-multus/multus-7f2mt" Oct 30 16:20:45.832318 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.832238 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f2de1ce0-1c1e-4276-80a9-66239e13e352-host-run-ovn-kubernetes\") pod \"ovnkube-node-2krz6\" (UID: \"f2de1ce0-1c1e-4276-80a9-66239e13e352\") " pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:45.832318 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.832308 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/25c51397-40b9-432c-949d-12ba9f36532a-host-slash\") pod \"iptables-alerter-d9kr8\" (UID: \"25c51397-40b9-432c-949d-12ba9f36532a\") " pod="openshift-network-operator/iptables-alerter-d9kr8" Oct 30 16:20:45.832318 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.832307 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/749b2b57-f20e-4cdb-8d89-d2bac352f751-etc-systemd\") pod \"tuned-c8x4k\" (UID: \"749b2b57-f20e-4cdb-8d89-d2bac352f751\") " pod="openshift-cluster-node-tuning-operator/tuned-c8x4k" Oct 30 16:20:45.832951 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.832348 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a6b95002-74f1-475d-a64a-6495733cf142-registration-dir\") pod \"aws-ebs-csi-driver-node-g69ll\" (UID: \"a6b95002-74f1-475d-a64a-6495733cf142\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g69ll" Oct 30 16:20:45.832951 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.832350 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/25c51397-40b9-432c-949d-12ba9f36532a-host-slash\") pod \"iptables-alerter-d9kr8\" (UID: \"25c51397-40b9-432c-949d-12ba9f36532a\") " pod="openshift-network-operator/iptables-alerter-d9kr8" Oct 30 16:20:45.832951 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.832354 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a6b95002-74f1-475d-a64a-6495733cf142-etc-selinux\") pod \"aws-ebs-csi-driver-node-g69ll\" (UID: \"a6b95002-74f1-475d-a64a-6495733cf142\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g69ll" Oct 30 16:20:45.832951 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:20:45.832391 5052 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6db39f8-bf92-44e8-9144-51eb3b9dee8d-insights-runtime-extractor-tls podName:d6db39f8-bf92-44e8-9144-51eb3b9dee8d nodeName:}" failed. No retries permitted until 2026-10-30 16:20:46.332355684 +0000 UTC m=+1.859690912 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d6db39f8-bf92-44e8-9144-51eb3b9dee8d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wxvph" (UID: "d6db39f8-bf92-44e8-9144-51eb3b9dee8d") : secret "insights-runtime-extractor-tls" not found Oct 30 16:20:45.832951 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.832391 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a6b95002-74f1-475d-a64a-6495733cf142-registration-dir\") pod \"aws-ebs-csi-driver-node-g69ll\" (UID: \"a6b95002-74f1-475d-a64a-6495733cf142\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g69ll" Oct 30 16:20:45.832951 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.832435 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/749b2b57-f20e-4cdb-8d89-d2bac352f751-var-lib-kubelet\") pod \"tuned-c8x4k\" (UID: \"749b2b57-f20e-4cdb-8d89-d2bac352f751\") " pod="openshift-cluster-node-tuning-operator/tuned-c8x4k" Oct 30 16:20:45.832951 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.832447 5052 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 16:20:45.832951 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.832473 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f2de1ce0-1c1e-4276-80a9-66239e13e352-etc-openvswitch\") pod \"ovnkube-node-2krz6\" (UID: \"f2de1ce0-1c1e-4276-80a9-66239e13e352\") " pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:45.832951 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.832490 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/749b2b57-f20e-4cdb-8d89-d2bac352f751-etc-sysctl-conf\") pod \"tuned-c8x4k\" (UID: \"749b2b57-f20e-4cdb-8d89-d2bac352f751\") " pod="openshift-cluster-node-tuning-operator/tuned-c8x4k" Oct 30 16:20:45.832951 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.832501 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/749b2b57-f20e-4cdb-8d89-d2bac352f751-var-lib-kubelet\") pod \"tuned-c8x4k\" (UID: \"749b2b57-f20e-4cdb-8d89-d2bac352f751\") " pod="openshift-cluster-node-tuning-operator/tuned-c8x4k" Oct 30 16:20:45.832951 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.832514 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/33b888e2-d349-439b-9aa1-5b519e0e08f7-host-run-netns\") pod \"multus-7f2mt\" (UID: \"33b888e2-d349-439b-9aa1-5b519e0e08f7\") " pod="openshift-multus/multus-7f2mt" Oct 30 16:20:45.832951 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.832529 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f2de1ce0-1c1e-4276-80a9-66239e13e352-etc-openvswitch\") pod \"ovnkube-node-2krz6\" (UID: \"f2de1ce0-1c1e-4276-80a9-66239e13e352\") " pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:45.832951 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.832553 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f2de1ce0-1c1e-4276-80a9-66239e13e352-var-lib-openvswitch\") pod \"ovnkube-node-2krz6\" (UID: \"f2de1ce0-1c1e-4276-80a9-66239e13e352\") " pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:45.832951 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.832574 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/33b888e2-d349-439b-9aa1-5b519e0e08f7-host-run-netns\") pod \"multus-7f2mt\" (UID: \"33b888e2-d349-439b-9aa1-5b519e0e08f7\") " pod="openshift-multus/multus-7f2mt" Oct 30 16:20:45.832951 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.832598 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f3d36d6-5098-43cc-8f8a-bd83d5dea5fc-metrics-certs\") pod \"network-metrics-daemon-xj6cp\" (UID: \"6f3d36d6-5098-43cc-8f8a-bd83d5dea5fc\") " pod="openshift-multus/network-metrics-daemon-xj6cp" Oct 30 16:20:45.832951 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.832626 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/749b2b57-f20e-4cdb-8d89-d2bac352f751-etc-modprobe-d\") pod \"tuned-c8x4k\" (UID: \"749b2b57-f20e-4cdb-8d89-d2bac352f751\") " pod="openshift-cluster-node-tuning-operator/tuned-c8x4k" Oct 30 16:20:45.832951 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.832625 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f2de1ce0-1c1e-4276-80a9-66239e13e352-var-lib-openvswitch\") pod \"ovnkube-node-2krz6\" (UID: \"f2de1ce0-1c1e-4276-80a9-66239e13e352\") " pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:45.832951 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.832650 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/33b888e2-d349-439b-9aa1-5b519e0e08f7-multus-conf-dir\") pod \"multus-7f2mt\" (UID: \"33b888e2-d349-439b-9aa1-5b519e0e08f7\") " pod="openshift-multus/multus-7f2mt" Oct 30 16:20:45.832951 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:20:45.832668 5052 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 16:20:45.832951 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.832687 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/33b888e2-d349-439b-9aa1-5b519e0e08f7-host-run-multus-certs\") pod \"multus-7f2mt\" (UID: \"33b888e2-d349-439b-9aa1-5b519e0e08f7\") " pod="openshift-multus/multus-7f2mt" Oct 30 16:20:45.832951 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.832709 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f2de1ce0-1c1e-4276-80a9-66239e13e352-host-run-netns\") pod \"ovnkube-node-2krz6\" (UID: \"f2de1ce0-1c1e-4276-80a9-66239e13e352\") " pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:45.832951 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.832733 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f2de1ce0-1c1e-4276-80a9-66239e13e352-run-systemd\") pod \"ovnkube-node-2krz6\" (UID: \"f2de1ce0-1c1e-4276-80a9-66239e13e352\") " pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:45.832951 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:20:45.832755 5052 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f3d36d6-5098-43cc-8f8a-bd83d5dea5fc-metrics-certs podName:6f3d36d6-5098-43cc-8f8a-bd83d5dea5fc nodeName:}" failed. No retries permitted until 2026-10-30 16:20:46.332739266 +0000 UTC m=+1.860074499 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6f3d36d6-5098-43cc-8f8a-bd83d5dea5fc-metrics-certs") pod "network-metrics-daemon-xj6cp" (UID: "6f3d36d6-5098-43cc-8f8a-bd83d5dea5fc") : secret "metrics-daemon-secret" not found Oct 30 16:20:45.832951 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.832762 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/749b2b57-f20e-4cdb-8d89-d2bac352f751-etc-modprobe-d\") pod \"tuned-c8x4k\" (UID: \"749b2b57-f20e-4cdb-8d89-d2bac352f751\") " pod="openshift-cluster-node-tuning-operator/tuned-c8x4k" Oct 30 16:20:45.832951 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.832666 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/749b2b57-f20e-4cdb-8d89-d2bac352f751-etc-sysctl-conf\") pod \"tuned-c8x4k\" (UID: \"749b2b57-f20e-4cdb-8d89-d2bac352f751\") " pod="openshift-cluster-node-tuning-operator/tuned-c8x4k" Oct 30 16:20:45.832951 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.832782 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/33b888e2-d349-439b-9aa1-5b519e0e08f7-etc-kubernetes\") pod \"multus-7f2mt\" (UID: \"33b888e2-d349-439b-9aa1-5b519e0e08f7\") " pod="openshift-multus/multus-7f2mt" Oct 30 16:20:45.832951 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.832813 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/029f4ecb-eb6b-4693-9e9e-ca61f554dd33-kubelet-config\") pod \"global-pull-secret-syncer-ncfzj\" (UID: \"029f4ecb-eb6b-4693-9e9e-ca61f554dd33\") " pod="kube-system/global-pull-secret-syncer-ncfzj" Oct 30 16:20:45.832951 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.832781 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/33b888e2-d349-439b-9aa1-5b519e0e08f7-host-run-multus-certs\") pod \"multus-7f2mt\" (UID: \"33b888e2-d349-439b-9aa1-5b519e0e08f7\") " pod="openshift-multus/multus-7f2mt" Oct 30 16:20:45.832951 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.832825 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/33b888e2-d349-439b-9aa1-5b519e0e08f7-etc-kubernetes\") pod \"multus-7f2mt\" (UID: \"33b888e2-d349-439b-9aa1-5b519e0e08f7\") " pod="openshift-multus/multus-7f2mt" Oct 30 16:20:45.832951 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.832814 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f2de1ce0-1c1e-4276-80a9-66239e13e352-host-run-netns\") pod \"ovnkube-node-2krz6\" (UID: \"f2de1ce0-1c1e-4276-80a9-66239e13e352\") " pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:45.832951 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.832814 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/33b888e2-d349-439b-9aa1-5b519e0e08f7-multus-conf-dir\") pod \"multus-7f2mt\" (UID: \"33b888e2-d349-439b-9aa1-5b519e0e08f7\") " pod="openshift-multus/multus-7f2mt" Oct 30 16:20:45.832951 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.832839 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f2de1ce0-1c1e-4276-80a9-66239e13e352-systemd-units\") pod \"ovnkube-node-2krz6\" (UID: \"f2de1ce0-1c1e-4276-80a9-66239e13e352\") " pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:45.832951 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.832784 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f2de1ce0-1c1e-4276-80a9-66239e13e352-run-systemd\") pod \"ovnkube-node-2krz6\" (UID: \"f2de1ce0-1c1e-4276-80a9-66239e13e352\") " pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:45.832951 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.832872 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f2de1ce0-1c1e-4276-80a9-66239e13e352-host-cni-netd\") pod \"ovnkube-node-2krz6\" (UID: \"f2de1ce0-1c1e-4276-80a9-66239e13e352\") " pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:45.832951 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.832913 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f2de1ce0-1c1e-4276-80a9-66239e13e352-host-cni-netd\") pod \"ovnkube-node-2krz6\" (UID: \"f2de1ce0-1c1e-4276-80a9-66239e13e352\") " pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:45.832951 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.832873 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/029f4ecb-eb6b-4693-9e9e-ca61f554dd33-kubelet-config\") pod \"global-pull-secret-syncer-ncfzj\" (UID: \"029f4ecb-eb6b-4693-9e9e-ca61f554dd33\") " pod="kube-system/global-pull-secret-syncer-ncfzj" Oct 30 16:20:45.832951 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.832873 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f2de1ce0-1c1e-4276-80a9-66239e13e352-systemd-units\") pod \"ovnkube-node-2krz6\" (UID: \"f2de1ce0-1c1e-4276-80a9-66239e13e352\") " pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:45.832951 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.832906 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/33b888e2-d349-439b-9aa1-5b519e0e08f7-system-cni-dir\") pod \"multus-7f2mt\" (UID: \"33b888e2-d349-439b-9aa1-5b519e0e08f7\") " pod="openshift-multus/multus-7f2mt" Oct 30 16:20:45.832951 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.832960 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f2de1ce0-1c1e-4276-80a9-66239e13e352-run-openvswitch\") pod \"ovnkube-node-2krz6\" (UID: \"f2de1ce0-1c1e-4276-80a9-66239e13e352\") " pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:45.832951 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.832980 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/33b888e2-d349-439b-9aa1-5b519e0e08f7-system-cni-dir\") pod \"multus-7f2mt\" (UID: \"33b888e2-d349-439b-9aa1-5b519e0e08f7\") " pod="openshift-multus/multus-7f2mt" Oct 30 16:20:45.834761 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.832983 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f2de1ce0-1c1e-4276-80a9-66239e13e352-node-log\") pod \"ovnkube-node-2krz6\" (UID: \"f2de1ce0-1c1e-4276-80a9-66239e13e352\") " pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:45.834761 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.833005 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f2de1ce0-1c1e-4276-80a9-66239e13e352-run-openvswitch\") pod \"ovnkube-node-2krz6\" (UID: \"f2de1ce0-1c1e-4276-80a9-66239e13e352\") " pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:45.834761 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.833007 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/749b2b57-f20e-4cdb-8d89-d2bac352f751-sys\") pod \"tuned-c8x4k\" (UID: \"749b2b57-f20e-4cdb-8d89-d2bac352f751\") " pod="openshift-cluster-node-tuning-operator/tuned-c8x4k" Oct 30 16:20:45.834761 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.833038 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f2de1ce0-1c1e-4276-80a9-66239e13e352-node-log\") pod \"ovnkube-node-2krz6\" (UID: \"f2de1ce0-1c1e-4276-80a9-66239e13e352\") " pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:45.834761 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.833039 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/749b2b57-f20e-4cdb-8d89-d2bac352f751-etc-kubernetes\") pod \"tuned-c8x4k\" (UID: \"749b2b57-f20e-4cdb-8d89-d2bac352f751\") " pod="openshift-cluster-node-tuning-operator/tuned-c8x4k" Oct 30 16:20:45.834761 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.833061 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/749b2b57-f20e-4cdb-8d89-d2bac352f751-sys\") pod \"tuned-c8x4k\" (UID: \"749b2b57-f20e-4cdb-8d89-d2bac352f751\") " pod="openshift-cluster-node-tuning-operator/tuned-c8x4k" Oct 30 16:20:45.834761 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.833068 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d6db39f8-bf92-44e8-9144-51eb3b9dee8d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wxvph\" (UID: \"d6db39f8-bf92-44e8-9144-51eb3b9dee8d\") " pod="openshift-insights/insights-runtime-extractor-wxvph" Oct 30 16:20:45.834761 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.833105 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/33b888e2-d349-439b-9aa1-5b519e0e08f7-multus-socket-dir-parent\") pod \"multus-7f2mt\" (UID: \"33b888e2-d349-439b-9aa1-5b519e0e08f7\") " pod="openshift-multus/multus-7f2mt" Oct 30 16:20:45.834761 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:20:45.833118 5052 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 16:20:45.834761 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.833068 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/749b2b57-f20e-4cdb-8d89-d2bac352f751-etc-kubernetes\") pod \"tuned-c8x4k\" (UID: \"749b2b57-f20e-4cdb-8d89-d2bac352f751\") " pod="openshift-cluster-node-tuning-operator/tuned-c8x4k" Oct 30 16:20:45.834761 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.833133 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f2de1ce0-1c1e-4276-80a9-66239e13e352-run-ovn\") pod \"ovnkube-node-2krz6\" (UID: \"f2de1ce0-1c1e-4276-80a9-66239e13e352\") " pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:45.834761 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.833160 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a6b95002-74f1-475d-a64a-6495733cf142-device-dir\") pod \"aws-ebs-csi-driver-node-g69ll\" (UID: \"a6b95002-74f1-475d-a64a-6495733cf142\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g69ll" Oct 30 16:20:45.834761 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.833166 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/33b888e2-d349-439b-9aa1-5b519e0e08f7-multus-socket-dir-parent\") pod \"multus-7f2mt\" (UID: \"33b888e2-d349-439b-9aa1-5b519e0e08f7\") " pod="openshift-multus/multus-7f2mt" Oct 30 16:20:45.834761 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:20:45.833181 5052 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d6db39f8-bf92-44e8-9144-51eb3b9dee8d-kube-rbac-proxy-cm podName:d6db39f8-bf92-44e8-9144-51eb3b9dee8d nodeName:}" failed. No retries permitted until 2026-10-30 16:20:46.333165696 +0000 UTC m=+1.860500912 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d6db39f8-bf92-44e8-9144-51eb3b9dee8d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wxvph" (UID: "d6db39f8-bf92-44e8-9144-51eb3b9dee8d") : configmap "kube-rbac-proxy" not found Oct 30 16:20:45.834761 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.833196 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f2de1ce0-1c1e-4276-80a9-66239e13e352-run-ovn\") pod \"ovnkube-node-2krz6\" (UID: \"f2de1ce0-1c1e-4276-80a9-66239e13e352\") " pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:45.834761 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.833206 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a6b95002-74f1-475d-a64a-6495733cf142-device-dir\") pod \"aws-ebs-csi-driver-node-g69ll\" (UID: \"a6b95002-74f1-475d-a64a-6495733cf142\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g69ll" Oct 30 16:20:45.834761 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.833214 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/749b2b57-f20e-4cdb-8d89-d2bac352f751-host\") pod \"tuned-c8x4k\" (UID: \"749b2b57-f20e-4cdb-8d89-d2bac352f751\") " pod="openshift-cluster-node-tuning-operator/tuned-c8x4k" Oct 30 16:20:45.834761 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.833242 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/33b888e2-d349-439b-9aa1-5b519e0e08f7-cnibin\") pod \"multus-7f2mt\" (UID: \"33b888e2-d349-439b-9aa1-5b519e0e08f7\") " pod="openshift-multus/multus-7f2mt" Oct 30 16:20:45.834761 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.833252 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/749b2b57-f20e-4cdb-8d89-d2bac352f751-host\") pod \"tuned-c8x4k\" (UID: \"749b2b57-f20e-4cdb-8d89-d2bac352f751\") " pod="openshift-cluster-node-tuning-operator/tuned-c8x4k" Oct 30 16:20:45.834761 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.833264 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f2de1ce0-1c1e-4276-80a9-66239e13e352-log-socket\") pod \"ovnkube-node-2krz6\" (UID: \"f2de1ce0-1c1e-4276-80a9-66239e13e352\") " pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:45.834761 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.833286 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f2de1ce0-1c1e-4276-80a9-66239e13e352-log-socket\") pod \"ovnkube-node-2krz6\" (UID: \"f2de1ce0-1c1e-4276-80a9-66239e13e352\") " pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:45.834761 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.833290 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f79e659b-79c9-43ee-8733-c685f60e87fb-registry-tls\") pod \"image-registry-65d65dd574-nbb2b\" (UID: \"f79e659b-79c9-43ee-8733-c685f60e87fb\") " pod="openshift-image-registry/image-registry-65d65dd574-nbb2b" Oct 30 16:20:45.834761 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.833299 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/33b888e2-d349-439b-9aa1-5b519e0e08f7-cnibin\") pod \"multus-7f2mt\" (UID: \"33b888e2-d349-439b-9aa1-5b519e0e08f7\") " pod="openshift-multus/multus-7f2mt" Oct 30 16:20:45.834761 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.833315 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/749b2b57-f20e-4cdb-8d89-d2bac352f751-etc-sysconfig\") pod \"tuned-c8x4k\" (UID: \"749b2b57-f20e-4cdb-8d89-d2bac352f751\") " pod="openshift-cluster-node-tuning-operator/tuned-c8x4k" Oct 30 16:20:45.834761 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.833339 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d6db39f8-bf92-44e8-9144-51eb3b9dee8d-crio-socket\") pod \"insights-runtime-extractor-wxvph\" (UID: \"d6db39f8-bf92-44e8-9144-51eb3b9dee8d\") " pod="openshift-insights/insights-runtime-extractor-wxvph" Oct 30 16:20:45.834761 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.833365 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a0cd0943-8d62-41e4-b3e7-29eaabec7277-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-sfq4m\" (UID: \"a0cd0943-8d62-41e4-b3e7-29eaabec7277\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sfq4m" Oct 30 16:20:45.834761 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.833391 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a6b95002-74f1-475d-a64a-6495733cf142-sys-fs\") pod \"aws-ebs-csi-driver-node-g69ll\" (UID: \"a6b95002-74f1-475d-a64a-6495733cf142\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g69ll" Oct 30 16:20:45.834761 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.833412 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/029f4ecb-eb6b-4693-9e9e-ca61f554dd33-dbus\") pod \"global-pull-secret-syncer-ncfzj\" (UID: \"029f4ecb-eb6b-4693-9e9e-ca61f554dd33\") " pod="kube-system/global-pull-secret-syncer-ncfzj" Oct 30 16:20:45.834761 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.833438 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d10078a0-c21c-4f18-9c5a-4915fb7c8af3-host\") pod \"node-ca-2l5ht\" (UID: \"d10078a0-c21c-4f18-9c5a-4915fb7c8af3\") " pod="openshift-image-registry/node-ca-2l5ht" Oct 30 16:20:45.834761 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.833444 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/749b2b57-f20e-4cdb-8d89-d2bac352f751-etc-sysconfig\") pod \"tuned-c8x4k\" (UID: \"749b2b57-f20e-4cdb-8d89-d2bac352f751\") " pod="openshift-cluster-node-tuning-operator/tuned-c8x4k" Oct 30 16:20:45.834761 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.833462 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9ea50669-2c08-403d-8d08-f9ffffa5b896-tuning-conf-dir\") pod \"multus-additional-cni-plugins-x9ddt\" (UID: \"9ea50669-2c08-403d-8d08-f9ffffa5b896\") " pod="openshift-multus/multus-additional-cni-plugins-x9ddt" Oct 30 16:20:45.834761 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.833479 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a6b95002-74f1-475d-a64a-6495733cf142-sys-fs\") pod \"aws-ebs-csi-driver-node-g69ll\" (UID: \"a6b95002-74f1-475d-a64a-6495733cf142\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g69ll" Oct 30 16:20:45.834761 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.833486 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/749b2b57-f20e-4cdb-8d89-d2bac352f751-run\") pod \"tuned-c8x4k\" (UID: \"749b2b57-f20e-4cdb-8d89-d2bac352f751\") " pod="openshift-cluster-node-tuning-operator/tuned-c8x4k" Oct 30 16:20:45.834761 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.833506 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d10078a0-c21c-4f18-9c5a-4915fb7c8af3-host\") pod \"node-ca-2l5ht\" (UID: \"d10078a0-c21c-4f18-9c5a-4915fb7c8af3\") " pod="openshift-image-registry/node-ca-2l5ht" Oct 30 16:20:45.834761 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.833515 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a6b95002-74f1-475d-a64a-6495733cf142-kubelet-dir\") pod \"aws-ebs-csi-driver-node-g69ll\" (UID: \"a6b95002-74f1-475d-a64a-6495733cf142\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g69ll" Oct 30 16:20:45.834761 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.833538 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/749b2b57-f20e-4cdb-8d89-d2bac352f751-lib-modules\") pod \"tuned-c8x4k\" (UID: \"749b2b57-f20e-4cdb-8d89-d2bac352f751\") " pod="openshift-cluster-node-tuning-operator/tuned-c8x4k" Oct 30 16:20:45.834761 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.833556 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/029f4ecb-eb6b-4693-9e9e-ca61f554dd33-dbus\") pod \"global-pull-secret-syncer-ncfzj\" (UID: \"029f4ecb-eb6b-4693-9e9e-ca61f554dd33\") " pod="kube-system/global-pull-secret-syncer-ncfzj" Oct 30 16:20:45.834761 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.833561 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9ec1217e-8623-42a6-a052-adee153936a2-hosts-file\") pod \"node-resolver-lghnm\" (UID: \"9ec1217e-8623-42a6-a052-adee153936a2\") " pod="openshift-dns/node-resolver-lghnm" Oct 30 16:20:45.834761 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.833439 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d6db39f8-bf92-44e8-9144-51eb3b9dee8d-crio-socket\") pod \"insights-runtime-extractor-wxvph\" (UID: \"d6db39f8-bf92-44e8-9144-51eb3b9dee8d\") " pod="openshift-insights/insights-runtime-extractor-wxvph" Oct 30 16:20:45.834761 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.833620 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3e90fa4d-66a8-4689-8f41-ca090d6697a9-metrics-tls\") pod \"dns-default-qg6t6\" (UID: \"3e90fa4d-66a8-4689-8f41-ca090d6697a9\") " pod="openshift-dns/dns-default-qg6t6" Oct 30 16:20:45.834761 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.833639 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9ec1217e-8623-42a6-a052-adee153936a2-hosts-file\") pod \"node-resolver-lghnm\" (UID: \"9ec1217e-8623-42a6-a052-adee153936a2\") " pod="openshift-dns/node-resolver-lghnm" Oct 30 16:20:45.834761 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.833659 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/749b2b57-f20e-4cdb-8d89-d2bac352f751-etc-sysctl-d\") pod \"tuned-c8x4k\" (UID: \"749b2b57-f20e-4cdb-8d89-d2bac352f751\") " pod="openshift-cluster-node-tuning-operator/tuned-c8x4k" Oct 30 16:20:45.834761 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.833684 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f2de1ce0-1c1e-4276-80a9-66239e13e352-host-kubelet\") pod \"ovnkube-node-2krz6\" (UID: \"f2de1ce0-1c1e-4276-80a9-66239e13e352\") " pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:45.834761 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.833695 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a6b95002-74f1-475d-a64a-6495733cf142-kubelet-dir\") pod \"aws-ebs-csi-driver-node-g69ll\" (UID: \"a6b95002-74f1-475d-a64a-6495733cf142\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g69ll" Oct 30 16:20:45.834761 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.833705 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/33b888e2-d349-439b-9aa1-5b519e0e08f7-host-var-lib-kubelet\") pod \"multus-7f2mt\" (UID: \"33b888e2-d349-439b-9aa1-5b519e0e08f7\") " pod="openshift-multus/multus-7f2mt" Oct 30 16:20:45.834761 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.833744 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9ea50669-2c08-403d-8d08-f9ffffa5b896-system-cni-dir\") pod \"multus-additional-cni-plugins-x9ddt\" (UID: \"9ea50669-2c08-403d-8d08-f9ffffa5b896\") " pod="openshift-multus/multus-additional-cni-plugins-x9ddt" Oct 30 16:20:45.834761 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.833750 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/749b2b57-f20e-4cdb-8d89-d2bac352f751-lib-modules\") pod \"tuned-c8x4k\" (UID: \"749b2b57-f20e-4cdb-8d89-d2bac352f751\") " pod="openshift-cluster-node-tuning-operator/tuned-c8x4k" Oct 30 16:20:45.834761 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.833657 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9ea50669-2c08-403d-8d08-f9ffffa5b896-tuning-conf-dir\") pod \"multus-additional-cni-plugins-x9ddt\" (UID: \"9ea50669-2c08-403d-8d08-f9ffffa5b896\") " pod="openshift-multus/multus-additional-cni-plugins-x9ddt" Oct 30 16:20:45.834761 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.833777 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f2de1ce0-1c1e-4276-80a9-66239e13e352-host-slash\") pod \"ovnkube-node-2krz6\" (UID: \"f2de1ce0-1c1e-4276-80a9-66239e13e352\") " pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:45.834761 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.833798 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/749b2b57-f20e-4cdb-8d89-d2bac352f751-etc-sysctl-d\") pod \"tuned-c8x4k\" (UID: \"749b2b57-f20e-4cdb-8d89-d2bac352f751\") " pod="openshift-cluster-node-tuning-operator/tuned-c8x4k" Oct 30 16:20:45.834761 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.833804 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b99d8a1f-000b-48ff-b4b0-7a62bc79c032-service-ca-bundle\") pod \"router-default-df85cd6ff-6pd8x\" (UID: \"b99d8a1f-000b-48ff-b4b0-7a62bc79c032\") " pod="openshift-ingress/router-default-df85cd6ff-6pd8x" Oct 30 16:20:45.834761 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.833794 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9ea50669-2c08-403d-8d08-f9ffffa5b896-system-cni-dir\") pod \"multus-additional-cni-plugins-x9ddt\" (UID: \"9ea50669-2c08-403d-8d08-f9ffffa5b896\") " pod="openshift-multus/multus-additional-cni-plugins-x9ddt" Oct 30 16:20:45.834761 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.833840 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/749b2b57-f20e-4cdb-8d89-d2bac352f751-run\") pod \"tuned-c8x4k\" (UID: \"749b2b57-f20e-4cdb-8d89-d2bac352f751\") " pod="openshift-cluster-node-tuning-operator/tuned-c8x4k" Oct 30 16:20:45.834761 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.833845 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f2de1ce0-1c1e-4276-80a9-66239e13e352-host-kubelet\") pod \"ovnkube-node-2krz6\" (UID: \"f2de1ce0-1c1e-4276-80a9-66239e13e352\") " pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:45.834761 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.833865 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b99d8a1f-000b-48ff-b4b0-7a62bc79c032-metrics-certs\") pod \"router-default-df85cd6ff-6pd8x\" (UID: \"b99d8a1f-000b-48ff-b4b0-7a62bc79c032\") " pod="openshift-ingress/router-default-df85cd6ff-6pd8x" Oct 30 16:20:45.834761 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:20:45.833882 5052 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 16:20:45.834761 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:20:45.833890 5052 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b99d8a1f-000b-48ff-b4b0-7a62bc79c032-service-ca-bundle podName:b99d8a1f-000b-48ff-b4b0-7a62bc79c032 nodeName:}" failed. No retries permitted until 2026-10-30 16:20:46.333876993 +0000 UTC m=+1.861212190 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b99d8a1f-000b-48ff-b4b0-7a62bc79c032-service-ca-bundle") pod "router-default-df85cd6ff-6pd8x" (UID: "b99d8a1f-000b-48ff-b4b0-7a62bc79c032") : configmap references non-existent config key: service-ca.crt Oct 30 16:20:45.834761 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:20:45.833963 5052 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e90fa4d-66a8-4689-8f41-ca090d6697a9-metrics-tls podName:3e90fa4d-66a8-4689-8f41-ca090d6697a9 nodeName:}" failed. No retries permitted until 2026-10-30 16:20:46.33394591 +0000 UTC m=+1.861281122 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3e90fa4d-66a8-4689-8f41-ca090d6697a9-metrics-tls") pod "dns-default-qg6t6" (UID: "3e90fa4d-66a8-4689-8f41-ca090d6697a9") : secret "dns-default-metrics-tls" not found Oct 30 16:20:45.834761 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.833990 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/33b888e2-d349-439b-9aa1-5b519e0e08f7-host-var-lib-cni-multus\") pod \"multus-7f2mt\" (UID: \"33b888e2-d349-439b-9aa1-5b519e0e08f7\") " pod="openshift-multus/multus-7f2mt" Oct 30 16:20:45.834761 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.834005 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f2de1ce0-1c1e-4276-80a9-66239e13e352-host-slash\") pod \"ovnkube-node-2krz6\" (UID: \"f2de1ce0-1c1e-4276-80a9-66239e13e352\") " pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:45.834761 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.834017 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9ea50669-2c08-403d-8d08-f9ffffa5b896-cnibin\") pod \"multus-additional-cni-plugins-x9ddt\" (UID: \"9ea50669-2c08-403d-8d08-f9ffffa5b896\") " pod="openshift-multus/multus-additional-cni-plugins-x9ddt" Oct 30 16:20:45.834761 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.834044 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a6b95002-74f1-475d-a64a-6495733cf142-socket-dir\") pod \"aws-ebs-csi-driver-node-g69ll\" (UID: \"a6b95002-74f1-475d-a64a-6495733cf142\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g69ll" Oct 30 16:20:45.834761 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.834062 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/33b888e2-d349-439b-9aa1-5b519e0e08f7-host-var-lib-kubelet\") pod \"multus-7f2mt\" (UID: \"33b888e2-d349-439b-9aa1-5b519e0e08f7\") " pod="openshift-multus/multus-7f2mt" Oct 30 16:20:45.834761 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.834072 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/33b888e2-d349-439b-9aa1-5b519e0e08f7-host-var-lib-cni-bin\") pod \"multus-7f2mt\" (UID: \"33b888e2-d349-439b-9aa1-5b519e0e08f7\") " pod="openshift-multus/multus-7f2mt" Oct 30 16:20:45.834761 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.834112 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9ea50669-2c08-403d-8d08-f9ffffa5b896-cnibin\") pod \"multus-additional-cni-plugins-x9ddt\" (UID: \"9ea50669-2c08-403d-8d08-f9ffffa5b896\") " pod="openshift-multus/multus-additional-cni-plugins-x9ddt" Oct 30 16:20:45.834761 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.834115 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9fafe556-9653-49df-a851-100b51bb7f47-registry-tls\") pod \"image-registry-f8dddb4c-dcbgb\" (UID: \"9fafe556-9653-49df-a851-100b51bb7f47\") " pod="openshift-image-registry/image-registry-f8dddb4c-dcbgb" Oct 30 16:20:45.834761 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.834142 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f2de1ce0-1c1e-4276-80a9-66239e13e352-host-cni-bin\") pod \"ovnkube-node-2krz6\" (UID: \"f2de1ce0-1c1e-4276-80a9-66239e13e352\") " pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:45.834761 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.834169 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/33b888e2-d349-439b-9aa1-5b519e0e08f7-host-var-lib-cni-multus\") pod \"multus-7f2mt\" (UID: \"33b888e2-d349-439b-9aa1-5b519e0e08f7\") " pod="openshift-multus/multus-7f2mt" Oct 30 16:20:45.834761 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.834171 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f2de1ce0-1c1e-4276-80a9-66239e13e352-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2krz6\" (UID: \"f2de1ce0-1c1e-4276-80a9-66239e13e352\") " pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:45.834761 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.834214 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f2de1ce0-1c1e-4276-80a9-66239e13e352-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2krz6\" (UID: \"f2de1ce0-1c1e-4276-80a9-66239e13e352\") " pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:45.834761 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.834304 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f2de1ce0-1c1e-4276-80a9-66239e13e352-host-cni-bin\") pod \"ovnkube-node-2krz6\" (UID: \"f2de1ce0-1c1e-4276-80a9-66239e13e352\") " pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:45.834761 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.834373 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a6b95002-74f1-475d-a64a-6495733cf142-socket-dir\") pod \"aws-ebs-csi-driver-node-g69ll\" (UID: \"a6b95002-74f1-475d-a64a-6495733cf142\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g69ll" Oct 30 16:20:45.834761 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.834405 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/33b888e2-d349-439b-9aa1-5b519e0e08f7-host-var-lib-cni-bin\") pod \"multus-7f2mt\" (UID: \"33b888e2-d349-439b-9aa1-5b519e0e08f7\") " pod="openshift-multus/multus-7f2mt" Oct 30 16:20:45.838001 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.835198 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1042378b-9f46-4e19-88e4-2628e532e8c3-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-992nb\" (UID: \"1042378b-9f46-4e19-88e4-2628e532e8c3\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-992nb" Oct 30 16:20:45.838001 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.835943 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/06332900-7903-4275-8e33-de06de7f6580-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-cswcz\" (UID: \"06332900-7903-4275-8e33-de06de7f6580\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-cswcz" Oct 30 16:20:45.838001 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.836198 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f79e659b-79c9-43ee-8733-c685f60e87fb-registry-tls\") pod \"image-registry-65d65dd574-nbb2b\" (UID: \"f79e659b-79c9-43ee-8733-c685f60e87fb\") " pod="openshift-image-registry/image-registry-65d65dd574-nbb2b" Oct 30 16:20:45.838001 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.836502 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b99d8a1f-000b-48ff-b4b0-7a62bc79c032-metrics-certs\") pod \"router-default-df85cd6ff-6pd8x\" (UID: \"b99d8a1f-000b-48ff-b4b0-7a62bc79c032\") " pod="openshift-ingress/router-default-df85cd6ff-6pd8x" Oct 30 16:20:45.838001 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.836651 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a0cd0943-8d62-41e4-b3e7-29eaabec7277-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-sfq4m\" (UID: \"a0cd0943-8d62-41e4-b3e7-29eaabec7277\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sfq4m" Oct 30 16:20:45.838001 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.837369 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9fafe556-9653-49df-a851-100b51bb7f47-registry-tls\") pod \"image-registry-f8dddb4c-dcbgb\" (UID: \"9fafe556-9653-49df-a851-100b51bb7f47\") " pod="openshift-image-registry/image-registry-f8dddb4c-dcbgb" Oct 30 16:20:45.899347 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.899322 5052 scope.go:122] "RemoveContainer" containerID="1a368df4208847e057012340c138c128bc78134c8e9c801e988b863ea39e70fd" Oct 30 16:20:45.899477 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.899367 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-f8dddb4c-dcbgb" Oct 30 16:20:45.899744 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.899728 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sfq4m" Oct 30 16:20:45.901291 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.901271 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-cswcz" Oct 30 16:20:45.903253 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.903239 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-992nb" Oct 30 16:20:45.910503 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:45.910486 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-65d65dd574-nbb2b" Oct 30 16:20:46.103696 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:46.103633 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:46.231346 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:46.231325 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-sfq4m"] Oct 30 16:20:46.235145 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:46.233841 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0cd0943_8d62_41e4_b3e7_29eaabec7277.slice/crio-c29e1ab3eabf22673dc42d9bf9a91e8d737894aad02f8f12a8eeafd9faf4b68e WatchSource:0}: Error finding container c29e1ab3eabf22673dc42d9bf9a91e8d737894aad02f8f12a8eeafd9faf4b68e: Status 404 returned error can't find the container with id c29e1ab3eabf22673dc42d9bf9a91e8d737894aad02f8f12a8eeafd9faf4b68e Oct 30 16:20:46.235339 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:46.235163 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-f8dddb4c-dcbgb"] Oct 30 16:20:46.236434 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:46.236418 5052 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 16:20:46.240538 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:46.240517 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fafe556_9653_49df_a851_100b51bb7f47.slice/crio-d524f0f7e174284fada419ee192122aa2cc17519cb2161c1e1ac90be23891592 WatchSource:0}: Error finding container d524f0f7e174284fada419ee192122aa2cc17519cb2161c1e1ac90be23891592: Status 404 returned error can't find the container with id d524f0f7e174284fada419ee192122aa2cc17519cb2161c1e1ac90be23891592 Oct 30 16:20:46.254392 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:46.254345 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-65d65dd574-nbb2b"] Oct 30 16:20:46.257080 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:46.257056 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf79e659b_79c9_43ee_8733_c685f60e87fb.slice/crio-11f929fe58d7dded4d64576ed87d3de315d7bb574f1401ca7bf98ef5d4b5069a WatchSource:0}: Error finding container 11f929fe58d7dded4d64576ed87d3de315d7bb574f1401ca7bf98ef5d4b5069a: Status 404 returned error can't find the container with id 11f929fe58d7dded4d64576ed87d3de315d7bb574f1401ca7bf98ef5d4b5069a Oct 30 16:20:46.263570 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:46.263554 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-cswcz"] Oct 30 16:20:46.266633 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:46.266609 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-992nb"] Oct 30 16:20:46.270177 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:46.270145 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1042378b_9f46_4e19_88e4_2628e532e8c3.slice/crio-cf8bbecdefe3dfdd650afdc50d7b57effff628a61652f13e1cd19ef71f8e2587 WatchSource:0}: Error finding container cf8bbecdefe3dfdd650afdc50d7b57effff628a61652f13e1cd19ef71f8e2587: Status 404 returned error can't find the container with id cf8bbecdefe3dfdd650afdc50d7b57effff628a61652f13e1cd19ef71f8e2587 Oct 30 16:20:46.339982 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:46.339947 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3e90fa4d-66a8-4689-8f41-ca090d6697a9-metrics-tls\") pod \"dns-default-qg6t6\" (UID: \"3e90fa4d-66a8-4689-8f41-ca090d6697a9\") " pod="openshift-dns/dns-default-qg6t6" Oct 30 16:20:46.340128 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:46.340003 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b99d8a1f-000b-48ff-b4b0-7a62bc79c032-service-ca-bundle\") pod \"router-default-df85cd6ff-6pd8x\" (UID: \"b99d8a1f-000b-48ff-b4b0-7a62bc79c032\") " pod="openshift-ingress/router-default-df85cd6ff-6pd8x" Oct 30 16:20:46.340128 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:46.340052 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d6db39f8-bf92-44e8-9144-51eb3b9dee8d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wxvph\" (UID: \"d6db39f8-bf92-44e8-9144-51eb3b9dee8d\") " pod="openshift-insights/insights-runtime-extractor-wxvph" Oct 30 16:20:46.340128 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:20:46.340103 5052 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 16:20:46.340496 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:46.340112 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f3d36d6-5098-43cc-8f8a-bd83d5dea5fc-metrics-certs\") pod \"network-metrics-daemon-xj6cp\" (UID: \"6f3d36d6-5098-43cc-8f8a-bd83d5dea5fc\") " pod="openshift-multus/network-metrics-daemon-xj6cp" Oct 30 16:20:46.340496 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:20:46.340166 5052 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e90fa4d-66a8-4689-8f41-ca090d6697a9-metrics-tls podName:3e90fa4d-66a8-4689-8f41-ca090d6697a9 nodeName:}" failed. No retries permitted until 2026-10-30 16:20:47.340146066 +0000 UTC m=+2.867481280 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3e90fa4d-66a8-4689-8f41-ca090d6697a9-metrics-tls") pod "dns-default-qg6t6" (UID: "3e90fa4d-66a8-4689-8f41-ca090d6697a9") : secret "dns-default-metrics-tls" not found Oct 30 16:20:46.340496 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:20:46.340185 5052 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 16:20:46.340496 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:20:46.340207 5052 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b99d8a1f-000b-48ff-b4b0-7a62bc79c032-service-ca-bundle podName:b99d8a1f-000b-48ff-b4b0-7a62bc79c032 nodeName:}" failed. No retries permitted until 2026-10-30 16:20:47.340187262 +0000 UTC m=+2.867522474 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b99d8a1f-000b-48ff-b4b0-7a62bc79c032-service-ca-bundle") pod "router-default-df85cd6ff-6pd8x" (UID: "b99d8a1f-000b-48ff-b4b0-7a62bc79c032") : configmap references non-existent config key: service-ca.crt Oct 30 16:20:46.340496 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:20:46.340229 5052 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 16:20:46.340496 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:20:46.340231 5052 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f3d36d6-5098-43cc-8f8a-bd83d5dea5fc-metrics-certs podName:6f3d36d6-5098-43cc-8f8a-bd83d5dea5fc nodeName:}" failed. No retries permitted until 2026-10-30 16:20:47.340220186 +0000 UTC m=+2.867555384 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6f3d36d6-5098-43cc-8f8a-bd83d5dea5fc-metrics-certs") pod "network-metrics-daemon-xj6cp" (UID: "6f3d36d6-5098-43cc-8f8a-bd83d5dea5fc") : secret "metrics-daemon-secret" not found Oct 30 16:20:46.340496 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:20:46.340280 5052 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6db39f8-bf92-44e8-9144-51eb3b9dee8d-insights-runtime-extractor-tls podName:d6db39f8-bf92-44e8-9144-51eb3b9dee8d nodeName:}" failed. No retries permitted until 2026-10-30 16:20:47.340269782 +0000 UTC m=+2.867604996 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d6db39f8-bf92-44e8-9144-51eb3b9dee8d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wxvph" (UID: "d6db39f8-bf92-44e8-9144-51eb3b9dee8d") : secret "insights-runtime-extractor-tls" not found Oct 30 16:20:46.340496 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:46.340331 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d6db39f8-bf92-44e8-9144-51eb3b9dee8d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wxvph\" (UID: \"d6db39f8-bf92-44e8-9144-51eb3b9dee8d\") " pod="openshift-insights/insights-runtime-extractor-wxvph" Oct 30 16:20:46.340936 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:46.340889 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d6db39f8-bf92-44e8-9144-51eb3b9dee8d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wxvph\" (UID: \"d6db39f8-bf92-44e8-9144-51eb3b9dee8d\") " pod="openshift-insights/insights-runtime-extractor-wxvph" Oct 30 16:20:46.667544 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:46.667501 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-65d65dd574-nbb2b" event={"ID":"f79e659b-79c9-43ee-8733-c685f60e87fb","Type":"ContainerStarted","Data":"f83c82fa4b1237aa8a0ce2a7bca50d3ee16e5a90efeff5f2528eed7fe6f81c65"} Oct 30 16:20:46.668032 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:46.667565 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-65d65dd574-nbb2b" event={"ID":"f79e659b-79c9-43ee-8733-c685f60e87fb","Type":"ContainerStarted","Data":"11f929fe58d7dded4d64576ed87d3de315d7bb574f1401ca7bf98ef5d4b5069a"} Oct 30 16:20:46.668782 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:46.668755 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-cswcz" event={"ID":"06332900-7903-4275-8e33-de06de7f6580","Type":"ContainerStarted","Data":"135e1e2c94e50750ea007f854c65bed78d7092b963be452d971d1a21d7d4d5d4"} Oct 30 16:20:46.669942 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:46.669919 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sfq4m" event={"ID":"a0cd0943-8d62-41e4-b3e7-29eaabec7277","Type":"ContainerStarted","Data":"c29e1ab3eabf22673dc42d9bf9a91e8d737894aad02f8f12a8eeafd9faf4b68e"} Oct 30 16:20:46.671449 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:46.671429 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4wrf8_4f0f46f4-5cda-4d67-b6f3-47df06eb844c/console-operator/2.log" Oct 30 16:20:46.671884 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:46.671868 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4wrf8_4f0f46f4-5cda-4d67-b6f3-47df06eb844c/console-operator/1.log" Oct 30 16:20:46.671972 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:46.671901 5052 generic.go:356] "Generic (PLEG): container finished" podID="4f0f46f4-5cda-4d67-b6f3-47df06eb844c" containerID="05a45b04bb7aeb7f8a488bbf8297cae4a920facacc71e3a01247ce731ac01030" exitCode=255 Oct 30 16:20:46.671972 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:46.671952 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-4wrf8" event={"ID":"4f0f46f4-5cda-4d67-b6f3-47df06eb844c","Type":"ContainerDied","Data":"05a45b04bb7aeb7f8a488bbf8297cae4a920facacc71e3a01247ce731ac01030"} Oct 30 16:20:46.672051 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:46.671994 5052 scope.go:122] "RemoveContainer" containerID="1a368df4208847e057012340c138c128bc78134c8e9c801e988b863ea39e70fd" Oct 30 16:20:46.672318 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:46.672292 5052 scope.go:122] "RemoveContainer" containerID="05a45b04bb7aeb7f8a488bbf8297cae4a920facacc71e3a01247ce731ac01030" Oct 30 16:20:46.672521 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:20:46.672491 5052 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-4wrf8_openshift-console-operator(4f0f46f4-5cda-4d67-b6f3-47df06eb844c)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-4wrf8" podUID="4f0f46f4-5cda-4d67-b6f3-47df06eb844c" Oct 30 16:20:46.675793 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:46.675625 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-992nb" event={"ID":"1042378b-9f46-4e19-88e4-2628e532e8c3","Type":"ContainerStarted","Data":"cf8bbecdefe3dfdd650afdc50d7b57effff628a61652f13e1cd19ef71f8e2587"} Oct 30 16:20:46.678413 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:46.678392 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-f8dddb4c-dcbgb" event={"ID":"9fafe556-9653-49df-a851-100b51bb7f47","Type":"ContainerStarted","Data":"dae4370af678f34a2a0a938baad1c3c80f9df6cdecb3d708dcc52f1adb325964"} Oct 30 16:20:46.678522 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:46.678425 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-f8dddb4c-dcbgb" event={"ID":"9fafe556-9653-49df-a851-100b51bb7f47","Type":"ContainerStarted","Data":"d524f0f7e174284fada419ee192122aa2cc17519cb2161c1e1ac90be23891592"} Oct 30 16:20:47.351864 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:47.351824 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3e90fa4d-66a8-4689-8f41-ca090d6697a9-metrics-tls\") pod \"dns-default-qg6t6\" (UID: \"3e90fa4d-66a8-4689-8f41-ca090d6697a9\") " pod="openshift-dns/dns-default-qg6t6" Oct 30 16:20:47.352061 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:47.351874 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b99d8a1f-000b-48ff-b4b0-7a62bc79c032-service-ca-bundle\") pod \"router-default-df85cd6ff-6pd8x\" (UID: \"b99d8a1f-000b-48ff-b4b0-7a62bc79c032\") " pod="openshift-ingress/router-default-df85cd6ff-6pd8x" Oct 30 16:20:47.352061 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:47.351913 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d6db39f8-bf92-44e8-9144-51eb3b9dee8d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wxvph\" (UID: \"d6db39f8-bf92-44e8-9144-51eb3b9dee8d\") " pod="openshift-insights/insights-runtime-extractor-wxvph" Oct 30 16:20:47.352061 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:47.351964 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f3d36d6-5098-43cc-8f8a-bd83d5dea5fc-metrics-certs\") pod \"network-metrics-daemon-xj6cp\" (UID: \"6f3d36d6-5098-43cc-8f8a-bd83d5dea5fc\") " pod="openshift-multus/network-metrics-daemon-xj6cp" Oct 30 16:20:47.352224 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:20:47.352102 5052 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 16:20:47.352224 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:20:47.352166 5052 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f3d36d6-5098-43cc-8f8a-bd83d5dea5fc-metrics-certs podName:6f3d36d6-5098-43cc-8f8a-bd83d5dea5fc nodeName:}" failed. No retries permitted until 2026-10-30 16:20:49.352146944 +0000 UTC m=+4.879482153 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6f3d36d6-5098-43cc-8f8a-bd83d5dea5fc-metrics-certs") pod "network-metrics-daemon-xj6cp" (UID: "6f3d36d6-5098-43cc-8f8a-bd83d5dea5fc") : secret "metrics-daemon-secret" not found Oct 30 16:20:47.352570 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:20:47.352548 5052 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 16:20:47.352696 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:20:47.352616 5052 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e90fa4d-66a8-4689-8f41-ca090d6697a9-metrics-tls podName:3e90fa4d-66a8-4689-8f41-ca090d6697a9 nodeName:}" failed. No retries permitted until 2026-10-30 16:20:49.352601243 +0000 UTC m=+4.879936441 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3e90fa4d-66a8-4689-8f41-ca090d6697a9-metrics-tls") pod "dns-default-qg6t6" (UID: "3e90fa4d-66a8-4689-8f41-ca090d6697a9") : secret "dns-default-metrics-tls" not found Oct 30 16:20:47.353254 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:47.353223 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b99d8a1f-000b-48ff-b4b0-7a62bc79c032-service-ca-bundle\") pod \"router-default-df85cd6ff-6pd8x\" (UID: \"b99d8a1f-000b-48ff-b4b0-7a62bc79c032\") " pod="openshift-ingress/router-default-df85cd6ff-6pd8x" Oct 30 16:20:47.353354 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:20:47.353322 5052 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 16:20:47.353420 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:20:47.353374 5052 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6db39f8-bf92-44e8-9144-51eb3b9dee8d-insights-runtime-extractor-tls podName:d6db39f8-bf92-44e8-9144-51eb3b9dee8d nodeName:}" failed. No retries permitted until 2026-10-30 16:20:49.353359353 +0000 UTC m=+4.880694556 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d6db39f8-bf92-44e8-9144-51eb3b9dee8d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wxvph" (UID: "d6db39f8-bf92-44e8-9144-51eb3b9dee8d") : secret "insights-runtime-extractor-tls" not found Oct 30 16:20:47.402942 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:47.402904 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-df85cd6ff-6pd8x" Oct 30 16:20:47.685419 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:47.685386 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4wrf8_4f0f46f4-5cda-4d67-b6f3-47df06eb844c/console-operator/2.log" Oct 30 16:20:47.687492 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:47.687451 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-992nb" event={"ID":"1042378b-9f46-4e19-88e4-2628e532e8c3","Type":"ContainerStarted","Data":"b029f6dbaa278552a8277bbb7392623e67347f1e2bcd9f08b3b99d08d3027d09"} Oct 30 16:20:47.687637 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:47.687495 5052 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 16:20:47.687637 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:47.687519 5052 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 16:20:47.762124 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:47.762081 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-df85cd6ff-6pd8x"] Oct 30 16:20:48.575794 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:48.575751 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-65d65dd574-nbb2b" Oct 30 16:20:48.692826 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:48.692782 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-cswcz" event={"ID":"06332900-7903-4275-8e33-de06de7f6580","Type":"ContainerStarted","Data":"206d6ee4e7a99770d0deb85b731ac48a1172d52c2f32f4d23c06008e6b24ff16"} Oct 30 16:20:48.693229 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:48.692856 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-cswcz" event={"ID":"06332900-7903-4275-8e33-de06de7f6580","Type":"ContainerStarted","Data":"de8baa92acc5fbb622a412156c604a695ee9d7e48c759a91c9dfaaf15a796c34"} Oct 30 16:20:48.694735 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:48.694708 5052 generic.go:356] "Generic (PLEG): container finished" podID="154b2258-1b09-444b-ad80-2b194195e2cb" containerID="3a438e647b652e073cebbfa3cc04eb65dfbbd23b0bc9def2f41b113a0855b5ad" exitCode=0 Oct 30 16:20:48.694906 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:48.694785 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-p4rk6" event={"ID":"154b2258-1b09-444b-ad80-2b194195e2cb","Type":"ContainerDied","Data":"3a438e647b652e073cebbfa3cc04eb65dfbbd23b0bc9def2f41b113a0855b5ad"} Oct 30 16:20:48.695105 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:48.695090 5052 scope.go:122] "RemoveContainer" containerID="3a438e647b652e073cebbfa3cc04eb65dfbbd23b0bc9def2f41b113a0855b5ad" Oct 30 16:20:48.696493 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:48.696468 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sfq4m" event={"ID":"a0cd0943-8d62-41e4-b3e7-29eaabec7277","Type":"ContainerStarted","Data":"f00c51c6d33737e8092c8a61fea8d5dcf81f80441156dc4fcda2df38337b9db1"} Oct 30 16:20:48.698633 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:48.698609 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-df85cd6ff-6pd8x" event={"ID":"b99d8a1f-000b-48ff-b4b0-7a62bc79c032","Type":"ContainerStarted","Data":"31f56c63d09085a338d2b7a71bc6120fefda5dbbe25d339cc43870a8565a4437"} Oct 30 16:20:48.698729 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:48.698648 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-df85cd6ff-6pd8x" event={"ID":"b99d8a1f-000b-48ff-b4b0-7a62bc79c032","Type":"ContainerStarted","Data":"8566baf8580715860550296e651bca52cd1daf041fb6331182d0cf2856da64ab"} Oct 30 16:20:49.372694 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:49.372607 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3e90fa4d-66a8-4689-8f41-ca090d6697a9-metrics-tls\") pod \"dns-default-qg6t6\" (UID: \"3e90fa4d-66a8-4689-8f41-ca090d6697a9\") " pod="openshift-dns/dns-default-qg6t6" Oct 30 16:20:49.372694 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:49.372671 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d6db39f8-bf92-44e8-9144-51eb3b9dee8d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wxvph\" (UID: \"d6db39f8-bf92-44e8-9144-51eb3b9dee8d\") " pod="openshift-insights/insights-runtime-extractor-wxvph" Oct 30 16:20:49.372907 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:49.372721 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f3d36d6-5098-43cc-8f8a-bd83d5dea5fc-metrics-certs\") pod \"network-metrics-daemon-xj6cp\" (UID: \"6f3d36d6-5098-43cc-8f8a-bd83d5dea5fc\") " pod="openshift-multus/network-metrics-daemon-xj6cp" Oct 30 16:20:49.372907 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:20:49.372826 5052 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 16:20:49.372907 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:20:49.372841 5052 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 16:20:49.372907 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:20:49.372898 5052 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f3d36d6-5098-43cc-8f8a-bd83d5dea5fc-metrics-certs podName:6f3d36d6-5098-43cc-8f8a-bd83d5dea5fc nodeName:}" failed. No retries permitted until 2026-10-30 16:20:53.372879731 +0000 UTC m=+8.900214944 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6f3d36d6-5098-43cc-8f8a-bd83d5dea5fc-metrics-certs") pod "network-metrics-daemon-xj6cp" (UID: "6f3d36d6-5098-43cc-8f8a-bd83d5dea5fc") : secret "metrics-daemon-secret" not found Oct 30 16:20:49.373047 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:20:49.372915 5052 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6db39f8-bf92-44e8-9144-51eb3b9dee8d-insights-runtime-extractor-tls podName:d6db39f8-bf92-44e8-9144-51eb3b9dee8d nodeName:}" failed. No retries permitted until 2026-10-30 16:20:53.372906291 +0000 UTC m=+8.900241497 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d6db39f8-bf92-44e8-9144-51eb3b9dee8d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wxvph" (UID: "d6db39f8-bf92-44e8-9144-51eb3b9dee8d") : secret "insights-runtime-extractor-tls" not found Oct 30 16:20:49.374928 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:49.374906 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3e90fa4d-66a8-4689-8f41-ca090d6697a9-metrics-tls\") pod \"dns-default-qg6t6\" (UID: \"3e90fa4d-66a8-4689-8f41-ca090d6697a9\") " pod="openshift-dns/dns-default-qg6t6" Oct 30 16:20:49.403278 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:49.403253 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-df85cd6ff-6pd8x" Oct 30 16:20:49.405911 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:49.405893 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-df85cd6ff-6pd8x" Oct 30 16:20:49.511787 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:49.511752 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qg6t6" Oct 30 16:20:49.703017 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:49.702986 5052 generic.go:356] "Generic (PLEG): container finished" podID="209633e5-92f7-4a7f-9d68-3d48473ed2bf" containerID="7dcb1eee8cb8e4fb65b44247f158d5e763106751ffdaf2f79781b2f7a41d68fd" exitCode=0 Oct 30 16:20:49.703403 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:49.703066 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jrdmg" event={"ID":"209633e5-92f7-4a7f-9d68-3d48473ed2bf","Type":"ContainerDied","Data":"7dcb1eee8cb8e4fb65b44247f158d5e763106751ffdaf2f79781b2f7a41d68fd"} Oct 30 16:20:49.703445 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:49.703416 5052 scope.go:122] "RemoveContainer" containerID="7dcb1eee8cb8e4fb65b44247f158d5e763106751ffdaf2f79781b2f7a41d68fd" Oct 30 16:20:49.704665 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:49.704643 5052 generic.go:356] "Generic (PLEG): container finished" podID="ef87558b-2ba2-41ce-9a0b-9e4b18663a6c" containerID="c8238ea199508356b3cbc4176329692e250298b0a3db3299e40ee4b66076624e" exitCode=0 Oct 30 16:20:49.704778 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:49.704711 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-4nn7z" event={"ID":"ef87558b-2ba2-41ce-9a0b-9e4b18663a6c","Type":"ContainerDied","Data":"c8238ea199508356b3cbc4176329692e250298b0a3db3299e40ee4b66076624e"} Oct 30 16:20:49.705034 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:49.705020 5052 scope.go:122] "RemoveContainer" containerID="c8238ea199508356b3cbc4176329692e250298b0a3db3299e40ee4b66076624e" Oct 30 16:20:49.706982 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:49.706962 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-p4rk6" event={"ID":"154b2258-1b09-444b-ad80-2b194195e2cb","Type":"ContainerStarted","Data":"74cb5064d4f612f2eb7d0ba813b639e02ce6e5e460de188b986b673347597c18"} Oct 30 16:20:49.707474 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:49.707456 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-df85cd6ff-6pd8x" Oct 30 16:20:49.708551 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:49.708532 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-df85cd6ff-6pd8x" Oct 30 16:20:49.813006 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:49.812982 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qg6t6"] Oct 30 16:20:49.815554 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:49.815530 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e90fa4d_66a8_4689_8f41_ca090d6697a9.slice/crio-4bf0250785879aee5c30a4b3dee4aa4a549ef7b5510b79c377687c22ad70a4e3 WatchSource:0}: Error finding container 4bf0250785879aee5c30a4b3dee4aa4a549ef7b5510b79c377687c22ad70a4e3: Status 404 returned error can't find the container with id 4bf0250785879aee5c30a4b3dee4aa4a549ef7b5510b79c377687c22ad70a4e3 Oct 30 16:20:50.712110 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:50.712076 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qg6t6" event={"ID":"3e90fa4d-66a8-4689-8f41-ca090d6697a9","Type":"ContainerStarted","Data":"4bf0250785879aee5c30a4b3dee4aa4a549ef7b5510b79c377687c22ad70a4e3"} Oct 30 16:20:50.713834 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:50.713808 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-4nn7z" event={"ID":"ef87558b-2ba2-41ce-9a0b-9e4b18663a6c","Type":"ContainerStarted","Data":"875555be51037a4bb6c791f1b6e574b81b9cdc49030aa915dce76e328b3948e3"} Oct 30 16:20:50.715567 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:50.715545 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jrdmg" event={"ID":"209633e5-92f7-4a7f-9d68-3d48473ed2bf","Type":"ContainerStarted","Data":"073fdb752d20a816b634e499d74d38f8071961cd8dd26dfee70341dae91e3b73"} Oct 30 16:20:52.426193 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:52.426159 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-b4ljl" Oct 30 16:20:52.726712 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:52.726644 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qg6t6" event={"ID":"3e90fa4d-66a8-4689-8f41-ca090d6697a9","Type":"ContainerStarted","Data":"348028a70e2831adb756e2d7653e215d9acb939ed185ca30a72029e81e618e56"} Oct 30 16:20:52.726712 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:52.726693 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qg6t6" event={"ID":"3e90fa4d-66a8-4689-8f41-ca090d6697a9","Type":"ContainerStarted","Data":"b39ed24b1772b492c22cb93e8a76f7340a9582c573f6444830da1be943783b9c"} Oct 30 16:20:52.726877 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:52.726757 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-qg6t6" Oct 30 16:20:53.404269 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:53.404230 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f3d36d6-5098-43cc-8f8a-bd83d5dea5fc-metrics-certs\") pod \"network-metrics-daemon-xj6cp\" (UID: \"6f3d36d6-5098-43cc-8f8a-bd83d5dea5fc\") " pod="openshift-multus/network-metrics-daemon-xj6cp" Oct 30 16:20:53.404447 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:53.404293 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d6db39f8-bf92-44e8-9144-51eb3b9dee8d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wxvph\" (UID: \"d6db39f8-bf92-44e8-9144-51eb3b9dee8d\") " pod="openshift-insights/insights-runtime-extractor-wxvph" Oct 30 16:20:53.406551 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:53.406530 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d6db39f8-bf92-44e8-9144-51eb3b9dee8d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wxvph\" (UID: \"d6db39f8-bf92-44e8-9144-51eb3b9dee8d\") " pod="openshift-insights/insights-runtime-extractor-wxvph" Oct 30 16:20:53.406672 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:53.406602 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f3d36d6-5098-43cc-8f8a-bd83d5dea5fc-metrics-certs\") pod \"network-metrics-daemon-xj6cp\" (UID: \"6f3d36d6-5098-43cc-8f8a-bd83d5dea5fc\") " pod="openshift-multus/network-metrics-daemon-xj6cp" Oct 30 16:20:53.408575 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:53.408550 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xj6cp" Oct 30 16:20:53.411956 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:53.411934 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wxvph" Oct 30 16:20:53.620284 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:53.620249 5052 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-4wrf8" Oct 30 16:20:53.620702 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:53.620622 5052 scope.go:122] "RemoveContainer" containerID="05a45b04bb7aeb7f8a488bbf8297cae4a920facacc71e3a01247ce731ac01030" Oct 30 16:20:53.620825 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:20:53.620801 5052 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-4wrf8_openshift-console-operator(4f0f46f4-5cda-4d67-b6f3-47df06eb844c)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-4wrf8" podUID="4f0f46f4-5cda-4d67-b6f3-47df06eb844c" Oct 30 16:20:53.738871 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:53.738759 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xj6cp"] Oct 30 16:20:53.741291 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:53.741269 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f3d36d6_5098_43cc_8f8a_bd83d5dea5fc.slice/crio-1e6e10b30a39f52759eb4a1f2d901afb29ae90df9d66d2305c94ec077e8d88e9 WatchSource:0}: Error finding container 1e6e10b30a39f52759eb4a1f2d901afb29ae90df9d66d2305c94ec077e8d88e9: Status 404 returned error can't find the container with id 1e6e10b30a39f52759eb4a1f2d901afb29ae90df9d66d2305c94ec077e8d88e9 Oct 30 16:20:53.782679 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:53.782640 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wxvph"] Oct 30 16:20:53.787164 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:20:53.787137 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6db39f8_bf92_44e8_9144_51eb3b9dee8d.slice/crio-44dabec1c01877033ed673f7d976f4fddaa25ee6dc236a5af0fcd92fa991af4f WatchSource:0}: Error finding container 44dabec1c01877033ed673f7d976f4fddaa25ee6dc236a5af0fcd92fa991af4f: Status 404 returned error can't find the container with id 44dabec1c01877033ed673f7d976f4fddaa25ee6dc236a5af0fcd92fa991af4f Oct 30 16:20:54.075642 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:54.075564 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:20:54.075818 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:54.075769 5052 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 16:20:54.422674 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:54.422639 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-4wrf8" Oct 30 16:20:54.423051 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:54.423036 5052 scope.go:122] "RemoveContainer" containerID="05a45b04bb7aeb7f8a488bbf8297cae4a920facacc71e3a01247ce731ac01030" Oct 30 16:20:54.423253 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:20:54.423229 5052 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-4wrf8_openshift-console-operator(4f0f46f4-5cda-4d67-b6f3-47df06eb844c)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-4wrf8" podUID="4f0f46f4-5cda-4d67-b6f3-47df06eb844c" Oct 30 16:20:54.735882 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:54.735795 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wxvph" event={"ID":"d6db39f8-bf92-44e8-9144-51eb3b9dee8d","Type":"ContainerStarted","Data":"49a075a00466de9f54b53f1905529b7827df7d34d05b93456186aee07d72d3b7"} Oct 30 16:20:54.735882 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:54.735870 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wxvph" event={"ID":"d6db39f8-bf92-44e8-9144-51eb3b9dee8d","Type":"ContainerStarted","Data":"44dabec1c01877033ed673f7d976f4fddaa25ee6dc236a5af0fcd92fa991af4f"} Oct 30 16:20:54.737176 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:54.737146 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xj6cp" event={"ID":"6f3d36d6-5098-43cc-8f8a-bd83d5dea5fc","Type":"ContainerStarted","Data":"1e6e10b30a39f52759eb4a1f2d901afb29ae90df9d66d2305c94ec077e8d88e9"} Oct 30 16:20:55.272317 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:55.272295 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-f8dddb4c-dcbgb" Oct 30 16:20:55.744077 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:55.744045 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xj6cp" event={"ID":"6f3d36d6-5098-43cc-8f8a-bd83d5dea5fc","Type":"ContainerStarted","Data":"3176a5329313a115ee05d4b013208c19d4a900dcf76a8493af95939ca55c9f2f"} Oct 30 16:20:55.744483 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:55.744107 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xj6cp" event={"ID":"6f3d36d6-5098-43cc-8f8a-bd83d5dea5fc","Type":"ContainerStarted","Data":"8985bff7201cb64f991e7aa43a816f760fe9e563cb2c4acc0bd0fd7692cfef77"} Oct 30 16:20:55.745718 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:55.745690 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wxvph" event={"ID":"d6db39f8-bf92-44e8-9144-51eb3b9dee8d","Type":"ContainerStarted","Data":"928bb014e603c1a911a063d87b45bcc3ac41fe0486d72fae56d6d27a3e619174"} Oct 30 16:20:56.597006 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:56.596971 5052 scope.go:122] "RemoveContainer" containerID="05a45b04bb7aeb7f8a488bbf8297cae4a920facacc71e3a01247ce731ac01030" Oct 30 16:20:56.754405 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:56.754374 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4wrf8_4f0f46f4-5cda-4d67-b6f3-47df06eb844c/console-operator/2.log" Oct 30 16:20:56.754806 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:56.754503 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-4wrf8" event={"ID":"4f0f46f4-5cda-4d67-b6f3-47df06eb844c","Type":"ContainerStarted","Data":"e570113b78ac4786a913e01ca44b79c38b3f987785f2fb73cc7d24ac7a7c2644"} Oct 30 16:20:56.754892 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:56.754871 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-4wrf8" Oct 30 16:20:56.755871 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:56.755834 5052 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-4wrf8 container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.22:8443/readyz\": dial tcp 10.132.0.22:8443: connect: connection refused" start-of-body= Oct 30 16:20:56.755970 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:56.755897 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-4wrf8" podUID="4f0f46f4-5cda-4d67-b6f3-47df06eb844c" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.22:8443/readyz\": dial tcp 10.132.0.22:8443: connect: connection refused" Oct 30 16:20:57.612061 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:57.612027 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-sg9tg" Oct 30 16:20:57.761032 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:57.760995 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wxvph" event={"ID":"d6db39f8-bf92-44e8-9144-51eb3b9dee8d","Type":"ContainerStarted","Data":"61fd2306d7f77a389ad825188c3848706aff205341f939704428c128b5280f14"} Oct 30 16:20:57.765103 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:57.765080 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-4wrf8" Oct 30 16:20:59.108786 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:59.108749 5052 patch_prober.go:29] interesting pod/image-registry-65d65dd574-nbb2b container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Oct 30 16:20:59.109164 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:20:59.108811 5052 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-65d65dd574-nbb2b" podUID="f79e659b-79c9-43ee-8733-c685f60e87fb" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 16:21:01.721814 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:01.721785 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-wb6mp-dt2hf"] Oct 30 16:21:01.757753 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:01.757705 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-wb6mp-dt2hf"] Oct 30 16:21:01.757931 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:01.757857 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-wb6mp-dt2hf" Oct 30 16:21:01.764483 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:01.764451 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qmvpb"] Oct 30 16:21:01.773206 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:01.773174 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c87197db-c5ed-4649-a4bc-ea4cb45ea0eb-archives-path\") pod \"periodic-gathering-wb6mp-dt2hf\" (UID: \"c87197db-c5ed-4649-a4bc-ea4cb45ea0eb\") " pod="openshift-insights/periodic-gathering-wb6mp-dt2hf" Oct 30 16:21:01.773322 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:01.773234 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c87197db-c5ed-4649-a4bc-ea4cb45ea0eb-trusted-ca-bundle\") pod \"periodic-gathering-wb6mp-dt2hf\" (UID: \"c87197db-c5ed-4649-a4bc-ea4cb45ea0eb\") " pod="openshift-insights/periodic-gathering-wb6mp-dt2hf" Oct 30 16:21:01.773439 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:01.773361 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnb9f\" (UniqueName: \"kubernetes.io/projected/c87197db-c5ed-4649-a4bc-ea4cb45ea0eb-kube-api-access-fnb9f\") pod \"periodic-gathering-wb6mp-dt2hf\" (UID: \"c87197db-c5ed-4649-a4bc-ea4cb45ea0eb\") " pod="openshift-insights/periodic-gathering-wb6mp-dt2hf" Oct 30 16:21:01.773486 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:01.773447 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c87197db-c5ed-4649-a4bc-ea4cb45ea0eb-service-ca-bundle\") pod \"periodic-gathering-wb6mp-dt2hf\" (UID: \"c87197db-c5ed-4649-a4bc-ea4cb45ea0eb\") " pod="openshift-insights/periodic-gathering-wb6mp-dt2hf" Oct 30 16:21:01.779576 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:01.779556 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-gr95f"] Oct 30 16:21:01.779757 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:01.779675 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qmvpb" Oct 30 16:21:01.796165 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:01.796084 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-gr95f" Oct 30 16:21:01.799047 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:01.799023 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-f8dddb4c-dcbgb"] Oct 30 16:21:01.800178 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:01.800158 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qmvpb"] Oct 30 16:21:01.804278 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:01.804258 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-x9846\"" Oct 30 16:21:01.804479 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:01.804354 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Oct 30 16:21:01.808459 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:01.806592 5052 patch_prober.go:29] interesting pod/image-registry-f8dddb4c-dcbgb container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Oct 30 16:21:01.808459 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:01.806637 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-f8dddb4c-dcbgb" podUID="9fafe556-9653-49df-a851-100b51bb7f47" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 16:21:01.808459 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:01.808333 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-gr95f"] Oct 30 16:21:01.818740 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:01.818710 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 30 16:21:01.818826 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:01.818761 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 30 16:21:01.862835 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:01.862803 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-db6c9d67-kdw6z"] Oct 30 16:21:01.874244 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:01.874210 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c87197db-c5ed-4649-a4bc-ea4cb45ea0eb-service-ca-bundle\") pod \"periodic-gathering-wb6mp-dt2hf\" (UID: \"c87197db-c5ed-4649-a4bc-ea4cb45ea0eb\") " pod="openshift-insights/periodic-gathering-wb6mp-dt2hf" Oct 30 16:21:01.874387 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:01.874248 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1cabe56f-84a6-42b2-b4c5-979d56a0d838-tmp\") pod \"downloads-6dd789c8b7-gr95f\" (UID: \"1cabe56f-84a6-42b2-b4c5-979d56a0d838\") " pod="openshift-console/downloads-6dd789c8b7-gr95f" Oct 30 16:21:01.874387 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:01.874282 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zk54\" (UniqueName: \"kubernetes.io/projected/1cabe56f-84a6-42b2-b4c5-979d56a0d838-kube-api-access-7zk54\") pod \"downloads-6dd789c8b7-gr95f\" (UID: \"1cabe56f-84a6-42b2-b4c5-979d56a0d838\") " pod="openshift-console/downloads-6dd789c8b7-gr95f" Oct 30 16:21:01.874387 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:01.874304 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c87197db-c5ed-4649-a4bc-ea4cb45ea0eb-archives-path\") pod \"periodic-gathering-wb6mp-dt2hf\" (UID: \"c87197db-c5ed-4649-a4bc-ea4cb45ea0eb\") " pod="openshift-insights/periodic-gathering-wb6mp-dt2hf" Oct 30 16:21:01.874387 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:01.874370 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/29041a5c-549c-47ec-ba84-a199ba0931b0-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-qmvpb\" (UID: \"29041a5c-549c-47ec-ba84-a199ba0931b0\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qmvpb" Oct 30 16:21:01.874616 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:01.874469 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c87197db-c5ed-4649-a4bc-ea4cb45ea0eb-trusted-ca-bundle\") pod \"periodic-gathering-wb6mp-dt2hf\" (UID: \"c87197db-c5ed-4649-a4bc-ea4cb45ea0eb\") " pod="openshift-insights/periodic-gathering-wb6mp-dt2hf" Oct 30 16:21:01.874616 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:01.874554 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnb9f\" (UniqueName: \"kubernetes.io/projected/c87197db-c5ed-4649-a4bc-ea4cb45ea0eb-kube-api-access-fnb9f\") pod \"periodic-gathering-wb6mp-dt2hf\" (UID: \"c87197db-c5ed-4649-a4bc-ea4cb45ea0eb\") " pod="openshift-insights/periodic-gathering-wb6mp-dt2hf" Oct 30 16:21:01.874708 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:01.874691 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c87197db-c5ed-4649-a4bc-ea4cb45ea0eb-archives-path\") pod \"periodic-gathering-wb6mp-dt2hf\" (UID: \"c87197db-c5ed-4649-a4bc-ea4cb45ea0eb\") " pod="openshift-insights/periodic-gathering-wb6mp-dt2hf" Oct 30 16:21:01.874942 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:01.874926 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c87197db-c5ed-4649-a4bc-ea4cb45ea0eb-service-ca-bundle\") pod \"periodic-gathering-wb6mp-dt2hf\" (UID: \"c87197db-c5ed-4649-a4bc-ea4cb45ea0eb\") " pod="openshift-insights/periodic-gathering-wb6mp-dt2hf" Oct 30 16:21:01.875157 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:01.875139 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c87197db-c5ed-4649-a4bc-ea4cb45ea0eb-trusted-ca-bundle\") pod \"periodic-gathering-wb6mp-dt2hf\" (UID: \"c87197db-c5ed-4649-a4bc-ea4cb45ea0eb\") " pod="openshift-insights/periodic-gathering-wb6mp-dt2hf" Oct 30 16:21:01.883374 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:01.883357 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-db6c9d67-kdw6z" Oct 30 16:21:01.899166 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:01.899144 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-db6c9d67-kdw6z"] Oct 30 16:21:01.899879 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:01.899862 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-q76qg\"" Oct 30 16:21:01.906104 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:01.906071 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnb9f\" (UniqueName: \"kubernetes.io/projected/c87197db-c5ed-4649-a4bc-ea4cb45ea0eb-kube-api-access-fnb9f\") pod \"periodic-gathering-wb6mp-dt2hf\" (UID: \"c87197db-c5ed-4649-a4bc-ea4cb45ea0eb\") " pod="openshift-insights/periodic-gathering-wb6mp-dt2hf" Oct 30 16:21:01.975632 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:01.975525 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4db30ca2-0f2f-4573-9f1c-ed4306cab9c5-ca-trust-extracted\") pod \"image-registry-db6c9d67-kdw6z\" (UID: \"4db30ca2-0f2f-4573-9f1c-ed4306cab9c5\") " pod="openshift-image-registry/image-registry-db6c9d67-kdw6z" Oct 30 16:21:01.975632 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:01.975570 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4db30ca2-0f2f-4573-9f1c-ed4306cab9c5-trusted-ca\") pod \"image-registry-db6c9d67-kdw6z\" (UID: \"4db30ca2-0f2f-4573-9f1c-ed4306cab9c5\") " pod="openshift-image-registry/image-registry-db6c9d67-kdw6z" Oct 30 16:21:01.975632 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:01.975598 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4db30ca2-0f2f-4573-9f1c-ed4306cab9c5-bound-sa-token\") pod \"image-registry-db6c9d67-kdw6z\" (UID: \"4db30ca2-0f2f-4573-9f1c-ed4306cab9c5\") " pod="openshift-image-registry/image-registry-db6c9d67-kdw6z" Oct 30 16:21:01.975889 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:01.975718 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4db30ca2-0f2f-4573-9f1c-ed4306cab9c5-registry-certificates\") pod \"image-registry-db6c9d67-kdw6z\" (UID: \"4db30ca2-0f2f-4573-9f1c-ed4306cab9c5\") " pod="openshift-image-registry/image-registry-db6c9d67-kdw6z" Oct 30 16:21:01.975889 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:01.975761 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfzsk\" (UniqueName: \"kubernetes.io/projected/4db30ca2-0f2f-4573-9f1c-ed4306cab9c5-kube-api-access-kfzsk\") pod \"image-registry-db6c9d67-kdw6z\" (UID: \"4db30ca2-0f2f-4573-9f1c-ed4306cab9c5\") " pod="openshift-image-registry/image-registry-db6c9d67-kdw6z" Oct 30 16:21:01.975889 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:01.975839 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zk54\" (UniqueName: \"kubernetes.io/projected/1cabe56f-84a6-42b2-b4c5-979d56a0d838-kube-api-access-7zk54\") pod \"downloads-6dd789c8b7-gr95f\" (UID: \"1cabe56f-84a6-42b2-b4c5-979d56a0d838\") " pod="openshift-console/downloads-6dd789c8b7-gr95f" Oct 30 16:21:01.975889 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:01.975868 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4db30ca2-0f2f-4573-9f1c-ed4306cab9c5-installation-pull-secrets\") pod \"image-registry-db6c9d67-kdw6z\" (UID: \"4db30ca2-0f2f-4573-9f1c-ed4306cab9c5\") " pod="openshift-image-registry/image-registry-db6c9d67-kdw6z" Oct 30 16:21:01.976070 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:01.975896 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/29041a5c-549c-47ec-ba84-a199ba0931b0-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-qmvpb\" (UID: \"29041a5c-549c-47ec-ba84-a199ba0931b0\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qmvpb" Oct 30 16:21:01.976070 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:01.975929 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4db30ca2-0f2f-4573-9f1c-ed4306cab9c5-registry-tls\") pod \"image-registry-db6c9d67-kdw6z\" (UID: \"4db30ca2-0f2f-4573-9f1c-ed4306cab9c5\") " pod="openshift-image-registry/image-registry-db6c9d67-kdw6z" Oct 30 16:21:01.976070 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:01.975986 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4db30ca2-0f2f-4573-9f1c-ed4306cab9c5-image-registry-private-configuration\") pod \"image-registry-db6c9d67-kdw6z\" (UID: \"4db30ca2-0f2f-4573-9f1c-ed4306cab9c5\") " pod="openshift-image-registry/image-registry-db6c9d67-kdw6z" Oct 30 16:21:01.976070 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:01.976012 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1cabe56f-84a6-42b2-b4c5-979d56a0d838-tmp\") pod \"downloads-6dd789c8b7-gr95f\" (UID: \"1cabe56f-84a6-42b2-b4c5-979d56a0d838\") " pod="openshift-console/downloads-6dd789c8b7-gr95f" Oct 30 16:21:01.976362 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:01.976345 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1cabe56f-84a6-42b2-b4c5-979d56a0d838-tmp\") pod \"downloads-6dd789c8b7-gr95f\" (UID: \"1cabe56f-84a6-42b2-b4c5-979d56a0d838\") " pod="openshift-console/downloads-6dd789c8b7-gr95f" Oct 30 16:21:01.978397 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:01.978375 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/29041a5c-549c-47ec-ba84-a199ba0931b0-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-qmvpb\" (UID: \"29041a5c-549c-47ec-ba84-a199ba0931b0\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qmvpb" Oct 30 16:21:02.000876 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:02.000855 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zk54\" (UniqueName: \"kubernetes.io/projected/1cabe56f-84a6-42b2-b4c5-979d56a0d838-kube-api-access-7zk54\") pod \"downloads-6dd789c8b7-gr95f\" (UID: \"1cabe56f-84a6-42b2-b4c5-979d56a0d838\") " pod="openshift-console/downloads-6dd789c8b7-gr95f" Oct 30 16:21:02.076718 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:02.076687 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4db30ca2-0f2f-4573-9f1c-ed4306cab9c5-image-registry-private-configuration\") pod \"image-registry-db6c9d67-kdw6z\" (UID: \"4db30ca2-0f2f-4573-9f1c-ed4306cab9c5\") " pod="openshift-image-registry/image-registry-db6c9d67-kdw6z" Oct 30 16:21:02.076884 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:02.076736 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4db30ca2-0f2f-4573-9f1c-ed4306cab9c5-ca-trust-extracted\") pod \"image-registry-db6c9d67-kdw6z\" (UID: \"4db30ca2-0f2f-4573-9f1c-ed4306cab9c5\") " pod="openshift-image-registry/image-registry-db6c9d67-kdw6z" Oct 30 16:21:02.076884 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:02.076756 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4db30ca2-0f2f-4573-9f1c-ed4306cab9c5-trusted-ca\") pod \"image-registry-db6c9d67-kdw6z\" (UID: \"4db30ca2-0f2f-4573-9f1c-ed4306cab9c5\") " pod="openshift-image-registry/image-registry-db6c9d67-kdw6z" Oct 30 16:21:02.076884 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:02.076770 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4db30ca2-0f2f-4573-9f1c-ed4306cab9c5-bound-sa-token\") pod \"image-registry-db6c9d67-kdw6z\" (UID: \"4db30ca2-0f2f-4573-9f1c-ed4306cab9c5\") " pod="openshift-image-registry/image-registry-db6c9d67-kdw6z" Oct 30 16:21:02.076884 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:02.076787 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4db30ca2-0f2f-4573-9f1c-ed4306cab9c5-registry-certificates\") pod \"image-registry-db6c9d67-kdw6z\" (UID: \"4db30ca2-0f2f-4573-9f1c-ed4306cab9c5\") " pod="openshift-image-registry/image-registry-db6c9d67-kdw6z" Oct 30 16:21:02.076884 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:02.076808 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfzsk\" (UniqueName: \"kubernetes.io/projected/4db30ca2-0f2f-4573-9f1c-ed4306cab9c5-kube-api-access-kfzsk\") pod \"image-registry-db6c9d67-kdw6z\" (UID: \"4db30ca2-0f2f-4573-9f1c-ed4306cab9c5\") " pod="openshift-image-registry/image-registry-db6c9d67-kdw6z" Oct 30 16:21:02.076884 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:02.076834 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4db30ca2-0f2f-4573-9f1c-ed4306cab9c5-installation-pull-secrets\") pod \"image-registry-db6c9d67-kdw6z\" (UID: \"4db30ca2-0f2f-4573-9f1c-ed4306cab9c5\") " pod="openshift-image-registry/image-registry-db6c9d67-kdw6z" Oct 30 16:21:02.077193 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:02.076893 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4db30ca2-0f2f-4573-9f1c-ed4306cab9c5-registry-tls\") pod \"image-registry-db6c9d67-kdw6z\" (UID: \"4db30ca2-0f2f-4573-9f1c-ed4306cab9c5\") " pod="openshift-image-registry/image-registry-db6c9d67-kdw6z" Oct 30 16:21:02.077252 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:02.077225 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4db30ca2-0f2f-4573-9f1c-ed4306cab9c5-ca-trust-extracted\") pod \"image-registry-db6c9d67-kdw6z\" (UID: \"4db30ca2-0f2f-4573-9f1c-ed4306cab9c5\") " pod="openshift-image-registry/image-registry-db6c9d67-kdw6z" Oct 30 16:21:02.077813 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:02.077775 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4db30ca2-0f2f-4573-9f1c-ed4306cab9c5-registry-certificates\") pod \"image-registry-db6c9d67-kdw6z\" (UID: \"4db30ca2-0f2f-4573-9f1c-ed4306cab9c5\") " pod="openshift-image-registry/image-registry-db6c9d67-kdw6z" Oct 30 16:21:02.077968 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:02.077952 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4db30ca2-0f2f-4573-9f1c-ed4306cab9c5-trusted-ca\") pod \"image-registry-db6c9d67-kdw6z\" (UID: \"4db30ca2-0f2f-4573-9f1c-ed4306cab9c5\") " pod="openshift-image-registry/image-registry-db6c9d67-kdw6z" Oct 30 16:21:02.079393 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:02.079365 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4db30ca2-0f2f-4573-9f1c-ed4306cab9c5-installation-pull-secrets\") pod \"image-registry-db6c9d67-kdw6z\" (UID: \"4db30ca2-0f2f-4573-9f1c-ed4306cab9c5\") " pod="openshift-image-registry/image-registry-db6c9d67-kdw6z" Oct 30 16:21:02.079639 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:02.079620 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4db30ca2-0f2f-4573-9f1c-ed4306cab9c5-image-registry-private-configuration\") pod \"image-registry-db6c9d67-kdw6z\" (UID: \"4db30ca2-0f2f-4573-9f1c-ed4306cab9c5\") " pod="openshift-image-registry/image-registry-db6c9d67-kdw6z" Oct 30 16:21:02.079713 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:02.079692 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4db30ca2-0f2f-4573-9f1c-ed4306cab9c5-registry-tls\") pod \"image-registry-db6c9d67-kdw6z\" (UID: \"4db30ca2-0f2f-4573-9f1c-ed4306cab9c5\") " pod="openshift-image-registry/image-registry-db6c9d67-kdw6z" Oct 30 16:21:02.085944 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:02.085923 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-wb6mp-dt2hf" Oct 30 16:21:02.103640 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:02.103614 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4db30ca2-0f2f-4573-9f1c-ed4306cab9c5-bound-sa-token\") pod \"image-registry-db6c9d67-kdw6z\" (UID: \"4db30ca2-0f2f-4573-9f1c-ed4306cab9c5\") " pod="openshift-image-registry/image-registry-db6c9d67-kdw6z" Oct 30 16:21:02.103747 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:02.103730 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfzsk\" (UniqueName: \"kubernetes.io/projected/4db30ca2-0f2f-4573-9f1c-ed4306cab9c5-kube-api-access-kfzsk\") pod \"image-registry-db6c9d67-kdw6z\" (UID: \"4db30ca2-0f2f-4573-9f1c-ed4306cab9c5\") " pod="openshift-image-registry/image-registry-db6c9d67-kdw6z" Oct 30 16:21:02.114621 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:02.114573 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qmvpb" Oct 30 16:21:02.135497 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:02.135440 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-gr95f" Oct 30 16:21:02.218902 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:02.218866 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-db6c9d67-kdw6z" Oct 30 16:21:02.432409 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:02.432377 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-wb6mp-dt2hf"] Oct 30 16:21:02.434956 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:21:02.434929 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc87197db_c5ed_4649_a4bc_ea4cb45ea0eb.slice/crio-e8e305c9615ff99d53603801ec6773d4ccb6cded23886cc6a9e5c415dc98a5b6 WatchSource:0}: Error finding container e8e305c9615ff99d53603801ec6773d4ccb6cded23886cc6a9e5c415dc98a5b6: Status 404 returned error can't find the container with id e8e305c9615ff99d53603801ec6773d4ccb6cded23886cc6a9e5c415dc98a5b6 Oct 30 16:21:02.471555 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:02.471505 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qmvpb"] Oct 30 16:21:02.473521 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:21:02.473496 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29041a5c_549c_47ec_ba84_a199ba0931b0.slice/crio-610c2ada697dfa47c07150301b71fab85769c718b66c3d4a7b1e3652e2281708 WatchSource:0}: Error finding container 610c2ada697dfa47c07150301b71fab85769c718b66c3d4a7b1e3652e2281708: Status 404 returned error can't find the container with id 610c2ada697dfa47c07150301b71fab85769c718b66c3d4a7b1e3652e2281708 Oct 30 16:21:02.498740 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:02.498718 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-gr95f"] Oct 30 16:21:02.501139 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:21:02.501110 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cabe56f_84a6_42b2_b4c5_979d56a0d838.slice/crio-a3d6d273a47554f85f8aa4550f91e6fd65a5bb920b01e7c17473cf57ddf2a33e WatchSource:0}: Error finding container a3d6d273a47554f85f8aa4550f91e6fd65a5bb920b01e7c17473cf57ddf2a33e: Status 404 returned error can't find the container with id a3d6d273a47554f85f8aa4550f91e6fd65a5bb920b01e7c17473cf57ddf2a33e Oct 30 16:21:02.555382 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:02.555336 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-db6c9d67-kdw6z"] Oct 30 16:21:02.557926 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:21:02.557900 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4db30ca2_0f2f_4573_9f1c_ed4306cab9c5.slice/crio-dd9d44c9c1485601de2315c8503ee8bfc3e188f39ac6abdc6398da2225aa727d WatchSource:0}: Error finding container dd9d44c9c1485601de2315c8503ee8bfc3e188f39ac6abdc6398da2225aa727d: Status 404 returned error can't find the container with id dd9d44c9c1485601de2315c8503ee8bfc3e188f39ac6abdc6398da2225aa727d Oct 30 16:21:02.732998 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:02.732925 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-qg6t6" Oct 30 16:21:02.791823 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:02.791434 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-db6c9d67-kdw6z" event={"ID":"4db30ca2-0f2f-4573-9f1c-ed4306cab9c5","Type":"ContainerStarted","Data":"ebb702b5791b8af8191db96a8cf082405bd236907ce0501b582c56fa43f27cfa"} Oct 30 16:21:02.791823 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:02.791503 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-db6c9d67-kdw6z" event={"ID":"4db30ca2-0f2f-4573-9f1c-ed4306cab9c5","Type":"ContainerStarted","Data":"dd9d44c9c1485601de2315c8503ee8bfc3e188f39ac6abdc6398da2225aa727d"} Oct 30 16:21:02.791823 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:02.791541 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-db6c9d67-kdw6z" Oct 30 16:21:02.794943 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:02.794915 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-wb6mp-dt2hf" event={"ID":"c87197db-c5ed-4649-a4bc-ea4cb45ea0eb","Type":"ContainerStarted","Data":"910a7a49e52bedf5aad9b1d44cbc589e18de6f2a77fc824c06783113455f2ad7"} Oct 30 16:21:02.795145 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:02.795120 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-wb6mp-dt2hf" event={"ID":"c87197db-c5ed-4649-a4bc-ea4cb45ea0eb","Type":"ContainerStarted","Data":"e8e305c9615ff99d53603801ec6773d4ccb6cded23886cc6a9e5c415dc98a5b6"} Oct 30 16:21:02.796576 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:02.796545 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-gr95f" event={"ID":"1cabe56f-84a6-42b2-b4c5-979d56a0d838","Type":"ContainerStarted","Data":"a3d6d273a47554f85f8aa4550f91e6fd65a5bb920b01e7c17473cf57ddf2a33e"} Oct 30 16:21:02.797967 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:02.797943 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qmvpb" event={"ID":"29041a5c-549c-47ec-ba84-a199ba0931b0","Type":"ContainerStarted","Data":"610c2ada697dfa47c07150301b71fab85769c718b66c3d4a7b1e3652e2281708"} Oct 30 16:21:02.883641 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:02.883561 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-db6c9d67-kdw6z" podStartSLOduration=1.883546227 podStartE2EDuration="1.883546227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:21:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:21:02.840150176 +0000 UTC m=+18.367485396" watchObservedRunningTime="2026-10-30 16:21:02.883546227 +0000 UTC m=+18.410881446" Oct 30 16:21:02.884323 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:02.884278 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-wb6mp-dt2hf" podStartSLOduration=1.884268233 podStartE2EDuration="1.884268233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:21:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:21:02.883256868 +0000 UTC m=+18.410592087" watchObservedRunningTime="2026-10-30 16:21:02.884268233 +0000 UTC m=+18.411603454" Oct 30 16:21:03.607459 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:03.607434 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qg6t6_3e90fa4d-66a8-4689-8f41-ca090d6697a9/dns/0.log" Oct 30 16:21:03.735356 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:03.735288 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qg6t6_3e90fa4d-66a8-4689-8f41-ca090d6697a9/kube-rbac-proxy/0.log" Oct 30 16:21:04.131425 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:04.131399 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-lghnm_9ec1217e-8623-42a6-a052-adee153936a2/dns-node-resolver/0.log" Oct 30 16:21:04.531626 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:04.531591 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-65d65dd574-nbb2b_f79e659b-79c9-43ee-8733-c685f60e87fb/registry/0.log" Oct 30 16:21:04.730422 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:04.730387 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-db6c9d67-kdw6z_4db30ca2-0f2f-4573-9f1c-ed4306cab9c5/registry/0.log" Oct 30 16:21:04.809434 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:04.809363 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qmvpb" event={"ID":"29041a5c-549c-47ec-ba84-a199ba0931b0","Type":"ContainerStarted","Data":"01d012f4d7ab42060188b170658c03b021afacf4c380eb9165c640fc5c42e55e"} Oct 30 16:21:04.809862 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:04.809562 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qmvpb" Oct 30 16:21:04.814871 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:04.814845 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qmvpb" Oct 30 16:21:04.852524 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:04.852472 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qmvpb" podStartSLOduration=2.319213776 podStartE2EDuration="3.852457223s" totalImagesPullingTime="1.533243447s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:21:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:21:04.851943374 +0000 UTC m=+20.379278594" watchObservedRunningTime="2026-10-30 16:21:04.852457223 +0000 UTC m=+20.379792444" Oct 30 16:21:04.931774 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:04.931745 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-f8dddb4c-dcbgb_9fafe556-9653-49df-a851-100b51bb7f47/registry/0.log" Oct 30 16:21:05.133273 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:05.133199 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2l5ht_d10078a0-c21c-4f18-9c5a-4915fb7c8af3/node-ca/0.log" Oct 30 16:21:05.732010 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:05.731980 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-df85cd6ff-6pd8x_b99d8a1f-000b-48ff-b4b0-7a62bc79c032/router/0.log" Oct 30 16:21:06.131026 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:06.130953 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-l4mhk_774953d5-23d3-4db1-aec9-3b78ef81017d/serve-healthcheck-canary/0.log" Oct 30 16:21:08.704361 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:08.704335 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-65d65dd574-nbb2b" Oct 30 16:21:09.967624 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:09.966680 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-nwkb6"] Oct 30 16:21:09.972520 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:09.972486 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-nwkb6" Oct 30 16:21:10.002557 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:10.002531 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 16:21:10.007434 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:10.007411 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-95j9k\"" Oct 30 16:21:10.007616 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:10.007462 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 16:21:10.035508 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:10.035484 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 16:21:10.035776 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:10.035572 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 16:21:10.045138 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:10.045108 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l46df\" (UniqueName: \"kubernetes.io/projected/ed3e3fe0-ec0c-4bce-914f-d1932f552782-kube-api-access-l46df\") pod \"node-exporter-nwkb6\" (UID: \"ed3e3fe0-ec0c-4bce-914f-d1932f552782\") " pod="openshift-monitoring/node-exporter-nwkb6" Oct 30 16:21:10.045432 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:10.045150 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ed3e3fe0-ec0c-4bce-914f-d1932f552782-root\") pod \"node-exporter-nwkb6\" (UID: \"ed3e3fe0-ec0c-4bce-914f-d1932f552782\") " pod="openshift-monitoring/node-exporter-nwkb6" Oct 30 16:21:10.045432 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:10.045173 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ed3e3fe0-ec0c-4bce-914f-d1932f552782-node-exporter-tls\") pod \"node-exporter-nwkb6\" (UID: \"ed3e3fe0-ec0c-4bce-914f-d1932f552782\") " pod="openshift-monitoring/node-exporter-nwkb6" Oct 30 16:21:10.045432 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:10.045190 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ed3e3fe0-ec0c-4bce-914f-d1932f552782-node-exporter-accelerators-collector-config\") pod \"node-exporter-nwkb6\" (UID: \"ed3e3fe0-ec0c-4bce-914f-d1932f552782\") " pod="openshift-monitoring/node-exporter-nwkb6" Oct 30 16:21:10.045432 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:10.045332 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ed3e3fe0-ec0c-4bce-914f-d1932f552782-sys\") pod \"node-exporter-nwkb6\" (UID: \"ed3e3fe0-ec0c-4bce-914f-d1932f552782\") " pod="openshift-monitoring/node-exporter-nwkb6" Oct 30 16:21:10.045432 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:10.045367 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ed3e3fe0-ec0c-4bce-914f-d1932f552782-node-exporter-wtmp\") pod \"node-exporter-nwkb6\" (UID: \"ed3e3fe0-ec0c-4bce-914f-d1932f552782\") " pod="openshift-monitoring/node-exporter-nwkb6" Oct 30 16:21:10.045432 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:10.045396 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ed3e3fe0-ec0c-4bce-914f-d1932f552782-node-exporter-textfile\") pod \"node-exporter-nwkb6\" (UID: \"ed3e3fe0-ec0c-4bce-914f-d1932f552782\") " pod="openshift-monitoring/node-exporter-nwkb6" Oct 30 16:21:10.045721 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:10.045445 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ed3e3fe0-ec0c-4bce-914f-d1932f552782-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nwkb6\" (UID: \"ed3e3fe0-ec0c-4bce-914f-d1932f552782\") " pod="openshift-monitoring/node-exporter-nwkb6" Oct 30 16:21:10.045721 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:10.045469 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ed3e3fe0-ec0c-4bce-914f-d1932f552782-metrics-client-ca\") pod \"node-exporter-nwkb6\" (UID: \"ed3e3fe0-ec0c-4bce-914f-d1932f552782\") " pod="openshift-monitoring/node-exporter-nwkb6" Oct 30 16:21:10.146515 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:10.146473 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ed3e3fe0-ec0c-4bce-914f-d1932f552782-node-exporter-wtmp\") pod \"node-exporter-nwkb6\" (UID: \"ed3e3fe0-ec0c-4bce-914f-d1932f552782\") " pod="openshift-monitoring/node-exporter-nwkb6" Oct 30 16:21:10.146515 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:10.146515 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ed3e3fe0-ec0c-4bce-914f-d1932f552782-node-exporter-textfile\") pod \"node-exporter-nwkb6\" (UID: \"ed3e3fe0-ec0c-4bce-914f-d1932f552782\") " pod="openshift-monitoring/node-exporter-nwkb6" Oct 30 16:21:10.146782 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:10.146564 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ed3e3fe0-ec0c-4bce-914f-d1932f552782-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nwkb6\" (UID: \"ed3e3fe0-ec0c-4bce-914f-d1932f552782\") " pod="openshift-monitoring/node-exporter-nwkb6" Oct 30 16:21:10.146782 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:10.146613 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ed3e3fe0-ec0c-4bce-914f-d1932f552782-metrics-client-ca\") pod \"node-exporter-nwkb6\" (UID: \"ed3e3fe0-ec0c-4bce-914f-d1932f552782\") " pod="openshift-monitoring/node-exporter-nwkb6" Oct 30 16:21:10.146782 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:10.146674 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l46df\" (UniqueName: \"kubernetes.io/projected/ed3e3fe0-ec0c-4bce-914f-d1932f552782-kube-api-access-l46df\") pod \"node-exporter-nwkb6\" (UID: \"ed3e3fe0-ec0c-4bce-914f-d1932f552782\") " pod="openshift-monitoring/node-exporter-nwkb6" Oct 30 16:21:10.146782 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:10.146693 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ed3e3fe0-ec0c-4bce-914f-d1932f552782-node-exporter-wtmp\") pod \"node-exporter-nwkb6\" (UID: \"ed3e3fe0-ec0c-4bce-914f-d1932f552782\") " pod="openshift-monitoring/node-exporter-nwkb6" Oct 30 16:21:10.146782 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:10.146712 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ed3e3fe0-ec0c-4bce-914f-d1932f552782-root\") pod \"node-exporter-nwkb6\" (UID: \"ed3e3fe0-ec0c-4bce-914f-d1932f552782\") " pod="openshift-monitoring/node-exporter-nwkb6" Oct 30 16:21:10.146782 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:10.146758 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ed3e3fe0-ec0c-4bce-914f-d1932f552782-root\") pod \"node-exporter-nwkb6\" (UID: \"ed3e3fe0-ec0c-4bce-914f-d1932f552782\") " pod="openshift-monitoring/node-exporter-nwkb6" Oct 30 16:21:10.147040 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:10.146785 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ed3e3fe0-ec0c-4bce-914f-d1932f552782-node-exporter-tls\") pod \"node-exporter-nwkb6\" (UID: \"ed3e3fe0-ec0c-4bce-914f-d1932f552782\") " pod="openshift-monitoring/node-exporter-nwkb6" Oct 30 16:21:10.147040 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:10.146817 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ed3e3fe0-ec0c-4bce-914f-d1932f552782-node-exporter-accelerators-collector-config\") pod \"node-exporter-nwkb6\" (UID: \"ed3e3fe0-ec0c-4bce-914f-d1932f552782\") " pod="openshift-monitoring/node-exporter-nwkb6" Oct 30 16:21:10.147040 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:10.146865 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ed3e3fe0-ec0c-4bce-914f-d1932f552782-sys\") pod \"node-exporter-nwkb6\" (UID: \"ed3e3fe0-ec0c-4bce-914f-d1932f552782\") " pod="openshift-monitoring/node-exporter-nwkb6" Oct 30 16:21:10.147040 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:10.146931 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ed3e3fe0-ec0c-4bce-914f-d1932f552782-sys\") pod \"node-exporter-nwkb6\" (UID: \"ed3e3fe0-ec0c-4bce-914f-d1932f552782\") " pod="openshift-monitoring/node-exporter-nwkb6" Oct 30 16:21:10.147040 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:10.146974 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ed3e3fe0-ec0c-4bce-914f-d1932f552782-node-exporter-textfile\") pod \"node-exporter-nwkb6\" (UID: \"ed3e3fe0-ec0c-4bce-914f-d1932f552782\") " pod="openshift-monitoring/node-exporter-nwkb6" Oct 30 16:21:10.147817 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:10.147791 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ed3e3fe0-ec0c-4bce-914f-d1932f552782-metrics-client-ca\") pod \"node-exporter-nwkb6\" (UID: \"ed3e3fe0-ec0c-4bce-914f-d1932f552782\") " pod="openshift-monitoring/node-exporter-nwkb6" Oct 30 16:21:10.147948 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:10.147855 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ed3e3fe0-ec0c-4bce-914f-d1932f552782-node-exporter-accelerators-collector-config\") pod \"node-exporter-nwkb6\" (UID: \"ed3e3fe0-ec0c-4bce-914f-d1932f552782\") " pod="openshift-monitoring/node-exporter-nwkb6" Oct 30 16:21:10.149468 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:10.149448 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ed3e3fe0-ec0c-4bce-914f-d1932f552782-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nwkb6\" (UID: \"ed3e3fe0-ec0c-4bce-914f-d1932f552782\") " pod="openshift-monitoring/node-exporter-nwkb6" Oct 30 16:21:10.149767 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:10.149743 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ed3e3fe0-ec0c-4bce-914f-d1932f552782-node-exporter-tls\") pod \"node-exporter-nwkb6\" (UID: \"ed3e3fe0-ec0c-4bce-914f-d1932f552782\") " pod="openshift-monitoring/node-exporter-nwkb6" Oct 30 16:21:10.172075 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:10.172045 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l46df\" (UniqueName: \"kubernetes.io/projected/ed3e3fe0-ec0c-4bce-914f-d1932f552782-kube-api-access-l46df\") pod \"node-exporter-nwkb6\" (UID: \"ed3e3fe0-ec0c-4bce-914f-d1932f552782\") " pod="openshift-monitoring/node-exporter-nwkb6" Oct 30 16:21:10.302637 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:10.302554 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-nwkb6" Oct 30 16:21:10.317088 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:21:10.317057 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded3e3fe0_ec0c_4bce_914f_d1932f552782.slice/crio-385f3d7b3c99f45446c74efd434a8130f4a08928ab2c8ed4b445ab0a6cec1e84 WatchSource:0}: Error finding container 385f3d7b3c99f45446c74efd434a8130f4a08928ab2c8ed4b445ab0a6cec1e84: Status 404 returned error can't find the container with id 385f3d7b3c99f45446c74efd434a8130f4a08928ab2c8ed4b445ab0a6cec1e84 Oct 30 16:21:10.843191 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:10.843140 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nwkb6" event={"ID":"ed3e3fe0-ec0c-4bce-914f-d1932f552782","Type":"ContainerStarted","Data":"385f3d7b3c99f45446c74efd434a8130f4a08928ab2c8ed4b445ab0a6cec1e84"} Oct 30 16:21:11.804227 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:11.804150 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-f8dddb4c-dcbgb" Oct 30 16:21:11.850473 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:11.850435 5052 generic.go:356] "Generic (PLEG): container finished" podID="ed3e3fe0-ec0c-4bce-914f-d1932f552782" containerID="5c16456378524a45b55320499c1e3b737421e9fd76cc16c1adbd59434a34776f" exitCode=0 Oct 30 16:21:11.850661 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:11.850518 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nwkb6" event={"ID":"ed3e3fe0-ec0c-4bce-914f-d1932f552782","Type":"ContainerDied","Data":"5c16456378524a45b55320499c1e3b737421e9fd76cc16c1adbd59434a34776f"} Oct 30 16:21:12.429660 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:12.429624 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-b4ljl" Oct 30 16:21:12.858447 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:12.858368 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nwkb6" event={"ID":"ed3e3fe0-ec0c-4bce-914f-d1932f552782","Type":"ContainerStarted","Data":"23485d1251c98ad2fa97d0c33c311b59526054e9e897cd79b690a55964d8cd1b"} Oct 30 16:21:12.858447 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:12.858427 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nwkb6" event={"ID":"ed3e3fe0-ec0c-4bce-914f-d1932f552782","Type":"ContainerStarted","Data":"170de4a8b3d89a9dc400fae4c8781137787809587a711a8e86fe623ff1392ade"} Oct 30 16:21:12.923316 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:12.923246 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-nwkb6" podStartSLOduration=2.914810518 podStartE2EDuration="3.923226742s" totalImagesPullingTime="932.334837ms" totalInitContainerRuntime="76.081387ms" isStatefulPod=true podCreationTimestamp="2026-10-30 16:21:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:21:12.921465022 +0000 UTC m=+28.448800242" watchObservedRunningTime="2026-10-30 16:21:12.923226742 +0000 UTC m=+28.450561964" Oct 30 16:21:15.839149 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:15.839118 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2krz6" Oct 30 16:21:18.844346 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:21:18.844305 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 16:21:18.889440 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:18.889393 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-gr95f" event={"ID":"1cabe56f-84a6-42b2-b4c5-979d56a0d838","Type":"ContainerStarted","Data":"914120030e2d14acb13ccd383ffeca3d1cc5a364018a9c4317a1baf0a95ac681"} Oct 30 16:21:18.889654 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:18.889607 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-gr95f" Oct 30 16:21:18.902241 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:18.902207 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-gr95f" Oct 30 16:21:18.967618 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:18.967545 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-gr95f" podStartSLOduration=2.301738374 podStartE2EDuration="17.967530709s" totalImagesPullingTime="15.665792335s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:21:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:21:18.952943265 +0000 UTC m=+34.480278496" watchObservedRunningTime="2026-10-30 16:21:18.967530709 +0000 UTC m=+34.494866206" Oct 30 16:21:23.808273 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:23.808244 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-db6c9d67-kdw6z" Oct 30 16:21:23.908924 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:23.908885 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-65d65dd574-nbb2b"] Oct 30 16:21:26.824226 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:26.824182 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-f8dddb4c-dcbgb" podUID="9fafe556-9653-49df-a851-100b51bb7f47" containerName="registry" containerID="cri-o://dae4370af678f34a2a0a938baad1c3c80f9df6cdecb3d708dcc52f1adb325964" gracePeriod=30 Oct 30 16:21:27.103645 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:27.103614 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-f8dddb4c-dcbgb" Oct 30 16:21:27.217675 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:27.217641 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9fafe556-9653-49df-a851-100b51bb7f47-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9fafe556-9653-49df-a851-100b51bb7f47-installation-pull-secrets\") pod \"9fafe556-9653-49df-a851-100b51bb7f47\" (UID: \"9fafe556-9653-49df-a851-100b51bb7f47\") " Oct 30 16:21:27.217876 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:27.217698 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fafe556-9653-49df-a851-100b51bb7f47-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9fafe556-9653-49df-a851-100b51bb7f47-trusted-ca\") pod \"9fafe556-9653-49df-a851-100b51bb7f47\" (UID: \"9fafe556-9653-49df-a851-100b51bb7f47\") " Oct 30 16:21:27.217876 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:27.217718 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fafe556-9653-49df-a851-100b51bb7f47-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9fafe556-9653-49df-a851-100b51bb7f47-registry-certificates\") pod \"9fafe556-9653-49df-a851-100b51bb7f47\" (UID: \"9fafe556-9653-49df-a851-100b51bb7f47\") " Oct 30 16:21:27.217876 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:27.217738 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fafe556-9653-49df-a851-100b51bb7f47-kube-api-access-httvd\" (UniqueName: \"kubernetes.io/projected/9fafe556-9653-49df-a851-100b51bb7f47-kube-api-access-httvd\") pod \"9fafe556-9653-49df-a851-100b51bb7f47\" (UID: \"9fafe556-9653-49df-a851-100b51bb7f47\") " Oct 30 16:21:27.217876 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:27.217771 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9fafe556-9653-49df-a851-100b51bb7f47-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9fafe556-9653-49df-a851-100b51bb7f47-image-registry-private-configuration\") pod \"9fafe556-9653-49df-a851-100b51bb7f47\" (UID: \"9fafe556-9653-49df-a851-100b51bb7f47\") " Oct 30 16:21:27.217876 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:27.217823 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fafe556-9653-49df-a851-100b51bb7f47-registry-tls\" (UniqueName: \"kubernetes.io/projected/9fafe556-9653-49df-a851-100b51bb7f47-registry-tls\") pod \"9fafe556-9653-49df-a851-100b51bb7f47\" (UID: \"9fafe556-9653-49df-a851-100b51bb7f47\") " Oct 30 16:21:27.217876 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:27.217841 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fafe556-9653-49df-a851-100b51bb7f47-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9fafe556-9653-49df-a851-100b51bb7f47-ca-trust-extracted\") pod \"9fafe556-9653-49df-a851-100b51bb7f47\" (UID: \"9fafe556-9653-49df-a851-100b51bb7f47\") " Oct 30 16:21:27.218214 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:27.217899 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fafe556-9653-49df-a851-100b51bb7f47-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9fafe556-9653-49df-a851-100b51bb7f47-bound-sa-token\") pod \"9fafe556-9653-49df-a851-100b51bb7f47\" (UID: \"9fafe556-9653-49df-a851-100b51bb7f47\") " Oct 30 16:21:27.218214 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:27.218190 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fafe556-9653-49df-a851-100b51bb7f47-trusted-ca" pod "9fafe556-9653-49df-a851-100b51bb7f47" (UID: "9fafe556-9653-49df-a851-100b51bb7f47"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:21:27.218326 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:27.218202 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fafe556-9653-49df-a851-100b51bb7f47-registry-certificates" pod "9fafe556-9653-49df-a851-100b51bb7f47" (UID: "9fafe556-9653-49df-a851-100b51bb7f47"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:21:27.220359 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:27.220311 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fafe556-9653-49df-a851-100b51bb7f47-image-registry-private-configuration" pod "9fafe556-9653-49df-a851-100b51bb7f47" (UID: "9fafe556-9653-49df-a851-100b51bb7f47"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:21:27.220487 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:27.220373 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fafe556-9653-49df-a851-100b51bb7f47-bound-sa-token" pod "9fafe556-9653-49df-a851-100b51bb7f47" (UID: "9fafe556-9653-49df-a851-100b51bb7f47"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:21:27.220547 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:27.220492 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fafe556-9653-49df-a851-100b51bb7f47-installation-pull-secrets" pod "9fafe556-9653-49df-a851-100b51bb7f47" (UID: "9fafe556-9653-49df-a851-100b51bb7f47"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:21:27.220624 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:27.220567 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fafe556-9653-49df-a851-100b51bb7f47-kube-api-access-httvd" pod "9fafe556-9653-49df-a851-100b51bb7f47" (UID: "9fafe556-9653-49df-a851-100b51bb7f47"). InnerVolumeSpecName "kube-api-access-httvd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:21:27.220663 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:27.220634 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fafe556-9653-49df-a851-100b51bb7f47-registry-tls" pod "9fafe556-9653-49df-a851-100b51bb7f47" (UID: "9fafe556-9653-49df-a851-100b51bb7f47"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:21:27.225477 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:27.225322 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fafe556-9653-49df-a851-100b51bb7f47-ca-trust-extracted" pod "9fafe556-9653-49df-a851-100b51bb7f47" (UID: "9fafe556-9653-49df-a851-100b51bb7f47"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:21:27.319399 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:27.319365 5052 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9fafe556-9653-49df-a851-100b51bb7f47-image-registry-private-configuration\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:21:27.319399 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:27.319393 5052 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9fafe556-9653-49df-a851-100b51bb7f47-registry-tls\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:21:27.319399 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:27.319404 5052 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9fafe556-9653-49df-a851-100b51bb7f47-ca-trust-extracted\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:21:27.319785 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:27.319413 5052 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9fafe556-9653-49df-a851-100b51bb7f47-bound-sa-token\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:21:27.319785 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:27.319421 5052 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9fafe556-9653-49df-a851-100b51bb7f47-installation-pull-secrets\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:21:27.319785 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:27.319430 5052 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9fafe556-9653-49df-a851-100b51bb7f47-trusted-ca\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:21:27.319785 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:27.319438 5052 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9fafe556-9653-49df-a851-100b51bb7f47-registry-certificates\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:21:27.319785 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:27.319446 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-httvd\" (UniqueName: \"kubernetes.io/projected/9fafe556-9653-49df-a851-100b51bb7f47-kube-api-access-httvd\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:21:27.938471 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:27.938432 5052 generic.go:356] "Generic (PLEG): container finished" podID="9fafe556-9653-49df-a851-100b51bb7f47" containerID="dae4370af678f34a2a0a938baad1c3c80f9df6cdecb3d708dcc52f1adb325964" exitCode=0 Oct 30 16:21:27.939093 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:27.938477 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-f8dddb4c-dcbgb" Oct 30 16:21:27.939093 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:27.938497 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-f8dddb4c-dcbgb" event={"ID":"9fafe556-9653-49df-a851-100b51bb7f47","Type":"ContainerDied","Data":"dae4370af678f34a2a0a938baad1c3c80f9df6cdecb3d708dcc52f1adb325964"} Oct 30 16:21:27.939093 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:27.938544 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-f8dddb4c-dcbgb" event={"ID":"9fafe556-9653-49df-a851-100b51bb7f47","Type":"ContainerDied","Data":"d524f0f7e174284fada419ee192122aa2cc17519cb2161c1e1ac90be23891592"} Oct 30 16:21:27.939093 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:27.938566 5052 scope.go:122] "RemoveContainer" containerID="dae4370af678f34a2a0a938baad1c3c80f9df6cdecb3d708dcc52f1adb325964" Oct 30 16:21:27.947362 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:27.947342 5052 scope.go:122] "RemoveContainer" containerID="dae4370af678f34a2a0a938baad1c3c80f9df6cdecb3d708dcc52f1adb325964" Oct 30 16:21:27.947673 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:21:27.947645 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dae4370af678f34a2a0a938baad1c3c80f9df6cdecb3d708dcc52f1adb325964\": container with ID starting with dae4370af678f34a2a0a938baad1c3c80f9df6cdecb3d708dcc52f1adb325964 not found: ID does not exist" containerID="dae4370af678f34a2a0a938baad1c3c80f9df6cdecb3d708dcc52f1adb325964" Oct 30 16:21:27.947778 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:27.947681 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dae4370af678f34a2a0a938baad1c3c80f9df6cdecb3d708dcc52f1adb325964"} err="failed to get container status \"dae4370af678f34a2a0a938baad1c3c80f9df6cdecb3d708dcc52f1adb325964\": rpc error: code = NotFound desc = could not find container \"dae4370af678f34a2a0a938baad1c3c80f9df6cdecb3d708dcc52f1adb325964\": container with ID starting with dae4370af678f34a2a0a938baad1c3c80f9df6cdecb3d708dcc52f1adb325964 not found: ID does not exist" Oct 30 16:21:28.015286 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:28.013571 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-f8dddb4c-dcbgb"] Oct 30 16:21:28.025253 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:28.025217 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-f8dddb4c-dcbgb"] Oct 30 16:21:28.594976 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:28.594940 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fafe556-9653-49df-a851-100b51bb7f47" path="/var/lib/kubelet/pods/9fafe556-9653-49df-a851-100b51bb7f47/volumes" Oct 30 16:21:34.147738 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:34.147703 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-cwrtz-glm8n"] Oct 30 16:21:34.148258 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:34.147954 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-cwrtz-glm8n" podUID="c471cf39-07e7-4f4d-87d5-0c6803fd54a0" containerName="insights-gathering" containerID="cri-o://61bd5ae8394445d96c3c4472ab6e08a058bc26e4a5313e8391fc320fc695b1a3" gracePeriod=30 Oct 30 16:21:34.431147 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:34.431111 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-cwrtz-glm8n" Oct 30 16:21:34.583423 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:34.583393 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c471cf39-07e7-4f4d-87d5-0c6803fd54a0-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c471cf39-07e7-4f4d-87d5-0c6803fd54a0-trusted-ca-bundle\") pod \"c471cf39-07e7-4f4d-87d5-0c6803fd54a0\" (UID: \"c471cf39-07e7-4f4d-87d5-0c6803fd54a0\") " Oct 30 16:21:34.583635 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:34.583440 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c471cf39-07e7-4f4d-87d5-0c6803fd54a0-kube-api-access-mc2zh\" (UniqueName: \"kubernetes.io/projected/c471cf39-07e7-4f4d-87d5-0c6803fd54a0-kube-api-access-mc2zh\") pod \"c471cf39-07e7-4f4d-87d5-0c6803fd54a0\" (UID: \"c471cf39-07e7-4f4d-87d5-0c6803fd54a0\") " Oct 30 16:21:34.583635 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:34.583519 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c471cf39-07e7-4f4d-87d5-0c6803fd54a0-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c471cf39-07e7-4f4d-87d5-0c6803fd54a0-service-ca-bundle\") pod \"c471cf39-07e7-4f4d-87d5-0c6803fd54a0\" (UID: \"c471cf39-07e7-4f4d-87d5-0c6803fd54a0\") " Oct 30 16:21:34.583635 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:34.583610 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c471cf39-07e7-4f4d-87d5-0c6803fd54a0-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c471cf39-07e7-4f4d-87d5-0c6803fd54a0-archives-path\") pod \"c471cf39-07e7-4f4d-87d5-0c6803fd54a0\" (UID: \"c471cf39-07e7-4f4d-87d5-0c6803fd54a0\") " Oct 30 16:21:34.583917 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:34.583887 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c471cf39-07e7-4f4d-87d5-0c6803fd54a0-trusted-ca-bundle" pod "c471cf39-07e7-4f4d-87d5-0c6803fd54a0" (UID: "c471cf39-07e7-4f4d-87d5-0c6803fd54a0"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:21:34.583985 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:34.583934 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c471cf39-07e7-4f4d-87d5-0c6803fd54a0-archives-path" pod "c471cf39-07e7-4f4d-87d5-0c6803fd54a0" (UID: "c471cf39-07e7-4f4d-87d5-0c6803fd54a0"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:21:34.583985 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:34.583964 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c471cf39-07e7-4f4d-87d5-0c6803fd54a0-service-ca-bundle" pod "c471cf39-07e7-4f4d-87d5-0c6803fd54a0" (UID: "c471cf39-07e7-4f4d-87d5-0c6803fd54a0"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:21:34.585817 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:34.585783 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c471cf39-07e7-4f4d-87d5-0c6803fd54a0-kube-api-access-mc2zh" pod "c471cf39-07e7-4f4d-87d5-0c6803fd54a0" (UID: "c471cf39-07e7-4f4d-87d5-0c6803fd54a0"). InnerVolumeSpecName "kube-api-access-mc2zh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:21:34.685011 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:34.684930 5052 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c471cf39-07e7-4f4d-87d5-0c6803fd54a0-trusted-ca-bundle\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:21:34.685011 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:34.684961 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mc2zh\" (UniqueName: \"kubernetes.io/projected/c471cf39-07e7-4f4d-87d5-0c6803fd54a0-kube-api-access-mc2zh\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:21:34.685011 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:34.684975 5052 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c471cf39-07e7-4f4d-87d5-0c6803fd54a0-service-ca-bundle\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:21:34.685011 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:34.684988 5052 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c471cf39-07e7-4f4d-87d5-0c6803fd54a0-archives-path\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:21:34.969799 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:34.969714 5052 generic.go:356] "Generic (PLEG): container finished" podID="c471cf39-07e7-4f4d-87d5-0c6803fd54a0" containerID="61bd5ae8394445d96c3c4472ab6e08a058bc26e4a5313e8391fc320fc695b1a3" exitCode=2 Oct 30 16:21:34.969799 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:34.969765 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-cwrtz-glm8n" Oct 30 16:21:34.970131 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:34.969806 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-cwrtz-glm8n" event={"ID":"c471cf39-07e7-4f4d-87d5-0c6803fd54a0","Type":"ContainerDied","Data":"61bd5ae8394445d96c3c4472ab6e08a058bc26e4a5313e8391fc320fc695b1a3"} Oct 30 16:21:34.970131 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:34.969862 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-cwrtz-glm8n" event={"ID":"c471cf39-07e7-4f4d-87d5-0c6803fd54a0","Type":"ContainerDied","Data":"146ae6117475bb6dfbc6bf6e29cc5a7c142260f8d77d5a6c93946c88f12e6888"} Oct 30 16:21:34.970131 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:34.969883 5052 scope.go:122] "RemoveContainer" containerID="61bd5ae8394445d96c3c4472ab6e08a058bc26e4a5313e8391fc320fc695b1a3" Oct 30 16:21:34.977836 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:34.977819 5052 scope.go:122] "RemoveContainer" containerID="61bd5ae8394445d96c3c4472ab6e08a058bc26e4a5313e8391fc320fc695b1a3" Oct 30 16:21:34.978090 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:21:34.978073 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61bd5ae8394445d96c3c4472ab6e08a058bc26e4a5313e8391fc320fc695b1a3\": container with ID starting with 61bd5ae8394445d96c3c4472ab6e08a058bc26e4a5313e8391fc320fc695b1a3 not found: ID does not exist" containerID="61bd5ae8394445d96c3c4472ab6e08a058bc26e4a5313e8391fc320fc695b1a3" Oct 30 16:21:34.978137 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:34.978100 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61bd5ae8394445d96c3c4472ab6e08a058bc26e4a5313e8391fc320fc695b1a3"} err="failed to get container status \"61bd5ae8394445d96c3c4472ab6e08a058bc26e4a5313e8391fc320fc695b1a3\": rpc error: code = NotFound desc = could not find container \"61bd5ae8394445d96c3c4472ab6e08a058bc26e4a5313e8391fc320fc695b1a3\": container with ID starting with 61bd5ae8394445d96c3c4472ab6e08a058bc26e4a5313e8391fc320fc695b1a3 not found: ID does not exist" Oct 30 16:21:35.023951 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:35.023917 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-cwrtz-glm8n"] Oct 30 16:21:35.037375 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:35.037346 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-cwrtz-glm8n"] Oct 30 16:21:36.595059 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:36.595020 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c471cf39-07e7-4f4d-87d5-0c6803fd54a0" path="/var/lib/kubelet/pods/c471cf39-07e7-4f4d-87d5-0c6803fd54a0/volumes" Oct 30 16:21:48.838124 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:21:48.838087 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 16:21:48.937225 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:48.937166 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-65d65dd574-nbb2b" podUID="f79e659b-79c9-43ee-8733-c685f60e87fb" containerName="registry" containerID="cri-o://f83c82fa4b1237aa8a0ce2a7bca50d3ee16e5a90efeff5f2528eed7fe6f81c65" gracePeriod=30 Oct 30 16:21:49.231541 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:49.231514 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-65d65dd574-nbb2b" Oct 30 16:21:49.305359 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:49.305330 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f79e659b-79c9-43ee-8733-c685f60e87fb-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f79e659b-79c9-43ee-8733-c685f60e87fb-image-registry-private-configuration\") pod \"f79e659b-79c9-43ee-8733-c685f60e87fb\" (UID: \"f79e659b-79c9-43ee-8733-c685f60e87fb\") " Oct 30 16:21:49.305511 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:49.305390 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f79e659b-79c9-43ee-8733-c685f60e87fb-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f79e659b-79c9-43ee-8733-c685f60e87fb-bound-sa-token\") pod \"f79e659b-79c9-43ee-8733-c685f60e87fb\" (UID: \"f79e659b-79c9-43ee-8733-c685f60e87fb\") " Oct 30 16:21:49.305511 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:49.305408 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f79e659b-79c9-43ee-8733-c685f60e87fb-kube-api-access-5qh2h\" (UniqueName: \"kubernetes.io/projected/f79e659b-79c9-43ee-8733-c685f60e87fb-kube-api-access-5qh2h\") pod \"f79e659b-79c9-43ee-8733-c685f60e87fb\" (UID: \"f79e659b-79c9-43ee-8733-c685f60e87fb\") " Oct 30 16:21:49.305611 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:49.305517 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f79e659b-79c9-43ee-8733-c685f60e87fb-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f79e659b-79c9-43ee-8733-c685f60e87fb-trusted-ca\") pod \"f79e659b-79c9-43ee-8733-c685f60e87fb\" (UID: \"f79e659b-79c9-43ee-8733-c685f60e87fb\") " Oct 30 16:21:49.305611 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:49.305568 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f79e659b-79c9-43ee-8733-c685f60e87fb-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f79e659b-79c9-43ee-8733-c685f60e87fb-registry-certificates\") pod \"f79e659b-79c9-43ee-8733-c685f60e87fb\" (UID: \"f79e659b-79c9-43ee-8733-c685f60e87fb\") " Oct 30 16:21:49.305727 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:49.305658 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f79e659b-79c9-43ee-8733-c685f60e87fb-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f79e659b-79c9-43ee-8733-c685f60e87fb-installation-pull-secrets\") pod \"f79e659b-79c9-43ee-8733-c685f60e87fb\" (UID: \"f79e659b-79c9-43ee-8733-c685f60e87fb\") " Oct 30 16:21:49.305727 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:49.305694 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f79e659b-79c9-43ee-8733-c685f60e87fb-registry-tls\" (UniqueName: \"kubernetes.io/projected/f79e659b-79c9-43ee-8733-c685f60e87fb-registry-tls\") pod \"f79e659b-79c9-43ee-8733-c685f60e87fb\" (UID: \"f79e659b-79c9-43ee-8733-c685f60e87fb\") " Oct 30 16:21:49.305830 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:49.305748 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f79e659b-79c9-43ee-8733-c685f60e87fb-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f79e659b-79c9-43ee-8733-c685f60e87fb-ca-trust-extracted\") pod \"f79e659b-79c9-43ee-8733-c685f60e87fb\" (UID: \"f79e659b-79c9-43ee-8733-c685f60e87fb\") " Oct 30 16:21:49.305987 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:49.305960 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f79e659b-79c9-43ee-8733-c685f60e87fb-registry-certificates" pod "f79e659b-79c9-43ee-8733-c685f60e87fb" (UID: "f79e659b-79c9-43ee-8733-c685f60e87fb"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:21:49.306308 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:49.306281 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f79e659b-79c9-43ee-8733-c685f60e87fb-trusted-ca" pod "f79e659b-79c9-43ee-8733-c685f60e87fb" (UID: "f79e659b-79c9-43ee-8733-c685f60e87fb"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:21:49.308622 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:49.308576 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f79e659b-79c9-43ee-8733-c685f60e87fb-kube-api-access-5qh2h" pod "f79e659b-79c9-43ee-8733-c685f60e87fb" (UID: "f79e659b-79c9-43ee-8733-c685f60e87fb"). InnerVolumeSpecName "kube-api-access-5qh2h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:21:49.308989 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:49.308962 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f79e659b-79c9-43ee-8733-c685f60e87fb-bound-sa-token" pod "f79e659b-79c9-43ee-8733-c685f60e87fb" (UID: "f79e659b-79c9-43ee-8733-c685f60e87fb"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:21:49.308989 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:49.308980 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f79e659b-79c9-43ee-8733-c685f60e87fb-installation-pull-secrets" pod "f79e659b-79c9-43ee-8733-c685f60e87fb" (UID: "f79e659b-79c9-43ee-8733-c685f60e87fb"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:21:49.309142 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:49.309028 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f79e659b-79c9-43ee-8733-c685f60e87fb-image-registry-private-configuration" pod "f79e659b-79c9-43ee-8733-c685f60e87fb" (UID: "f79e659b-79c9-43ee-8733-c685f60e87fb"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:21:49.309142 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:49.309086 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f79e659b-79c9-43ee-8733-c685f60e87fb-registry-tls" pod "f79e659b-79c9-43ee-8733-c685f60e87fb" (UID: "f79e659b-79c9-43ee-8733-c685f60e87fb"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:21:49.315068 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:49.315041 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f79e659b-79c9-43ee-8733-c685f60e87fb-ca-trust-extracted" pod "f79e659b-79c9-43ee-8733-c685f60e87fb" (UID: "f79e659b-79c9-43ee-8733-c685f60e87fb"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:21:49.406602 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:49.406554 5052 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f79e659b-79c9-43ee-8733-c685f60e87fb-ca-trust-extracted\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:21:49.406602 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:49.406601 5052 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f79e659b-79c9-43ee-8733-c685f60e87fb-image-registry-private-configuration\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:21:49.406793 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:49.406612 5052 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f79e659b-79c9-43ee-8733-c685f60e87fb-bound-sa-token\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:21:49.406793 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:49.406621 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qh2h\" (UniqueName: \"kubernetes.io/projected/f79e659b-79c9-43ee-8733-c685f60e87fb-kube-api-access-5qh2h\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:21:49.406793 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:49.406631 5052 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f79e659b-79c9-43ee-8733-c685f60e87fb-trusted-ca\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:21:49.406793 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:49.406639 5052 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f79e659b-79c9-43ee-8733-c685f60e87fb-registry-certificates\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:21:49.406793 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:49.406647 5052 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f79e659b-79c9-43ee-8733-c685f60e87fb-installation-pull-secrets\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:21:49.406793 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:49.406655 5052 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f79e659b-79c9-43ee-8733-c685f60e87fb-registry-tls\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:21:50.031868 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:50.031833 5052 generic.go:356] "Generic (PLEG): container finished" podID="f79e659b-79c9-43ee-8733-c685f60e87fb" containerID="f83c82fa4b1237aa8a0ce2a7bca50d3ee16e5a90efeff5f2528eed7fe6f81c65" exitCode=0 Oct 30 16:21:50.032320 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:50.031882 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-65d65dd574-nbb2b" Oct 30 16:21:50.032320 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:50.031914 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-65d65dd574-nbb2b" event={"ID":"f79e659b-79c9-43ee-8733-c685f60e87fb","Type":"ContainerDied","Data":"f83c82fa4b1237aa8a0ce2a7bca50d3ee16e5a90efeff5f2528eed7fe6f81c65"} Oct 30 16:21:50.032320 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:50.031965 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-65d65dd574-nbb2b" event={"ID":"f79e659b-79c9-43ee-8733-c685f60e87fb","Type":"ContainerDied","Data":"11f929fe58d7dded4d64576ed87d3de315d7bb574f1401ca7bf98ef5d4b5069a"} Oct 30 16:21:50.032320 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:50.031985 5052 scope.go:122] "RemoveContainer" containerID="f83c82fa4b1237aa8a0ce2a7bca50d3ee16e5a90efeff5f2528eed7fe6f81c65" Oct 30 16:21:50.042761 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:50.042737 5052 scope.go:122] "RemoveContainer" containerID="f83c82fa4b1237aa8a0ce2a7bca50d3ee16e5a90efeff5f2528eed7fe6f81c65" Oct 30 16:21:50.043397 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:21:50.043373 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f83c82fa4b1237aa8a0ce2a7bca50d3ee16e5a90efeff5f2528eed7fe6f81c65\": container with ID starting with f83c82fa4b1237aa8a0ce2a7bca50d3ee16e5a90efeff5f2528eed7fe6f81c65 not found: ID does not exist" containerID="f83c82fa4b1237aa8a0ce2a7bca50d3ee16e5a90efeff5f2528eed7fe6f81c65" Oct 30 16:21:50.043494 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:50.043410 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f83c82fa4b1237aa8a0ce2a7bca50d3ee16e5a90efeff5f2528eed7fe6f81c65"} err="failed to get container status \"f83c82fa4b1237aa8a0ce2a7bca50d3ee16e5a90efeff5f2528eed7fe6f81c65\": rpc error: code = NotFound desc = could not find container \"f83c82fa4b1237aa8a0ce2a7bca50d3ee16e5a90efeff5f2528eed7fe6f81c65\": container with ID starting with f83c82fa4b1237aa8a0ce2a7bca50d3ee16e5a90efeff5f2528eed7fe6f81c65 not found: ID does not exist" Oct 30 16:21:50.100804 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:50.100771 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-65d65dd574-nbb2b"] Oct 30 16:21:50.114227 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:50.114199 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-65d65dd574-nbb2b"] Oct 30 16:21:50.595924 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:21:50.595894 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f79e659b-79c9-43ee-8733-c685f60e87fb" path="/var/lib/kubelet/pods/f79e659b-79c9-43ee-8733-c685f60e87fb/volumes" Oct 30 16:22:18.838972 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:22:18.838927 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 16:22:28.184413 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:22:28.184376 5052 generic.go:356] "Generic (PLEG): container finished" podID="c87197db-c5ed-4649-a4bc-ea4cb45ea0eb" containerID="910a7a49e52bedf5aad9b1d44cbc589e18de6f2a77fc824c06783113455f2ad7" exitCode=0 Oct 30 16:22:28.184859 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:22:28.184448 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-wb6mp-dt2hf" event={"ID":"c87197db-c5ed-4649-a4bc-ea4cb45ea0eb","Type":"ContainerDied","Data":"910a7a49e52bedf5aad9b1d44cbc589e18de6f2a77fc824c06783113455f2ad7"} Oct 30 16:22:29.315925 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:22:29.315903 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-wb6mp-dt2hf" Oct 30 16:22:29.435450 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:22:29.435424 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c87197db-c5ed-4649-a4bc-ea4cb45ea0eb-kube-api-access-fnb9f\" (UniqueName: \"kubernetes.io/projected/c87197db-c5ed-4649-a4bc-ea4cb45ea0eb-kube-api-access-fnb9f\") pod \"c87197db-c5ed-4649-a4bc-ea4cb45ea0eb\" (UID: \"c87197db-c5ed-4649-a4bc-ea4cb45ea0eb\") " Oct 30 16:22:29.435624 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:22:29.435493 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c87197db-c5ed-4649-a4bc-ea4cb45ea0eb-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c87197db-c5ed-4649-a4bc-ea4cb45ea0eb-trusted-ca-bundle\") pod \"c87197db-c5ed-4649-a4bc-ea4cb45ea0eb\" (UID: \"c87197db-c5ed-4649-a4bc-ea4cb45ea0eb\") " Oct 30 16:22:29.435624 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:22:29.435523 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c87197db-c5ed-4649-a4bc-ea4cb45ea0eb-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c87197db-c5ed-4649-a4bc-ea4cb45ea0eb-archives-path\") pod \"c87197db-c5ed-4649-a4bc-ea4cb45ea0eb\" (UID: \"c87197db-c5ed-4649-a4bc-ea4cb45ea0eb\") " Oct 30 16:22:29.435624 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:22:29.435552 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c87197db-c5ed-4649-a4bc-ea4cb45ea0eb-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c87197db-c5ed-4649-a4bc-ea4cb45ea0eb-service-ca-bundle\") pod \"c87197db-c5ed-4649-a4bc-ea4cb45ea0eb\" (UID: \"c87197db-c5ed-4649-a4bc-ea4cb45ea0eb\") " Oct 30 16:22:29.435886 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:22:29.435865 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c87197db-c5ed-4649-a4bc-ea4cb45ea0eb-archives-path" pod "c87197db-c5ed-4649-a4bc-ea4cb45ea0eb" (UID: "c87197db-c5ed-4649-a4bc-ea4cb45ea0eb"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:22:29.435951 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:22:29.435929 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c87197db-c5ed-4649-a4bc-ea4cb45ea0eb-trusted-ca-bundle" pod "c87197db-c5ed-4649-a4bc-ea4cb45ea0eb" (UID: "c87197db-c5ed-4649-a4bc-ea4cb45ea0eb"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:22:29.436003 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:22:29.435960 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c87197db-c5ed-4649-a4bc-ea4cb45ea0eb-service-ca-bundle" pod "c87197db-c5ed-4649-a4bc-ea4cb45ea0eb" (UID: "c87197db-c5ed-4649-a4bc-ea4cb45ea0eb"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:22:29.437560 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:22:29.437537 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c87197db-c5ed-4649-a4bc-ea4cb45ea0eb-kube-api-access-fnb9f" pod "c87197db-c5ed-4649-a4bc-ea4cb45ea0eb" (UID: "c87197db-c5ed-4649-a4bc-ea4cb45ea0eb"). InnerVolumeSpecName "kube-api-access-fnb9f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:22:29.536260 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:22:29.536233 5052 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c87197db-c5ed-4649-a4bc-ea4cb45ea0eb-trusted-ca-bundle\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:22:29.536260 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:22:29.536256 5052 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c87197db-c5ed-4649-a4bc-ea4cb45ea0eb-archives-path\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:22:29.536397 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:22:29.536265 5052 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c87197db-c5ed-4649-a4bc-ea4cb45ea0eb-service-ca-bundle\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:22:29.536397 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:22:29.536274 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnb9f\" (UniqueName: \"kubernetes.io/projected/c87197db-c5ed-4649-a4bc-ea4cb45ea0eb-kube-api-access-fnb9f\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:22:30.193686 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:22:30.193654 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-wb6mp-dt2hf" Oct 30 16:22:30.193686 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:22:30.193668 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-wb6mp-dt2hf" event={"ID":"c87197db-c5ed-4649-a4bc-ea4cb45ea0eb","Type":"ContainerDied","Data":"e8e305c9615ff99d53603801ec6773d4ccb6cded23886cc6a9e5c415dc98a5b6"} Oct 30 16:22:30.193949 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:22:30.193717 5052 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e8e305c9615ff99d53603801ec6773d4ccb6cded23886cc6a9e5c415dc98a5b6" Oct 30 16:22:48.839194 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:22:48.839155 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 16:23:18.838898 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:23:18.838851 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 16:23:48.839373 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:23:48.839333 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 16:24:18.839170 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:24:18.839126 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 16:24:48.837649 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:24:48.837613 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 16:25:18.839077 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:25:18.839039 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 16:25:24.945233 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:25:24.945202 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-cbcd98d6d-l9znm"] Oct 30 16:25:24.945617 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:25:24.945502 5052 cpu_manager.go:414] "containerMap: removing container" podUID="9fafe556-9653-49df-a851-100b51bb7f47" containerName="registry" Oct 30 16:25:24.945617 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:25:24.945513 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fafe556-9653-49df-a851-100b51bb7f47" containerName="registry" Oct 30 16:25:24.945617 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:25:24.945526 5052 cpu_manager.go:414] "containerMap: removing container" podUID="c87197db-c5ed-4649-a4bc-ea4cb45ea0eb" containerName="insights-gathering" Oct 30 16:25:24.945617 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:25:24.945531 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c87197db-c5ed-4649-a4bc-ea4cb45ea0eb" containerName="insights-gathering" Oct 30 16:25:24.945617 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:25:24.945544 5052 cpu_manager.go:414] "containerMap: removing container" podUID="f79e659b-79c9-43ee-8733-c685f60e87fb" containerName="registry" Oct 30 16:25:24.945617 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:25:24.945550 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f79e659b-79c9-43ee-8733-c685f60e87fb" containerName="registry" Oct 30 16:25:24.945617 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:25:24.945557 5052 cpu_manager.go:414] "containerMap: removing container" podUID="c471cf39-07e7-4f4d-87d5-0c6803fd54a0" containerName="insights-gathering" Oct 30 16:25:24.945617 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:25:24.945562 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c471cf39-07e7-4f4d-87d5-0c6803fd54a0" containerName="insights-gathering" Oct 30 16:25:24.945867 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:25:24.945643 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="c87197db-c5ed-4649-a4bc-ea4cb45ea0eb" containerName="insights-gathering" Oct 30 16:25:24.945867 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:25:24.945653 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="f79e659b-79c9-43ee-8733-c685f60e87fb" containerName="registry" Oct 30 16:25:24.945867 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:25:24.945658 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="c471cf39-07e7-4f4d-87d5-0c6803fd54a0" containerName="insights-gathering" Oct 30 16:25:24.945867 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:25:24.945666 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fafe556-9653-49df-a851-100b51bb7f47" containerName="registry" Oct 30 16:25:24.948681 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:25:24.948664 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-cbcd98d6d-l9znm" Oct 30 16:25:24.970299 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:25:24.970276 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 16:25:24.972440 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:25:24.972421 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 16:25:24.980122 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:25:24.980102 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nglgh\" (UniqueName: \"kubernetes.io/projected/99a70d63-02f8-4891-be23-4d8b6891af04-kube-api-access-nglgh\") pod \"opendatahub-operator-controller-manager-cbcd98d6d-l9znm\" (UID: \"99a70d63-02f8-4891-be23-4d8b6891af04\") " pod="opendatahub/opendatahub-operator-controller-manager-cbcd98d6d-l9znm" Oct 30 16:25:24.980205 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:25:24.980137 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/99a70d63-02f8-4891-be23-4d8b6891af04-apiservice-cert\") pod \"opendatahub-operator-controller-manager-cbcd98d6d-l9znm\" (UID: \"99a70d63-02f8-4891-be23-4d8b6891af04\") " pod="opendatahub/opendatahub-operator-controller-manager-cbcd98d6d-l9znm" Oct 30 16:25:24.980205 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:25:24.980160 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/99a70d63-02f8-4891-be23-4d8b6891af04-tmp\") pod \"opendatahub-operator-controller-manager-cbcd98d6d-l9znm\" (UID: \"99a70d63-02f8-4891-be23-4d8b6891af04\") " pod="opendatahub/opendatahub-operator-controller-manager-cbcd98d6d-l9znm" Oct 30 16:25:24.980308 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:25:24.980295 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/99a70d63-02f8-4891-be23-4d8b6891af04-metrics-cert\") pod \"opendatahub-operator-controller-manager-cbcd98d6d-l9znm\" (UID: \"99a70d63-02f8-4891-be23-4d8b6891af04\") " pod="opendatahub/opendatahub-operator-controller-manager-cbcd98d6d-l9znm" Oct 30 16:25:24.980344 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:25:24.980328 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/99a70d63-02f8-4891-be23-4d8b6891af04-manifests\") pod \"opendatahub-operator-controller-manager-cbcd98d6d-l9znm\" (UID: \"99a70d63-02f8-4891-be23-4d8b6891af04\") " pod="opendatahub/opendatahub-operator-controller-manager-cbcd98d6d-l9znm" Oct 30 16:25:24.980387 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:25:24.980348 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/99a70d63-02f8-4891-be23-4d8b6891af04-webhook-cert\") pod \"opendatahub-operator-controller-manager-cbcd98d6d-l9znm\" (UID: \"99a70d63-02f8-4891-be23-4d8b6891af04\") " pod="opendatahub/opendatahub-operator-controller-manager-cbcd98d6d-l9znm" Oct 30 16:25:24.983951 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:25:24.983929 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-wc8qt\"" Oct 30 16:25:24.984903 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:25:24.984890 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 16:25:24.984964 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:25:24.984909 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-metrics-tls\"" Oct 30 16:25:24.984964 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:25:24.984930 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 16:25:24.990669 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:25:24.990645 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-cbcd98d6d-l9znm"] Oct 30 16:25:25.081446 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:25:25.081415 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/99a70d63-02f8-4891-be23-4d8b6891af04-metrics-cert\") pod \"opendatahub-operator-controller-manager-cbcd98d6d-l9znm\" (UID: \"99a70d63-02f8-4891-be23-4d8b6891af04\") " pod="opendatahub/opendatahub-operator-controller-manager-cbcd98d6d-l9znm" Oct 30 16:25:25.081651 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:25:25.081458 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/99a70d63-02f8-4891-be23-4d8b6891af04-manifests\") pod \"opendatahub-operator-controller-manager-cbcd98d6d-l9znm\" (UID: \"99a70d63-02f8-4891-be23-4d8b6891af04\") " pod="opendatahub/opendatahub-operator-controller-manager-cbcd98d6d-l9znm" Oct 30 16:25:25.081651 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:25:25.081479 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/99a70d63-02f8-4891-be23-4d8b6891af04-webhook-cert\") pod \"opendatahub-operator-controller-manager-cbcd98d6d-l9znm\" (UID: \"99a70d63-02f8-4891-be23-4d8b6891af04\") " pod="opendatahub/opendatahub-operator-controller-manager-cbcd98d6d-l9znm" Oct 30 16:25:25.081651 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:25:25.081504 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nglgh\" (UniqueName: \"kubernetes.io/projected/99a70d63-02f8-4891-be23-4d8b6891af04-kube-api-access-nglgh\") pod \"opendatahub-operator-controller-manager-cbcd98d6d-l9znm\" (UID: \"99a70d63-02f8-4891-be23-4d8b6891af04\") " pod="opendatahub/opendatahub-operator-controller-manager-cbcd98d6d-l9znm" Oct 30 16:25:25.081651 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:25:25.081526 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/99a70d63-02f8-4891-be23-4d8b6891af04-apiservice-cert\") pod \"opendatahub-operator-controller-manager-cbcd98d6d-l9znm\" (UID: \"99a70d63-02f8-4891-be23-4d8b6891af04\") " pod="opendatahub/opendatahub-operator-controller-manager-cbcd98d6d-l9znm" Oct 30 16:25:25.081651 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:25:25.081543 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/99a70d63-02f8-4891-be23-4d8b6891af04-tmp\") pod \"opendatahub-operator-controller-manager-cbcd98d6d-l9znm\" (UID: \"99a70d63-02f8-4891-be23-4d8b6891af04\") " pod="opendatahub/opendatahub-operator-controller-manager-cbcd98d6d-l9znm" Oct 30 16:25:25.081924 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:25:25.081859 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/99a70d63-02f8-4891-be23-4d8b6891af04-manifests\") pod \"opendatahub-operator-controller-manager-cbcd98d6d-l9znm\" (UID: \"99a70d63-02f8-4891-be23-4d8b6891af04\") " pod="opendatahub/opendatahub-operator-controller-manager-cbcd98d6d-l9znm" Oct 30 16:25:25.083973 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:25:25.083948 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/99a70d63-02f8-4891-be23-4d8b6891af04-tmp\") pod \"opendatahub-operator-controller-manager-cbcd98d6d-l9znm\" (UID: \"99a70d63-02f8-4891-be23-4d8b6891af04\") " pod="opendatahub/opendatahub-operator-controller-manager-cbcd98d6d-l9znm" Oct 30 16:25:25.084105 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:25:25.084046 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/99a70d63-02f8-4891-be23-4d8b6891af04-webhook-cert\") pod \"opendatahub-operator-controller-manager-cbcd98d6d-l9znm\" (UID: \"99a70d63-02f8-4891-be23-4d8b6891af04\") " pod="opendatahub/opendatahub-operator-controller-manager-cbcd98d6d-l9znm" Oct 30 16:25:25.084183 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:25:25.084172 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/99a70d63-02f8-4891-be23-4d8b6891af04-apiservice-cert\") pod \"opendatahub-operator-controller-manager-cbcd98d6d-l9znm\" (UID: \"99a70d63-02f8-4891-be23-4d8b6891af04\") " pod="opendatahub/opendatahub-operator-controller-manager-cbcd98d6d-l9znm" Oct 30 16:25:25.084258 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:25:25.084236 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/99a70d63-02f8-4891-be23-4d8b6891af04-metrics-cert\") pod \"opendatahub-operator-controller-manager-cbcd98d6d-l9znm\" (UID: \"99a70d63-02f8-4891-be23-4d8b6891af04\") " pod="opendatahub/opendatahub-operator-controller-manager-cbcd98d6d-l9znm" Oct 30 16:25:25.106194 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:25:25.106174 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nglgh\" (UniqueName: \"kubernetes.io/projected/99a70d63-02f8-4891-be23-4d8b6891af04-kube-api-access-nglgh\") pod \"opendatahub-operator-controller-manager-cbcd98d6d-l9znm\" (UID: \"99a70d63-02f8-4891-be23-4d8b6891af04\") " pod="opendatahub/opendatahub-operator-controller-manager-cbcd98d6d-l9znm" Oct 30 16:25:25.263745 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:25:25.263658 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-cbcd98d6d-l9znm" Oct 30 16:25:25.621438 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:25:25.621413 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-cbcd98d6d-l9znm"] Oct 30 16:25:25.623421 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:25:25.623392 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99a70d63_02f8_4891_be23_4d8b6891af04.slice/crio-834fad67c0885878a06c688067ec3c4cad8769e2b985c3ef6818fa7b804db793 WatchSource:0}: Error finding container 834fad67c0885878a06c688067ec3c4cad8769e2b985c3ef6818fa7b804db793: Status 404 returned error can't find the container with id 834fad67c0885878a06c688067ec3c4cad8769e2b985c3ef6818fa7b804db793 Oct 30 16:25:25.889425 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:25:25.889339 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-cbcd98d6d-l9znm" event={"ID":"99a70d63-02f8-4891-be23-4d8b6891af04","Type":"ContainerStarted","Data":"834fad67c0885878a06c688067ec3c4cad8769e2b985c3ef6818fa7b804db793"} Oct 30 16:25:28.903153 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:25:28.903119 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-cbcd98d6d-l9znm" event={"ID":"99a70d63-02f8-4891-be23-4d8b6891af04","Type":"ContainerStarted","Data":"48b6606d26a2b6a7830bd7dce7e759a313a5ad07ffa2d073841ed289696d6aac"} Oct 30 16:25:29.908826 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:25:29.908786 5052 generic.go:356] "Generic (PLEG): container finished" podID="99a70d63-02f8-4891-be23-4d8b6891af04" containerID="48b6606d26a2b6a7830bd7dce7e759a313a5ad07ffa2d073841ed289696d6aac" exitCode=0 Oct 30 16:25:29.909218 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:25:29.908870 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-cbcd98d6d-l9znm" event={"ID":"99a70d63-02f8-4891-be23-4d8b6891af04","Type":"ContainerDied","Data":"48b6606d26a2b6a7830bd7dce7e759a313a5ad07ffa2d073841ed289696d6aac"} Oct 30 16:25:30.915077 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:25:30.915044 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-cbcd98d6d-l9znm" event={"ID":"99a70d63-02f8-4891-be23-4d8b6891af04","Type":"ContainerStarted","Data":"eeee2f973bfbb752aed22f1be7de064088fc618a34e9e859a9b5baa7eb17da35"} Oct 30 16:25:30.915452 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:25:30.915142 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-cbcd98d6d-l9znm" Oct 30 16:25:31.012671 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:25:31.012604 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-cbcd98d6d-l9znm" podStartSLOduration=3.478096232 podStartE2EDuration="7.012571857s" totalImagesPullingTime="3.121773305s" totalInitContainerRuntime="412.70232ms" isStatefulPod=false podCreationTimestamp="2026-10-30 16:25:24 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:25:30.9981551 +0000 UTC m=+286.525490325" watchObservedRunningTime="2026-10-30 16:25:31.012571857 +0000 UTC m=+286.539907076" Oct 30 16:25:41.921798 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:25:41.921716 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-cbcd98d6d-l9znm" Oct 30 16:25:44.612981 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:25:44.612956 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4wrf8_4f0f46f4-5cda-4d67-b6f3-47df06eb844c/console-operator/2.log" Oct 30 16:25:44.613421 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:25:44.613038 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4wrf8_4f0f46f4-5cda-4d67-b6f3-47df06eb844c/console-operator/2.log" Oct 30 16:25:45.238976 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:25:45.238945 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-54d485b749-wgfjt"] Oct 30 16:25:45.243522 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:25:45.243501 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-54d485b749-wgfjt" Oct 30 16:25:45.274848 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:25:45.274825 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 30 16:25:45.274848 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:25:45.274825 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 30 16:25:45.275006 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:25:45.274938 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-7ptt6\"" Oct 30 16:25:45.276028 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:25:45.276005 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-54d485b749-wgfjt"] Oct 30 16:25:45.336699 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:25:45.336678 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8cae735b-c0f8-4066-9ad3-2a688079eecf-tmp\") pod \"kube-auth-proxy-54d485b749-wgfjt\" (UID: \"8cae735b-c0f8-4066-9ad3-2a688079eecf\") " pod="openshift-ingress/kube-auth-proxy-54d485b749-wgfjt" Oct 30 16:25:45.336824 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:25:45.336724 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8xrk\" (UniqueName: \"kubernetes.io/projected/8cae735b-c0f8-4066-9ad3-2a688079eecf-kube-api-access-s8xrk\") pod \"kube-auth-proxy-54d485b749-wgfjt\" (UID: \"8cae735b-c0f8-4066-9ad3-2a688079eecf\") " pod="openshift-ingress/kube-auth-proxy-54d485b749-wgfjt" Oct 30 16:25:45.336824 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:25:45.336791 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8cae735b-c0f8-4066-9ad3-2a688079eecf-tls-certs\") pod \"kube-auth-proxy-54d485b749-wgfjt\" (UID: \"8cae735b-c0f8-4066-9ad3-2a688079eecf\") " pod="openshift-ingress/kube-auth-proxy-54d485b749-wgfjt" Oct 30 16:25:45.437064 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:25:45.437036 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8xrk\" (UniqueName: \"kubernetes.io/projected/8cae735b-c0f8-4066-9ad3-2a688079eecf-kube-api-access-s8xrk\") pod \"kube-auth-proxy-54d485b749-wgfjt\" (UID: \"8cae735b-c0f8-4066-9ad3-2a688079eecf\") " pod="openshift-ingress/kube-auth-proxy-54d485b749-wgfjt" Oct 30 16:25:45.437181 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:25:45.437076 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8cae735b-c0f8-4066-9ad3-2a688079eecf-tls-certs\") pod \"kube-auth-proxy-54d485b749-wgfjt\" (UID: \"8cae735b-c0f8-4066-9ad3-2a688079eecf\") " pod="openshift-ingress/kube-auth-proxy-54d485b749-wgfjt" Oct 30 16:25:45.437181 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:25:45.437114 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8cae735b-c0f8-4066-9ad3-2a688079eecf-tmp\") pod \"kube-auth-proxy-54d485b749-wgfjt\" (UID: \"8cae735b-c0f8-4066-9ad3-2a688079eecf\") " pod="openshift-ingress/kube-auth-proxy-54d485b749-wgfjt" Oct 30 16:25:45.439489 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:25:45.439460 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8cae735b-c0f8-4066-9ad3-2a688079eecf-tmp\") pod \"kube-auth-proxy-54d485b749-wgfjt\" (UID: \"8cae735b-c0f8-4066-9ad3-2a688079eecf\") " pod="openshift-ingress/kube-auth-proxy-54d485b749-wgfjt" Oct 30 16:25:45.439605 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:25:45.439510 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8cae735b-c0f8-4066-9ad3-2a688079eecf-tls-certs\") pod \"kube-auth-proxy-54d485b749-wgfjt\" (UID: \"8cae735b-c0f8-4066-9ad3-2a688079eecf\") " pod="openshift-ingress/kube-auth-proxy-54d485b749-wgfjt" Oct 30 16:25:45.462389 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:25:45.462365 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8xrk\" (UniqueName: \"kubernetes.io/projected/8cae735b-c0f8-4066-9ad3-2a688079eecf-kube-api-access-s8xrk\") pod \"kube-auth-proxy-54d485b749-wgfjt\" (UID: \"8cae735b-c0f8-4066-9ad3-2a688079eecf\") " pod="openshift-ingress/kube-auth-proxy-54d485b749-wgfjt" Oct 30 16:25:45.555268 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:25:45.555205 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-54d485b749-wgfjt" Oct 30 16:25:45.852083 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:25:45.852060 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-54d485b749-wgfjt"] Oct 30 16:25:45.854251 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:25:45.854217 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cae735b_c0f8_4066_9ad3_2a688079eecf.slice/crio-87a5ada2e29fbe302a79c598bc04b2765493960767d7d90fe9bdaf1af26e1f24 WatchSource:0}: Error finding container 87a5ada2e29fbe302a79c598bc04b2765493960767d7d90fe9bdaf1af26e1f24: Status 404 returned error can't find the container with id 87a5ada2e29fbe302a79c598bc04b2765493960767d7d90fe9bdaf1af26e1f24 Oct 30 16:25:45.974799 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:25:45.974771 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-54d485b749-wgfjt" event={"ID":"8cae735b-c0f8-4066-9ad3-2a688079eecf","Type":"ContainerStarted","Data":"87a5ada2e29fbe302a79c598bc04b2765493960767d7d90fe9bdaf1af26e1f24"} Oct 30 16:25:46.896742 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:25:46.896708 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-gmxjm"] Oct 30 16:25:46.900179 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:25:46.900156 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gmxjm" Oct 30 16:25:46.916664 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:25:46.916642 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 16:25:46.916763 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:25:46.916642 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 16:25:46.916763 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:25:46.916644 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 16:25:46.916763 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:25:46.916685 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 16:25:46.929551 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:25:46.929528 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-4j6q4\"" Oct 30 16:25:46.930937 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:25:46.930915 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-gmxjm"] Oct 30 16:25:46.946896 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:25:46.946874 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd4baed5-76bf-4b75-a937-378c9623a0b1-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-gmxjm\" (UID: \"cd4baed5-76bf-4b75-a937-378c9623a0b1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gmxjm" Oct 30 16:25:46.947011 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:25:46.946910 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/cd4baed5-76bf-4b75-a937-378c9623a0b1-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-gmxjm\" (UID: \"cd4baed5-76bf-4b75-a937-378c9623a0b1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gmxjm" Oct 30 16:25:46.947011 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:25:46.946927 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ss94t\" (UniqueName: \"kubernetes.io/projected/cd4baed5-76bf-4b75-a937-378c9623a0b1-kube-api-access-ss94t\") pod \"istiod-openshift-gateway-7876bfcb74-gmxjm\" (UID: \"cd4baed5-76bf-4b75-a937-378c9623a0b1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gmxjm" Oct 30 16:25:46.947086 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:25:46.947051 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cd4baed5-76bf-4b75-a937-378c9623a0b1-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-gmxjm\" (UID: \"cd4baed5-76bf-4b75-a937-378c9623a0b1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gmxjm" Oct 30 16:25:46.947124 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:25:46.947110 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/cd4baed5-76bf-4b75-a937-378c9623a0b1-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-gmxjm\" (UID: \"cd4baed5-76bf-4b75-a937-378c9623a0b1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gmxjm" Oct 30 16:25:46.947157 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:25:46.947139 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/cd4baed5-76bf-4b75-a937-378c9623a0b1-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-gmxjm\" (UID: \"cd4baed5-76bf-4b75-a937-378c9623a0b1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gmxjm" Oct 30 16:25:46.947190 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:25:46.947161 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd4baed5-76bf-4b75-a937-378c9623a0b1-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-gmxjm\" (UID: \"cd4baed5-76bf-4b75-a937-378c9623a0b1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gmxjm" Oct 30 16:25:47.047961 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:25:47.047923 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd4baed5-76bf-4b75-a937-378c9623a0b1-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-gmxjm\" (UID: \"cd4baed5-76bf-4b75-a937-378c9623a0b1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gmxjm" Oct 30 16:25:47.048147 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:25:47.047977 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/cd4baed5-76bf-4b75-a937-378c9623a0b1-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-gmxjm\" (UID: \"cd4baed5-76bf-4b75-a937-378c9623a0b1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gmxjm" Oct 30 16:25:47.048147 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:25:47.047998 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ss94t\" (UniqueName: \"kubernetes.io/projected/cd4baed5-76bf-4b75-a937-378c9623a0b1-kube-api-access-ss94t\") pod \"istiod-openshift-gateway-7876bfcb74-gmxjm\" (UID: \"cd4baed5-76bf-4b75-a937-378c9623a0b1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gmxjm" Oct 30 16:25:47.048147 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:25:47.048030 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cd4baed5-76bf-4b75-a937-378c9623a0b1-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-gmxjm\" (UID: \"cd4baed5-76bf-4b75-a937-378c9623a0b1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gmxjm" Oct 30 16:25:47.048147 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:25:47.048073 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/cd4baed5-76bf-4b75-a937-378c9623a0b1-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-gmxjm\" (UID: \"cd4baed5-76bf-4b75-a937-378c9623a0b1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gmxjm" Oct 30 16:25:47.048147 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:25:47.048105 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/cd4baed5-76bf-4b75-a937-378c9623a0b1-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-gmxjm\" (UID: \"cd4baed5-76bf-4b75-a937-378c9623a0b1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gmxjm" Oct 30 16:25:47.048147 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:25:47.048137 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd4baed5-76bf-4b75-a937-378c9623a0b1-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-gmxjm\" (UID: \"cd4baed5-76bf-4b75-a937-378c9623a0b1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gmxjm" Oct 30 16:25:47.048776 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:25:47.048745 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/cd4baed5-76bf-4b75-a937-378c9623a0b1-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-gmxjm\" (UID: \"cd4baed5-76bf-4b75-a937-378c9623a0b1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gmxjm" Oct 30 16:25:47.050698 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:25:47.050670 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd4baed5-76bf-4b75-a937-378c9623a0b1-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-gmxjm\" (UID: \"cd4baed5-76bf-4b75-a937-378c9623a0b1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gmxjm" Oct 30 16:25:47.050836 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:25:47.050819 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/cd4baed5-76bf-4b75-a937-378c9623a0b1-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-gmxjm\" (UID: \"cd4baed5-76bf-4b75-a937-378c9623a0b1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gmxjm" Oct 30 16:25:47.050911 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:25:47.050893 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cd4baed5-76bf-4b75-a937-378c9623a0b1-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-gmxjm\" (UID: \"cd4baed5-76bf-4b75-a937-378c9623a0b1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gmxjm" Oct 30 16:25:47.051059 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:25:47.051043 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/cd4baed5-76bf-4b75-a937-378c9623a0b1-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-gmxjm\" (UID: \"cd4baed5-76bf-4b75-a937-378c9623a0b1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gmxjm" Oct 30 16:25:47.073640 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:25:47.073609 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ss94t\" (UniqueName: \"kubernetes.io/projected/cd4baed5-76bf-4b75-a937-378c9623a0b1-kube-api-access-ss94t\") pod \"istiod-openshift-gateway-7876bfcb74-gmxjm\" (UID: \"cd4baed5-76bf-4b75-a937-378c9623a0b1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gmxjm" Oct 30 16:25:47.073754 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:25:47.073614 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd4baed5-76bf-4b75-a937-378c9623a0b1-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-gmxjm\" (UID: \"cd4baed5-76bf-4b75-a937-378c9623a0b1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gmxjm" Oct 30 16:25:47.214467 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:25:47.214436 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gmxjm" Oct 30 16:25:47.542711 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:25:47.542684 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-gmxjm"] Oct 30 16:25:47.545320 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:25:47.545291 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd4baed5_76bf_4b75_a937_378c9623a0b1.slice/crio-c67ac6b9cc8d33fa6078e66bda00f877a4841c928536b4fbaf65150dc9c8974e WatchSource:0}: Error finding container c67ac6b9cc8d33fa6078e66bda00f877a4841c928536b4fbaf65150dc9c8974e: Status 404 returned error can't find the container with id c67ac6b9cc8d33fa6078e66bda00f877a4841c928536b4fbaf65150dc9c8974e Oct 30 16:25:47.547703 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:25:47.547682 5052 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 16:25:47.998625 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:25:47.998572 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gmxjm" event={"ID":"cd4baed5-76bf-4b75-a937-378c9623a0b1","Type":"ContainerStarted","Data":"c67ac6b9cc8d33fa6078e66bda00f877a4841c928536b4fbaf65150dc9c8974e"} Oct 30 16:25:48.843266 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:25:48.843223 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 16:25:49.005104 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:25:49.005064 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-54d485b749-wgfjt" event={"ID":"8cae735b-c0f8-4066-9ad3-2a688079eecf","Type":"ContainerStarted","Data":"2fbb1afb1e065a57a055aa684b53956bf48d901f44f15f9d728f4962096a7342"} Oct 30 16:25:49.049292 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:25:49.049241 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-54d485b749-wgfjt" podStartSLOduration=1.870031338 podStartE2EDuration="4.049228584s" totalImagesPullingTime="2.179197246s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:25:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:25:49.048018351 +0000 UTC m=+304.575353573" watchObservedRunningTime="2026-10-30 16:25:49.049228584 +0000 UTC m=+304.576563841" Oct 30 16:25:50.016995 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:25:50.016962 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:25:50.017414 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:25:50.017037 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:25:51.015090 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:25:51.015046 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gmxjm" event={"ID":"cd4baed5-76bf-4b75-a937-378c9623a0b1","Type":"ContainerStarted","Data":"91559a6238be773f5b746a9e4d03b24cf27af52c09269eacdd744f13c45a50a5"} Oct 30 16:25:51.015299 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:25:51.015250 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gmxjm" Oct 30 16:25:51.016953 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:25:51.016928 5052 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-gmxjm container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 16:25:51.017098 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:25:51.016979 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gmxjm" podUID="cd4baed5-76bf-4b75-a937-378c9623a0b1" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 16:25:51.079599 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:25:51.079533 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gmxjm" podStartSLOduration=2.611175117 podStartE2EDuration="5.07951997s" totalImagesPullingTime="2.468344853s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:25:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:25:51.077646607 +0000 UTC m=+306.604981827" watchObservedRunningTime="2026-10-30 16:25:51.07951997 +0000 UTC m=+306.606855188" Oct 30 16:25:52.020077 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:25:52.020048 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gmxjm" Oct 30 16:26:00.983987 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:26:00.983955 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-5cfb77bf8c-hv6k2"] Oct 30 16:26:00.987551 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:26:00.987534 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-hv6k2" Oct 30 16:26:01.004207 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:26:01.004179 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-hx642\"" Oct 30 16:26:01.045267 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:26:01.045242 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" Oct 30 16:26:01.050016 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:26:01.049985 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-5cfb77bf8c-hv6k2"] Oct 30 16:26:01.068342 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:26:01.068321 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dc986c17-d850-4f2b-8888-139ae5d8385c-tmp\") pod \"kserve-module-controller-manager-5cfb77bf8c-hv6k2\" (UID: \"dc986c17-d850-4f2b-8888-139ae5d8385c\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-hv6k2" Oct 30 16:26:01.068465 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:26:01.068374 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dc986c17-d850-4f2b-8888-139ae5d8385c-trusted-ca\") pod \"kserve-module-controller-manager-5cfb77bf8c-hv6k2\" (UID: \"dc986c17-d850-4f2b-8888-139ae5d8385c\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-hv6k2" Oct 30 16:26:01.068465 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:26:01.068396 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8rfb\" (UniqueName: \"kubernetes.io/projected/dc986c17-d850-4f2b-8888-139ae5d8385c-kube-api-access-r8rfb\") pod \"kserve-module-controller-manager-5cfb77bf8c-hv6k2\" (UID: \"dc986c17-d850-4f2b-8888-139ae5d8385c\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-hv6k2" Oct 30 16:26:01.068543 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:26:01.068496 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/dc986c17-d850-4f2b-8888-139ae5d8385c-manifests\") pod \"kserve-module-controller-manager-5cfb77bf8c-hv6k2\" (UID: \"dc986c17-d850-4f2b-8888-139ae5d8385c\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-hv6k2" Oct 30 16:26:01.169713 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:26:01.169687 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/dc986c17-d850-4f2b-8888-139ae5d8385c-manifests\") pod \"kserve-module-controller-manager-5cfb77bf8c-hv6k2\" (UID: \"dc986c17-d850-4f2b-8888-139ae5d8385c\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-hv6k2" Oct 30 16:26:01.169834 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:26:01.169737 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dc986c17-d850-4f2b-8888-139ae5d8385c-tmp\") pod \"kserve-module-controller-manager-5cfb77bf8c-hv6k2\" (UID: \"dc986c17-d850-4f2b-8888-139ae5d8385c\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-hv6k2" Oct 30 16:26:01.169834 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:26:01.169776 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dc986c17-d850-4f2b-8888-139ae5d8385c-trusted-ca\") pod \"kserve-module-controller-manager-5cfb77bf8c-hv6k2\" (UID: \"dc986c17-d850-4f2b-8888-139ae5d8385c\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-hv6k2" Oct 30 16:26:01.169834 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:26:01.169794 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8rfb\" (UniqueName: \"kubernetes.io/projected/dc986c17-d850-4f2b-8888-139ae5d8385c-kube-api-access-r8rfb\") pod \"kserve-module-controller-manager-5cfb77bf8c-hv6k2\" (UID: \"dc986c17-d850-4f2b-8888-139ae5d8385c\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-hv6k2" Oct 30 16:26:01.170112 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:26:01.170087 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/dc986c17-d850-4f2b-8888-139ae5d8385c-manifests\") pod \"kserve-module-controller-manager-5cfb77bf8c-hv6k2\" (UID: \"dc986c17-d850-4f2b-8888-139ae5d8385c\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-hv6k2" Oct 30 16:26:01.170213 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:26:01.170114 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dc986c17-d850-4f2b-8888-139ae5d8385c-tmp\") pod \"kserve-module-controller-manager-5cfb77bf8c-hv6k2\" (UID: \"dc986c17-d850-4f2b-8888-139ae5d8385c\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-hv6k2" Oct 30 16:26:01.170715 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:26:01.170694 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dc986c17-d850-4f2b-8888-139ae5d8385c-trusted-ca\") pod \"kserve-module-controller-manager-5cfb77bf8c-hv6k2\" (UID: \"dc986c17-d850-4f2b-8888-139ae5d8385c\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-hv6k2" Oct 30 16:26:01.194757 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:26:01.194727 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8rfb\" (UniqueName: \"kubernetes.io/projected/dc986c17-d850-4f2b-8888-139ae5d8385c-kube-api-access-r8rfb\") pod \"kserve-module-controller-manager-5cfb77bf8c-hv6k2\" (UID: \"dc986c17-d850-4f2b-8888-139ae5d8385c\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-hv6k2" Oct 30 16:26:01.299508 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:26:01.299430 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-hv6k2" Oct 30 16:26:01.611627 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:26:01.611601 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-5cfb77bf8c-hv6k2"] Oct 30 16:26:01.613721 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:26:01.613678 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc986c17_d850_4f2b_8888_139ae5d8385c.slice/crio-229d8a5c06d3a1fb579924ee95df16cf1bbee161f7bdb76840cad765f34fb1d7 WatchSource:0}: Error finding container 229d8a5c06d3a1fb579924ee95df16cf1bbee161f7bdb76840cad765f34fb1d7: Status 404 returned error can't find the container with id 229d8a5c06d3a1fb579924ee95df16cf1bbee161f7bdb76840cad765f34fb1d7 Oct 30 16:26:02.069233 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:26:02.069196 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-hv6k2" event={"ID":"dc986c17-d850-4f2b-8888-139ae5d8385c","Type":"ContainerStarted","Data":"229d8a5c06d3a1fb579924ee95df16cf1bbee161f7bdb76840cad765f34fb1d7"} Oct 30 16:26:03.075496 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:26:03.075463 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-hv6k2" event={"ID":"dc986c17-d850-4f2b-8888-139ae5d8385c","Type":"ContainerStarted","Data":"25354bba6d41338508963221abc0a1bccb90efdd615e8c0239e690af6753dd42"} Oct 30 16:26:03.075915 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:26:03.075616 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-hv6k2" Oct 30 16:26:03.135933 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:26:03.135886 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-hv6k2" podStartSLOduration=1.775823307 podStartE2EDuration="3.135871205s" totalImagesPullingTime="1.360047898s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:26:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:26:03.135115915 +0000 UTC m=+318.662451133" watchObservedRunningTime="2026-10-30 16:26:03.135871205 +0000 UTC m=+318.663206417" Oct 30 16:26:18.838698 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:26:18.838659 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 16:26:24.080319 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:26:24.080289 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-hv6k2" Oct 30 16:26:46.855433 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:26:46.855401 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gd2bm"] Oct 30 16:26:46.860561 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:26:46.860539 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gd2bm" Oct 30 16:26:46.891503 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:26:46.891473 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gd2bm"] Oct 30 16:26:46.934693 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:26:46.934657 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kg77s\" (UniqueName: \"kubernetes.io/projected/57007fdd-a5ca-4597-a8c4-901310d35117-kube-api-access-kg77s\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gd2bm\" (UID: \"57007fdd-a5ca-4597-a8c4-901310d35117\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gd2bm" Oct 30 16:26:46.934867 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:26:46.934704 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/57007fdd-a5ca-4597-a8c4-901310d35117-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gd2bm\" (UID: \"57007fdd-a5ca-4597-a8c4-901310d35117\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gd2bm" Oct 30 16:26:46.934867 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:26:46.934806 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/57007fdd-a5ca-4597-a8c4-901310d35117-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gd2bm\" (UID: \"57007fdd-a5ca-4597-a8c4-901310d35117\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gd2bm" Oct 30 16:26:46.934867 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:26:46.934859 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57007fdd-a5ca-4597-a8c4-901310d35117-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gd2bm\" (UID: \"57007fdd-a5ca-4597-a8c4-901310d35117\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gd2bm" Oct 30 16:26:46.935030 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:26:46.934898 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/57007fdd-a5ca-4597-a8c4-901310d35117-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gd2bm\" (UID: \"57007fdd-a5ca-4597-a8c4-901310d35117\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gd2bm" Oct 30 16:26:46.935030 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:26:46.934928 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/57007fdd-a5ca-4597-a8c4-901310d35117-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gd2bm\" (UID: \"57007fdd-a5ca-4597-a8c4-901310d35117\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gd2bm" Oct 30 16:26:46.935121 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:26:46.935031 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/57007fdd-a5ca-4597-a8c4-901310d35117-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gd2bm\" (UID: \"57007fdd-a5ca-4597-a8c4-901310d35117\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gd2bm" Oct 30 16:26:47.036140 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:26:47.036100 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/57007fdd-a5ca-4597-a8c4-901310d35117-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gd2bm\" (UID: \"57007fdd-a5ca-4597-a8c4-901310d35117\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gd2bm" Oct 30 16:26:47.036320 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:26:47.036165 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kg77s\" (UniqueName: \"kubernetes.io/projected/57007fdd-a5ca-4597-a8c4-901310d35117-kube-api-access-kg77s\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gd2bm\" (UID: \"57007fdd-a5ca-4597-a8c4-901310d35117\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gd2bm" Oct 30 16:26:47.036320 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:26:47.036188 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/57007fdd-a5ca-4597-a8c4-901310d35117-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gd2bm\" (UID: \"57007fdd-a5ca-4597-a8c4-901310d35117\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gd2bm" Oct 30 16:26:47.036320 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:26:47.036217 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/57007fdd-a5ca-4597-a8c4-901310d35117-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gd2bm\" (UID: \"57007fdd-a5ca-4597-a8c4-901310d35117\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gd2bm" Oct 30 16:26:47.036320 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:26:47.036237 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57007fdd-a5ca-4597-a8c4-901310d35117-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gd2bm\" (UID: \"57007fdd-a5ca-4597-a8c4-901310d35117\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gd2bm" Oct 30 16:26:47.036524 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:26:47.036412 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/57007fdd-a5ca-4597-a8c4-901310d35117-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gd2bm\" (UID: \"57007fdd-a5ca-4597-a8c4-901310d35117\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gd2bm" Oct 30 16:26:47.036524 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:26:47.036458 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/57007fdd-a5ca-4597-a8c4-901310d35117-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gd2bm\" (UID: \"57007fdd-a5ca-4597-a8c4-901310d35117\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gd2bm" Oct 30 16:26:47.037068 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:26:47.037044 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/57007fdd-a5ca-4597-a8c4-901310d35117-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gd2bm\" (UID: \"57007fdd-a5ca-4597-a8c4-901310d35117\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gd2bm" Oct 30 16:26:47.038746 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:26:47.038714 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/57007fdd-a5ca-4597-a8c4-901310d35117-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gd2bm\" (UID: \"57007fdd-a5ca-4597-a8c4-901310d35117\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gd2bm" Oct 30 16:26:47.038847 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:26:47.038784 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/57007fdd-a5ca-4597-a8c4-901310d35117-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gd2bm\" (UID: \"57007fdd-a5ca-4597-a8c4-901310d35117\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gd2bm" Oct 30 16:26:47.038887 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:26:47.038859 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/57007fdd-a5ca-4597-a8c4-901310d35117-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gd2bm\" (UID: \"57007fdd-a5ca-4597-a8c4-901310d35117\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gd2bm" Oct 30 16:26:47.039072 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:26:47.039051 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/57007fdd-a5ca-4597-a8c4-901310d35117-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gd2bm\" (UID: \"57007fdd-a5ca-4597-a8c4-901310d35117\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gd2bm" Oct 30 16:26:47.063229 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:26:47.063206 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57007fdd-a5ca-4597-a8c4-901310d35117-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gd2bm\" (UID: \"57007fdd-a5ca-4597-a8c4-901310d35117\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gd2bm" Oct 30 16:26:47.063477 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:26:47.063461 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kg77s\" (UniqueName: \"kubernetes.io/projected/57007fdd-a5ca-4597-a8c4-901310d35117-kube-api-access-kg77s\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gd2bm\" (UID: \"57007fdd-a5ca-4597-a8c4-901310d35117\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gd2bm" Oct 30 16:26:47.172515 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:26:47.172489 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gd2bm" Oct 30 16:26:47.494394 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:26:47.494317 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gd2bm"] Oct 30 16:26:47.497376 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:26:47.497350 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57007fdd_a5ca_4597_a8c4_901310d35117.slice/crio-5c8bdd5e7e763a2f6df70135cd2a259b01ec9cd1ca48022c6a12e84719028d4a WatchSource:0}: Error finding container 5c8bdd5e7e763a2f6df70135cd2a259b01ec9cd1ca48022c6a12e84719028d4a: Status 404 returned error can't find the container with id 5c8bdd5e7e763a2f6df70135cd2a259b01ec9cd1ca48022c6a12e84719028d4a Oct 30 16:26:47.499573 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:26:47.499543 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:26:47.499681 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:26:47.499628 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:26:48.278030 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:26:48.277996 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gd2bm" event={"ID":"57007fdd-a5ca-4597-a8c4-901310d35117","Type":"ContainerStarted","Data":"622a8d32838021cb176634adf4ec426bef77bb5f2d2232a9f9e420da26d0a0f3"} Oct 30 16:26:48.278478 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:26:48.278044 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gd2bm" event={"ID":"57007fdd-a5ca-4597-a8c4-901310d35117","Type":"ContainerStarted","Data":"5c8bdd5e7e763a2f6df70135cd2a259b01ec9cd1ca48022c6a12e84719028d4a"} Oct 30 16:26:48.278478 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:26:48.278172 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gd2bm" Oct 30 16:26:48.279806 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:26:48.279777 5052 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-gd2bm container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 16:26:48.279919 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:26:48.279850 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gd2bm" podUID="57007fdd-a5ca-4597-a8c4-901310d35117" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 16:26:48.326295 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:26:48.326247 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gd2bm" podStartSLOduration=2.326235662 podStartE2EDuration="2.326235662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:26:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:26:48.324742318 +0000 UTC m=+363.852077536" watchObservedRunningTime="2026-10-30 16:26:48.326235662 +0000 UTC m=+363.853570881" Oct 30 16:26:48.839150 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:26:48.839106 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 16:26:49.282632 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:26:49.282603 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gd2bm" Oct 30 16:26:49.378870 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:26:49.378833 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-gmxjm"] Oct 30 16:26:49.379085 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:26:49.379060 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gmxjm" podUID="cd4baed5-76bf-4b75-a937-378c9623a0b1" containerName="discovery" containerID="cri-o://91559a6238be773f5b746a9e4d03b24cf27af52c09269eacdd744f13c45a50a5" gracePeriod=30 Oct 30 16:26:49.622984 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:26:49.622961 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gmxjm" Oct 30 16:26:49.661473 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:26:49.661448 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd4baed5-76bf-4b75-a937-378c9623a0b1-istio-token\" (UniqueName: \"kubernetes.io/projected/cd4baed5-76bf-4b75-a937-378c9623a0b1-istio-token\") pod \"cd4baed5-76bf-4b75-a937-378c9623a0b1\" (UID: \"cd4baed5-76bf-4b75-a937-378c9623a0b1\") " Oct 30 16:26:49.661607 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:26:49.661550 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd4baed5-76bf-4b75-a937-378c9623a0b1-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cd4baed5-76bf-4b75-a937-378c9623a0b1-istio-kubeconfig\") pod \"cd4baed5-76bf-4b75-a937-378c9623a0b1\" (UID: \"cd4baed5-76bf-4b75-a937-378c9623a0b1\") " Oct 30 16:26:49.661653 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:26:49.661611 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd4baed5-76bf-4b75-a937-378c9623a0b1-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/cd4baed5-76bf-4b75-a937-378c9623a0b1-istio-csr-ca-configmap\") pod \"cd4baed5-76bf-4b75-a937-378c9623a0b1\" (UID: \"cd4baed5-76bf-4b75-a937-378c9623a0b1\") " Oct 30 16:26:49.661692 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:26:49.661649 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd4baed5-76bf-4b75-a937-378c9623a0b1-kube-api-access-ss94t\" (UniqueName: \"kubernetes.io/projected/cd4baed5-76bf-4b75-a937-378c9623a0b1-kube-api-access-ss94t\") pod \"cd4baed5-76bf-4b75-a937-378c9623a0b1\" (UID: \"cd4baed5-76bf-4b75-a937-378c9623a0b1\") " Oct 30 16:26:49.661725 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:26:49.661704 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd4baed5-76bf-4b75-a937-378c9623a0b1-cacerts\" (UniqueName: \"kubernetes.io/secret/cd4baed5-76bf-4b75-a937-378c9623a0b1-cacerts\") pod \"cd4baed5-76bf-4b75-a937-378c9623a0b1\" (UID: \"cd4baed5-76bf-4b75-a937-378c9623a0b1\") " Oct 30 16:26:49.661862 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:26:49.661836 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd4baed5-76bf-4b75-a937-378c9623a0b1-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/cd4baed5-76bf-4b75-a937-378c9623a0b1-istio-csr-dns-cert\") pod \"cd4baed5-76bf-4b75-a937-378c9623a0b1\" (UID: \"cd4baed5-76bf-4b75-a937-378c9623a0b1\") " Oct 30 16:26:49.662321 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:26:49.661896 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd4baed5-76bf-4b75-a937-378c9623a0b1-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd4baed5-76bf-4b75-a937-378c9623a0b1-local-certs\") pod \"cd4baed5-76bf-4b75-a937-378c9623a0b1\" (UID: \"cd4baed5-76bf-4b75-a937-378c9623a0b1\") " Oct 30 16:26:49.662321 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:26:49.662023 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd4baed5-76bf-4b75-a937-378c9623a0b1-istio-csr-ca-configmap" pod "cd4baed5-76bf-4b75-a937-378c9623a0b1" (UID: "cd4baed5-76bf-4b75-a937-378c9623a0b1"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:26:49.662321 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:26:49.662200 5052 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/cd4baed5-76bf-4b75-a937-378c9623a0b1-istio-csr-ca-configmap\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:26:49.663948 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:26:49.663924 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd4baed5-76bf-4b75-a937-378c9623a0b1-local-certs" pod "cd4baed5-76bf-4b75-a937-378c9623a0b1" (UID: "cd4baed5-76bf-4b75-a937-378c9623a0b1"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:26:49.664049 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:26:49.663993 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd4baed5-76bf-4b75-a937-378c9623a0b1-istio-kubeconfig" pod "cd4baed5-76bf-4b75-a937-378c9623a0b1" (UID: "cd4baed5-76bf-4b75-a937-378c9623a0b1"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:26:49.664166 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:26:49.664132 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd4baed5-76bf-4b75-a937-378c9623a0b1-cacerts" pod "cd4baed5-76bf-4b75-a937-378c9623a0b1" (UID: "cd4baed5-76bf-4b75-a937-378c9623a0b1"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:26:49.664278 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:26:49.664145 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd4baed5-76bf-4b75-a937-378c9623a0b1-istio-token" pod "cd4baed5-76bf-4b75-a937-378c9623a0b1" (UID: "cd4baed5-76bf-4b75-a937-378c9623a0b1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:26:49.664278 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:26:49.664267 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd4baed5-76bf-4b75-a937-378c9623a0b1-kube-api-access-ss94t" pod "cd4baed5-76bf-4b75-a937-378c9623a0b1" (UID: "cd4baed5-76bf-4b75-a937-378c9623a0b1"). InnerVolumeSpecName "kube-api-access-ss94t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:26:49.664392 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:26:49.664377 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd4baed5-76bf-4b75-a937-378c9623a0b1-istio-csr-dns-cert" pod "cd4baed5-76bf-4b75-a937-378c9623a0b1" (UID: "cd4baed5-76bf-4b75-a937-378c9623a0b1"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:26:49.763194 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:26:49.763165 5052 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/cd4baed5-76bf-4b75-a937-378c9623a0b1-cacerts\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:26:49.763194 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:26:49.763187 5052 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/cd4baed5-76bf-4b75-a937-378c9623a0b1-istio-csr-dns-cert\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:26:49.763194 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:26:49.763198 5052 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd4baed5-76bf-4b75-a937-378c9623a0b1-local-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:26:49.763402 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:26:49.763208 5052 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd4baed5-76bf-4b75-a937-378c9623a0b1-istio-token\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:26:49.763402 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:26:49.763217 5052 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cd4baed5-76bf-4b75-a937-378c9623a0b1-istio-kubeconfig\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:26:49.763402 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:26:49.763225 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ss94t\" (UniqueName: \"kubernetes.io/projected/cd4baed5-76bf-4b75-a937-378c9623a0b1-kube-api-access-ss94t\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:26:50.287357 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:26:50.287321 5052 generic.go:356] "Generic (PLEG): container finished" podID="cd4baed5-76bf-4b75-a937-378c9623a0b1" containerID="91559a6238be773f5b746a9e4d03b24cf27af52c09269eacdd744f13c45a50a5" exitCode=0 Oct 30 16:26:50.287877 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:26:50.287378 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gmxjm" Oct 30 16:26:50.287877 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:26:50.287407 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gmxjm" event={"ID":"cd4baed5-76bf-4b75-a937-378c9623a0b1","Type":"ContainerDied","Data":"91559a6238be773f5b746a9e4d03b24cf27af52c09269eacdd744f13c45a50a5"} Oct 30 16:26:50.287877 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:26:50.287460 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gmxjm" event={"ID":"cd4baed5-76bf-4b75-a937-378c9623a0b1","Type":"ContainerDied","Data":"c67ac6b9cc8d33fa6078e66bda00f877a4841c928536b4fbaf65150dc9c8974e"} Oct 30 16:26:50.287877 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:26:50.287481 5052 scope.go:122] "RemoveContainer" containerID="91559a6238be773f5b746a9e4d03b24cf27af52c09269eacdd744f13c45a50a5" Oct 30 16:26:50.296261 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:26:50.296247 5052 scope.go:122] "RemoveContainer" containerID="91559a6238be773f5b746a9e4d03b24cf27af52c09269eacdd744f13c45a50a5" Oct 30 16:26:50.296523 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:26:50.296504 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91559a6238be773f5b746a9e4d03b24cf27af52c09269eacdd744f13c45a50a5\": container with ID starting with 91559a6238be773f5b746a9e4d03b24cf27af52c09269eacdd744f13c45a50a5 not found: ID does not exist" containerID="91559a6238be773f5b746a9e4d03b24cf27af52c09269eacdd744f13c45a50a5" Oct 30 16:26:50.296565 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:26:50.296533 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91559a6238be773f5b746a9e4d03b24cf27af52c09269eacdd744f13c45a50a5"} err="failed to get container status \"91559a6238be773f5b746a9e4d03b24cf27af52c09269eacdd744f13c45a50a5\": rpc error: code = NotFound desc = could not find container \"91559a6238be773f5b746a9e4d03b24cf27af52c09269eacdd744f13c45a50a5\": container with ID starting with 91559a6238be773f5b746a9e4d03b24cf27af52c09269eacdd744f13c45a50a5 not found: ID does not exist" Oct 30 16:26:50.356020 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:26:50.355995 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-gmxjm"] Oct 30 16:26:50.369529 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:26:50.369508 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-gmxjm"] Oct 30 16:26:50.595854 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:26:50.595765 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd4baed5-76bf-4b75-a937-378c9623a0b1" path="/var/lib/kubelet/pods/cd4baed5-76bf-4b75-a937-378c9623a0b1/volumes" Oct 30 16:27:18.839808 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:27:18.839767 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 16:27:48.839499 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:27:48.839460 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 16:28:14.146374 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:28:14.146340 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-64ddq"] Oct 30 16:28:14.146862 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:28:14.146695 5052 cpu_manager.go:414] "containerMap: removing container" podUID="cd4baed5-76bf-4b75-a937-378c9623a0b1" containerName="discovery" Oct 30 16:28:14.146862 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:28:14.146707 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd4baed5-76bf-4b75-a937-378c9623a0b1" containerName="discovery" Oct 30 16:28:14.146862 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:28:14.146782 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd4baed5-76bf-4b75-a937-378c9623a0b1" containerName="discovery" Oct 30 16:28:14.149796 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:28:14.149780 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-64ddq" Oct 30 16:28:14.183103 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:28:14.183076 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-gwddn\"" Oct 30 16:28:14.185248 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:28:14.185229 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 16:28:14.232411 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:28:14.232362 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-64ddq"] Oct 30 16:28:14.260920 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:28:14.260891 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fp7rp\" (UniqueName: \"kubernetes.io/projected/a9daa4a2-7644-4157-aa02-b8bf5ab779b0-kube-api-access-fp7rp\") pod \"kuadrant-operator-controller-manager-6b6fb56846-64ddq\" (UID: \"a9daa4a2-7644-4157-aa02-b8bf5ab779b0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-64ddq" Oct 30 16:28:14.261044 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:28:14.260927 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a9daa4a2-7644-4157-aa02-b8bf5ab779b0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-64ddq\" (UID: \"a9daa4a2-7644-4157-aa02-b8bf5ab779b0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-64ddq" Oct 30 16:28:14.282801 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:28:14.282777 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 16:28:14.362371 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:28:14.362340 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fp7rp\" (UniqueName: \"kubernetes.io/projected/a9daa4a2-7644-4157-aa02-b8bf5ab779b0-kube-api-access-fp7rp\") pod \"kuadrant-operator-controller-manager-6b6fb56846-64ddq\" (UID: \"a9daa4a2-7644-4157-aa02-b8bf5ab779b0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-64ddq" Oct 30 16:28:14.362493 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:28:14.362376 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a9daa4a2-7644-4157-aa02-b8bf5ab779b0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-64ddq\" (UID: \"a9daa4a2-7644-4157-aa02-b8bf5ab779b0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-64ddq" Oct 30 16:28:14.362755 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:28:14.362736 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a9daa4a2-7644-4157-aa02-b8bf5ab779b0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-64ddq\" (UID: \"a9daa4a2-7644-4157-aa02-b8bf5ab779b0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-64ddq" Oct 30 16:28:14.401323 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:28:14.401300 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fp7rp\" (UniqueName: \"kubernetes.io/projected/a9daa4a2-7644-4157-aa02-b8bf5ab779b0-kube-api-access-fp7rp\") pod \"kuadrant-operator-controller-manager-6b6fb56846-64ddq\" (UID: \"a9daa4a2-7644-4157-aa02-b8bf5ab779b0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-64ddq" Oct 30 16:28:14.462203 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:28:14.462180 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-64ddq" Oct 30 16:28:14.795004 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:28:14.794980 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-64ddq"] Oct 30 16:28:14.797758 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:28:14.797720 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9daa4a2_7644_4157_aa02_b8bf5ab779b0.slice/crio-073d67501390691a5292263a7aa831e149a16903e70b802270bf41a55b7dbb58 WatchSource:0}: Error finding container 073d67501390691a5292263a7aa831e149a16903e70b802270bf41a55b7dbb58: Status 404 returned error can't find the container with id 073d67501390691a5292263a7aa831e149a16903e70b802270bf41a55b7dbb58 Oct 30 16:28:15.649289 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:28:15.649251 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-64ddq" event={"ID":"a9daa4a2-7644-4157-aa02-b8bf5ab779b0","Type":"ContainerStarted","Data":"073d67501390691a5292263a7aa831e149a16903e70b802270bf41a55b7dbb58"} Oct 30 16:28:16.165779 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:28:16.164383 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-nfbzb"] Oct 30 16:28:16.172635 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:28:16.172607 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-nfbzb" Oct 30 16:28:16.190229 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:28:16.190197 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-nfbzb"] Oct 30 16:28:16.202246 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:28:16.202220 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-zkl76\"" Oct 30 16:28:16.277761 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:28:16.277725 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4clkj\" (UniqueName: \"kubernetes.io/projected/96951616-caa9-42dd-8ffc-ff0cea3d1505-kube-api-access-4clkj\") pod \"limitador-operator-controller-manager-856dd584c5-nfbzb\" (UID: \"96951616-caa9-42dd-8ffc-ff0cea3d1505\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-nfbzb" Oct 30 16:28:16.378392 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:28:16.378356 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4clkj\" (UniqueName: \"kubernetes.io/projected/96951616-caa9-42dd-8ffc-ff0cea3d1505-kube-api-access-4clkj\") pod \"limitador-operator-controller-manager-856dd584c5-nfbzb\" (UID: \"96951616-caa9-42dd-8ffc-ff0cea3d1505\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-nfbzb" Oct 30 16:28:16.413758 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:28:16.413729 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4clkj\" (UniqueName: \"kubernetes.io/projected/96951616-caa9-42dd-8ffc-ff0cea3d1505-kube-api-access-4clkj\") pod \"limitador-operator-controller-manager-856dd584c5-nfbzb\" (UID: \"96951616-caa9-42dd-8ffc-ff0cea3d1505\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-nfbzb" Oct 30 16:28:16.491540 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:28:16.491439 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-nfbzb" Oct 30 16:28:16.814388 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:28:16.814313 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-nfbzb"] Oct 30 16:28:16.818761 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:28:16.818720 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96951616_caa9_42dd_8ffc_ff0cea3d1505.slice/crio-4f230d05d43ae7af58eeb439a9767eb5e6e8f0f546ec4551ac2592afdc431e3c WatchSource:0}: Error finding container 4f230d05d43ae7af58eeb439a9767eb5e6e8f0f546ec4551ac2592afdc431e3c: Status 404 returned error can't find the container with id 4f230d05d43ae7af58eeb439a9767eb5e6e8f0f546ec4551ac2592afdc431e3c Oct 30 16:28:17.661603 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:28:17.661551 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-nfbzb" event={"ID":"96951616-caa9-42dd-8ffc-ff0cea3d1505","Type":"ContainerStarted","Data":"4f230d05d43ae7af58eeb439a9767eb5e6e8f0f546ec4551ac2592afdc431e3c"} Oct 30 16:28:18.667646 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:28:18.667610 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-nfbzb" event={"ID":"96951616-caa9-42dd-8ffc-ff0cea3d1505","Type":"ContainerStarted","Data":"cc54ff70c7c5390be2409e6e9c352fab80127c1bcd2ccd533af7d613771e33ec"} Oct 30 16:28:18.668012 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:28:18.667757 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-nfbzb" Oct 30 16:28:18.716375 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:28:18.716283 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-nfbzb" podStartSLOduration=1.006214959 podStartE2EDuration="2.716269853s" totalImagesPullingTime="1.710054894s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:28:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:28:18.715129983 +0000 UTC m=+454.242465201" watchObservedRunningTime="2026-10-30 16:28:18.716269853 +0000 UTC m=+454.243605064" Oct 30 16:28:18.838440 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:28:18.838356 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 16:28:19.674029 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:28:19.673994 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-64ddq" event={"ID":"a9daa4a2-7644-4157-aa02-b8bf5ab779b0","Type":"ContainerStarted","Data":"1121b0fd0371aa2a2a47652712b8ad63b66204df54ae2001fb54189bf6029200"} Oct 30 16:28:19.674401 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:28:19.674253 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-64ddq" Oct 30 16:28:19.749284 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:28:19.749234 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-64ddq" podStartSLOduration=1.8536004830000001 podStartE2EDuration="5.749218115s" totalImagesPullingTime="3.895617632s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:28:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:28:19.734639651 +0000 UTC m=+455.261974869" watchObservedRunningTime="2026-10-30 16:28:19.749218115 +0000 UTC m=+455.276553333" Oct 30 16:28:29.677003 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:28:29.676974 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-nfbzb" Oct 30 16:28:30.681312 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:28:30.681283 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-64ddq" Oct 30 16:28:34.589640 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:28:34.589597 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-64ddq"] Oct 30 16:28:34.593903 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:28:34.593861 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-64ddq" podUID="a9daa4a2-7644-4157-aa02-b8bf5ab779b0" containerName="manager" containerID="cri-o://1121b0fd0371aa2a2a47652712b8ad63b66204df54ae2001fb54189bf6029200" gracePeriod=2 Oct 30 16:28:34.598499 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:28:34.598473 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-64ddq"] Oct 30 16:28:34.606837 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:28:34.606802 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-vmzr9"] Oct 30 16:28:34.607144 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:28:34.607130 5052 cpu_manager.go:414] "containerMap: removing container" podUID="a9daa4a2-7644-4157-aa02-b8bf5ab779b0" containerName="manager" Oct 30 16:28:34.607144 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:28:34.607143 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9daa4a2-7644-4157-aa02-b8bf5ab779b0" containerName="manager" Oct 30 16:28:34.607288 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:28:34.607221 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9daa4a2-7644-4157-aa02-b8bf5ab779b0" containerName="manager" Oct 30 16:28:34.608882 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:28:34.608853 5052 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-64ddq\" is forbidden: User \"system:node:ip-10-0-131-136.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-136.ec2.internal' and this object" podUID="a9daa4a2-7644-4157-aa02-b8bf5ab779b0" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-64ddq" Oct 30 16:28:34.611411 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:28:34.611389 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-vmzr9" Oct 30 16:28:34.632786 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:28:34.632752 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ef0508a7-f3e9-4bb6-8353-f0c6b7e96640-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-vmzr9\" (UID: \"ef0508a7-f3e9-4bb6-8353-f0c6b7e96640\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-vmzr9" Oct 30 16:28:34.633034 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:28:34.632981 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9r2sw\" (UniqueName: \"kubernetes.io/projected/ef0508a7-f3e9-4bb6-8353-f0c6b7e96640-kube-api-access-9r2sw\") pod \"kuadrant-operator-controller-manager-6b6fb56846-vmzr9\" (UID: \"ef0508a7-f3e9-4bb6-8353-f0c6b7e96640\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-vmzr9" Oct 30 16:28:34.637558 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:28:34.637526 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-vmzr9"] Oct 30 16:28:34.642711 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:28:34.642686 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-nfbzb"] Oct 30 16:28:34.642928 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:28:34.642893 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-nfbzb" podUID="96951616-caa9-42dd-8ffc-ff0cea3d1505" containerName="manager" containerID="cri-o://cc54ff70c7c5390be2409e6e9c352fab80127c1bcd2ccd533af7d613771e33ec" gracePeriod=2 Oct 30 16:28:34.652991 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:28:34.652954 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-nfbzb"] Oct 30 16:28:34.677245 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:28:34.677220 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-bh7c6"] Oct 30 16:28:34.677755 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:28:34.677737 5052 cpu_manager.go:414] "containerMap: removing container" podUID="96951616-caa9-42dd-8ffc-ff0cea3d1505" containerName="manager" Oct 30 16:28:34.677834 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:28:34.677756 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96951616-caa9-42dd-8ffc-ff0cea3d1505" containerName="manager" Oct 30 16:28:34.677904 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:28:34.677893 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="96951616-caa9-42dd-8ffc-ff0cea3d1505" containerName="manager" Oct 30 16:28:34.682705 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:28:34.682668 5052 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-64ddq\" is forbidden: User \"system:node:ip-10-0-131-136.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-136.ec2.internal' and this object" podUID="a9daa4a2-7644-4157-aa02-b8bf5ab779b0" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-64ddq" Oct 30 16:28:34.683893 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:28:34.683878 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-bh7c6" Oct 30 16:28:34.697807 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:28:34.697777 5052 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-nfbzb\" is forbidden: User \"system:node:ip-10-0-131-136.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-136.ec2.internal' and this object" podUID="96951616-caa9-42dd-8ffc-ff0cea3d1505" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-nfbzb" Oct 30 16:28:34.705356 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:28:34.705329 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-bh7c6"] Oct 30 16:28:34.712979 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:28:34.712950 5052 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-64ddq\" is forbidden: User \"system:node:ip-10-0-131-136.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-136.ec2.internal' and this object" podUID="a9daa4a2-7644-4157-aa02-b8bf5ab779b0" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-64ddq" Oct 30 16:28:34.728383 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:28:34.728347 5052 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-nfbzb\" is forbidden: User \"system:node:ip-10-0-131-136.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-136.ec2.internal' and this object" podUID="96951616-caa9-42dd-8ffc-ff0cea3d1505" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-nfbzb" Oct 30 16:28:34.733720 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:28:34.733694 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9r2sw\" (UniqueName: \"kubernetes.io/projected/ef0508a7-f3e9-4bb6-8353-f0c6b7e96640-kube-api-access-9r2sw\") pod \"kuadrant-operator-controller-manager-6b6fb56846-vmzr9\" (UID: \"ef0508a7-f3e9-4bb6-8353-f0c6b7e96640\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-vmzr9" Oct 30 16:28:34.733828 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:28:34.733774 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqmzf\" (UniqueName: \"kubernetes.io/projected/8c01f5e2-a20b-45c3-a6c6-4b62b02e88cd-kube-api-access-tqmzf\") pod \"limitador-operator-controller-manager-856dd584c5-bh7c6\" (UID: \"8c01f5e2-a20b-45c3-a6c6-4b62b02e88cd\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-bh7c6" Oct 30 16:28:34.733828 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:28:34.733797 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ef0508a7-f3e9-4bb6-8353-f0c6b7e96640-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-vmzr9\" (UID: \"ef0508a7-f3e9-4bb6-8353-f0c6b7e96640\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-vmzr9" Oct 30 16:28:34.734313 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:28:34.734292 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ef0508a7-f3e9-4bb6-8353-f0c6b7e96640-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-vmzr9\" (UID: \"ef0508a7-f3e9-4bb6-8353-f0c6b7e96640\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-vmzr9" Oct 30 16:28:34.740168 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:28:34.740144 5052 generic.go:356] "Generic (PLEG): container finished" podID="a9daa4a2-7644-4157-aa02-b8bf5ab779b0" containerID="1121b0fd0371aa2a2a47652712b8ad63b66204df54ae2001fb54189bf6029200" exitCode=0 Oct 30 16:28:34.743387 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:28:34.743365 5052 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-64ddq\" is forbidden: User \"system:node:ip-10-0-131-136.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-136.ec2.internal' and this object" podUID="a9daa4a2-7644-4157-aa02-b8bf5ab779b0" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-64ddq" Oct 30 16:28:34.762026 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:28:34.762002 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9r2sw\" (UniqueName: \"kubernetes.io/projected/ef0508a7-f3e9-4bb6-8353-f0c6b7e96640-kube-api-access-9r2sw\") pod \"kuadrant-operator-controller-manager-6b6fb56846-vmzr9\" (UID: \"ef0508a7-f3e9-4bb6-8353-f0c6b7e96640\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-vmzr9" Oct 30 16:28:34.834515 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:28:34.834484 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqmzf\" (UniqueName: \"kubernetes.io/projected/8c01f5e2-a20b-45c3-a6c6-4b62b02e88cd-kube-api-access-tqmzf\") pod \"limitador-operator-controller-manager-856dd584c5-bh7c6\" (UID: \"8c01f5e2-a20b-45c3-a6c6-4b62b02e88cd\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-bh7c6" Oct 30 16:28:34.860220 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:28:34.860144 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqmzf\" (UniqueName: \"kubernetes.io/projected/8c01f5e2-a20b-45c3-a6c6-4b62b02e88cd-kube-api-access-tqmzf\") pod \"limitador-operator-controller-manager-856dd584c5-bh7c6\" (UID: \"8c01f5e2-a20b-45c3-a6c6-4b62b02e88cd\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-bh7c6" Oct 30 16:28:34.885002 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:28:34.884982 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-64ddq" Oct 30 16:28:34.888563 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:28:34.888544 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-nfbzb" Oct 30 16:28:34.900236 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:28:34.900206 5052 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-64ddq\" is forbidden: User \"system:node:ip-10-0-131-136.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-136.ec2.internal' and this object" podUID="a9daa4a2-7644-4157-aa02-b8bf5ab779b0" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-64ddq" Oct 30 16:28:34.915491 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:28:34.915464 5052 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-nfbzb\" is forbidden: User \"system:node:ip-10-0-131-136.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-136.ec2.internal' and this object" podUID="96951616-caa9-42dd-8ffc-ff0cea3d1505" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-nfbzb" Oct 30 16:28:34.930447 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:28:34.930424 5052 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-nfbzb\" is forbidden: User \"system:node:ip-10-0-131-136.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-136.ec2.internal' and this object" podUID="96951616-caa9-42dd-8ffc-ff0cea3d1505" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-nfbzb" Oct 30 16:28:34.935728 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:28:34.935704 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9daa4a2-7644-4157-aa02-b8bf5ab779b0-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a9daa4a2-7644-4157-aa02-b8bf5ab779b0-extensions-socket-volume\") pod \"a9daa4a2-7644-4157-aa02-b8bf5ab779b0\" (UID: \"a9daa4a2-7644-4157-aa02-b8bf5ab779b0\") " Oct 30 16:28:34.935817 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:28:34.935749 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9daa4a2-7644-4157-aa02-b8bf5ab779b0-kube-api-access-fp7rp\" (UniqueName: \"kubernetes.io/projected/a9daa4a2-7644-4157-aa02-b8bf5ab779b0-kube-api-access-fp7rp\") pod \"a9daa4a2-7644-4157-aa02-b8bf5ab779b0\" (UID: \"a9daa4a2-7644-4157-aa02-b8bf5ab779b0\") " Oct 30 16:28:34.935817 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:28:34.935782 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96951616-caa9-42dd-8ffc-ff0cea3d1505-kube-api-access-4clkj\" (UniqueName: \"kubernetes.io/projected/96951616-caa9-42dd-8ffc-ff0cea3d1505-kube-api-access-4clkj\") pod \"96951616-caa9-42dd-8ffc-ff0cea3d1505\" (UID: \"96951616-caa9-42dd-8ffc-ff0cea3d1505\") " Oct 30 16:28:34.936249 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:28:34.936222 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9daa4a2-7644-4157-aa02-b8bf5ab779b0-extensions-socket-volume" pod "a9daa4a2-7644-4157-aa02-b8bf5ab779b0" (UID: "a9daa4a2-7644-4157-aa02-b8bf5ab779b0"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:28:34.937935 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:28:34.937912 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96951616-caa9-42dd-8ffc-ff0cea3d1505-kube-api-access-4clkj" pod "96951616-caa9-42dd-8ffc-ff0cea3d1505" (UID: "96951616-caa9-42dd-8ffc-ff0cea3d1505"). InnerVolumeSpecName "kube-api-access-4clkj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:28:34.938001 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:28:34.937943 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9daa4a2-7644-4157-aa02-b8bf5ab779b0-kube-api-access-fp7rp" pod "a9daa4a2-7644-4157-aa02-b8bf5ab779b0" (UID: "a9daa4a2-7644-4157-aa02-b8bf5ab779b0"). InnerVolumeSpecName "kube-api-access-fp7rp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:28:34.945383 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:28:34.945360 5052 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-64ddq\" is forbidden: User \"system:node:ip-10-0-131-136.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-136.ec2.internal' and this object" podUID="a9daa4a2-7644-4157-aa02-b8bf5ab779b0" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-64ddq" Oct 30 16:28:35.037244 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:28:35.037213 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fp7rp\" (UniqueName: \"kubernetes.io/projected/a9daa4a2-7644-4157-aa02-b8bf5ab779b0-kube-api-access-fp7rp\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:28:35.037244 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:28:35.037242 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4clkj\" (UniqueName: \"kubernetes.io/projected/96951616-caa9-42dd-8ffc-ff0cea3d1505-kube-api-access-4clkj\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:28:35.037422 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:28:35.037252 5052 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a9daa4a2-7644-4157-aa02-b8bf5ab779b0-extensions-socket-volume\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:28:35.048392 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:28:35.048363 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-vmzr9" Oct 30 16:28:35.054996 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:28:35.054974 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-bh7c6" Oct 30 16:28:35.385164 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:28:35.385116 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-bh7c6"] Oct 30 16:28:35.386570 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:28:35.386542 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c01f5e2_a20b_45c3_a6c6_4b62b02e88cd.slice/crio-e7b0058de4c8ff348c1422ac5c30d9a7073c24d709face1148676cf7cf5b9c14 WatchSource:0}: Error finding container e7b0058de4c8ff348c1422ac5c30d9a7073c24d709face1148676cf7cf5b9c14: Status 404 returned error can't find the container with id e7b0058de4c8ff348c1422ac5c30d9a7073c24d709face1148676cf7cf5b9c14 Oct 30 16:28:35.394087 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:28:35.394063 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-vmzr9"] Oct 30 16:28:35.399389 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:28:35.399365 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef0508a7_f3e9_4bb6_8353_f0c6b7e96640.slice/crio-e5a147c76d57e4c61666bacafa8dd9ecb5a5ea38c67e707f64f8100b37d53c39 WatchSource:0}: Error finding container e5a147c76d57e4c61666bacafa8dd9ecb5a5ea38c67e707f64f8100b37d53c39: Status 404 returned error can't find the container with id e5a147c76d57e4c61666bacafa8dd9ecb5a5ea38c67e707f64f8100b37d53c39 Oct 30 16:28:35.746686 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:28:35.746651 5052 generic.go:356] "Generic (PLEG): container finished" podID="96951616-caa9-42dd-8ffc-ff0cea3d1505" containerID="cc54ff70c7c5390be2409e6e9c352fab80127c1bcd2ccd533af7d613771e33ec" exitCode=0 Oct 30 16:28:35.747130 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:28:35.746712 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-nfbzb" Oct 30 16:28:35.747130 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:28:35.746741 5052 scope.go:122] "RemoveContainer" containerID="cc54ff70c7c5390be2409e6e9c352fab80127c1bcd2ccd533af7d613771e33ec" Oct 30 16:28:35.748428 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:28:35.748403 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-64ddq" Oct 30 16:28:35.750422 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:28:35.750229 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-bh7c6" event={"ID":"8c01f5e2-a20b-45c3-a6c6-4b62b02e88cd","Type":"ContainerStarted","Data":"2ef213248a24c7911deb7a285a2fecb9a3ae43a146df1241d227319b340a9fc0"} Oct 30 16:28:35.750422 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:28:35.750299 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-bh7c6" Oct 30 16:28:35.750422 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:28:35.750324 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-bh7c6" event={"ID":"8c01f5e2-a20b-45c3-a6c6-4b62b02e88cd","Type":"ContainerStarted","Data":"e7b0058de4c8ff348c1422ac5c30d9a7073c24d709face1148676cf7cf5b9c14"} Oct 30 16:28:35.752278 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:28:35.752243 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-vmzr9" event={"ID":"ef0508a7-f3e9-4bb6-8353-f0c6b7e96640","Type":"ContainerStarted","Data":"0824aac957e81da7890b1462c51e5b400a2b8b4f99f3d6719df0f19917808770"} Oct 30 16:28:35.752379 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:28:35.752294 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-vmzr9" event={"ID":"ef0508a7-f3e9-4bb6-8353-f0c6b7e96640","Type":"ContainerStarted","Data":"e5a147c76d57e4c61666bacafa8dd9ecb5a5ea38c67e707f64f8100b37d53c39"} Oct 30 16:28:35.752379 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:28:35.752323 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-vmzr9" Oct 30 16:28:35.757622 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:28:35.757601 5052 scope.go:122] "RemoveContainer" containerID="cc54ff70c7c5390be2409e6e9c352fab80127c1bcd2ccd533af7d613771e33ec" Oct 30 16:28:35.757882 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:28:35.757864 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc54ff70c7c5390be2409e6e9c352fab80127c1bcd2ccd533af7d613771e33ec\": container with ID starting with cc54ff70c7c5390be2409e6e9c352fab80127c1bcd2ccd533af7d613771e33ec not found: ID does not exist" containerID="cc54ff70c7c5390be2409e6e9c352fab80127c1bcd2ccd533af7d613771e33ec" Oct 30 16:28:35.757968 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:28:35.757893 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc54ff70c7c5390be2409e6e9c352fab80127c1bcd2ccd533af7d613771e33ec"} err="failed to get container status \"cc54ff70c7c5390be2409e6e9c352fab80127c1bcd2ccd533af7d613771e33ec\": rpc error: code = NotFound desc = could not find container \"cc54ff70c7c5390be2409e6e9c352fab80127c1bcd2ccd533af7d613771e33ec\": container with ID starting with cc54ff70c7c5390be2409e6e9c352fab80127c1bcd2ccd533af7d613771e33ec not found: ID does not exist" Oct 30 16:28:35.757968 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:28:35.757915 5052 scope.go:122] "RemoveContainer" containerID="1121b0fd0371aa2a2a47652712b8ad63b66204df54ae2001fb54189bf6029200" Oct 30 16:28:35.764676 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:28:35.762209 5052 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-nfbzb\" is forbidden: User \"system:node:ip-10-0-131-136.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-136.ec2.internal' and this object" podUID="96951616-caa9-42dd-8ffc-ff0cea3d1505" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-nfbzb" Oct 30 16:28:35.777787 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:28:35.777764 5052 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-64ddq\" is forbidden: User \"system:node:ip-10-0-131-136.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-136.ec2.internal' and this object" podUID="a9daa4a2-7644-4157-aa02-b8bf5ab779b0" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-64ddq" Oct 30 16:28:35.830201 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:28:35.830136 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-vmzr9" podStartSLOduration=1.8301179680000002 podStartE2EDuration="1.830117968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:28:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:28:35.827624615 +0000 UTC m=+471.354959831" watchObservedRunningTime="2026-10-30 16:28:35.830117968 +0000 UTC m=+471.357453274" Oct 30 16:28:35.872017 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:28:35.871974 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-bh7c6" podStartSLOduration=1.871962312 podStartE2EDuration="1.871962312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:28:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:28:35.871539143 +0000 UTC m=+471.398874362" watchObservedRunningTime="2026-10-30 16:28:35.871962312 +0000 UTC m=+471.399297531" Oct 30 16:28:35.886491 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:28:35.886458 5052 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-nfbzb\" is forbidden: User \"system:node:ip-10-0-131-136.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-136.ec2.internal' and this object" podUID="96951616-caa9-42dd-8ffc-ff0cea3d1505" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-nfbzb" Oct 30 16:28:35.901681 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:28:35.901654 5052 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-64ddq\" is forbidden: User \"system:node:ip-10-0-131-136.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-136.ec2.internal' and this object" podUID="a9daa4a2-7644-4157-aa02-b8bf5ab779b0" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-64ddq" Oct 30 16:28:36.595623 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:28:36.595576 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96951616-caa9-42dd-8ffc-ff0cea3d1505" path="/var/lib/kubelet/pods/96951616-caa9-42dd-8ffc-ff0cea3d1505/volumes" Oct 30 16:28:36.595945 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:28:36.595933 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9daa4a2-7644-4157-aa02-b8bf5ab779b0" path="/var/lib/kubelet/pods/a9daa4a2-7644-4157-aa02-b8bf5ab779b0/volumes" Oct 30 16:28:46.760735 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:28:46.760707 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-vmzr9" Oct 30 16:28:46.761272 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:28:46.760779 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-bh7c6" Oct 30 16:28:48.838354 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:28:48.838313 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 16:28:49.115835 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:28:49.115758 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-vmzr9"] Oct 30 16:28:49.116062 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:28:49.116034 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-vmzr9" podUID="ef0508a7-f3e9-4bb6-8353-f0c6b7e96640" containerName="manager" containerID="cri-o://0824aac957e81da7890b1462c51e5b400a2b8b4f99f3d6719df0f19917808770" gracePeriod=10 Oct 30 16:28:49.363989 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:28:49.363966 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-vmzr9" Oct 30 16:28:49.450987 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:28:49.450958 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef0508a7-f3e9-4bb6-8353-f0c6b7e96640-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ef0508a7-f3e9-4bb6-8353-f0c6b7e96640-extensions-socket-volume\") pod \"ef0508a7-f3e9-4bb6-8353-f0c6b7e96640\" (UID: \"ef0508a7-f3e9-4bb6-8353-f0c6b7e96640\") " Oct 30 16:28:49.451134 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:28:49.451016 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef0508a7-f3e9-4bb6-8353-f0c6b7e96640-kube-api-access-9r2sw\" (UniqueName: \"kubernetes.io/projected/ef0508a7-f3e9-4bb6-8353-f0c6b7e96640-kube-api-access-9r2sw\") pod \"ef0508a7-f3e9-4bb6-8353-f0c6b7e96640\" (UID: \"ef0508a7-f3e9-4bb6-8353-f0c6b7e96640\") " Oct 30 16:28:49.451282 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:28:49.451261 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef0508a7-f3e9-4bb6-8353-f0c6b7e96640-extensions-socket-volume" pod "ef0508a7-f3e9-4bb6-8353-f0c6b7e96640" (UID: "ef0508a7-f3e9-4bb6-8353-f0c6b7e96640"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:28:49.452973 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:28:49.452954 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef0508a7-f3e9-4bb6-8353-f0c6b7e96640-kube-api-access-9r2sw" pod "ef0508a7-f3e9-4bb6-8353-f0c6b7e96640" (UID: "ef0508a7-f3e9-4bb6-8353-f0c6b7e96640"). InnerVolumeSpecName "kube-api-access-9r2sw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:28:49.552447 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:28:49.552422 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9r2sw\" (UniqueName: \"kubernetes.io/projected/ef0508a7-f3e9-4bb6-8353-f0c6b7e96640-kube-api-access-9r2sw\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:28:49.552447 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:28:49.552444 5052 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ef0508a7-f3e9-4bb6-8353-f0c6b7e96640-extensions-socket-volume\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:28:49.815967 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:28:49.815882 5052 generic.go:356] "Generic (PLEG): container finished" podID="ef0508a7-f3e9-4bb6-8353-f0c6b7e96640" containerID="0824aac957e81da7890b1462c51e5b400a2b8b4f99f3d6719df0f19917808770" exitCode=0 Oct 30 16:28:49.815967 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:28:49.815937 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-vmzr9" Oct 30 16:28:49.815967 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:28:49.815955 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-vmzr9" event={"ID":"ef0508a7-f3e9-4bb6-8353-f0c6b7e96640","Type":"ContainerDied","Data":"0824aac957e81da7890b1462c51e5b400a2b8b4f99f3d6719df0f19917808770"} Oct 30 16:28:49.816245 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:28:49.816010 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-vmzr9" event={"ID":"ef0508a7-f3e9-4bb6-8353-f0c6b7e96640","Type":"ContainerDied","Data":"e5a147c76d57e4c61666bacafa8dd9ecb5a5ea38c67e707f64f8100b37d53c39"} Oct 30 16:28:49.816245 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:28:49.816041 5052 scope.go:122] "RemoveContainer" containerID="0824aac957e81da7890b1462c51e5b400a2b8b4f99f3d6719df0f19917808770" Oct 30 16:28:49.824917 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:28:49.824899 5052 scope.go:122] "RemoveContainer" containerID="0824aac957e81da7890b1462c51e5b400a2b8b4f99f3d6719df0f19917808770" Oct 30 16:28:49.825196 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:28:49.825168 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0824aac957e81da7890b1462c51e5b400a2b8b4f99f3d6719df0f19917808770\": container with ID starting with 0824aac957e81da7890b1462c51e5b400a2b8b4f99f3d6719df0f19917808770 not found: ID does not exist" containerID="0824aac957e81da7890b1462c51e5b400a2b8b4f99f3d6719df0f19917808770" Oct 30 16:28:49.825264 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:28:49.825208 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0824aac957e81da7890b1462c51e5b400a2b8b4f99f3d6719df0f19917808770"} err="failed to get container status \"0824aac957e81da7890b1462c51e5b400a2b8b4f99f3d6719df0f19917808770\": rpc error: code = NotFound desc = could not find container \"0824aac957e81da7890b1462c51e5b400a2b8b4f99f3d6719df0f19917808770\": container with ID starting with 0824aac957e81da7890b1462c51e5b400a2b8b4f99f3d6719df0f19917808770 not found: ID does not exist" Oct 30 16:28:49.883570 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:28:49.883535 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-vmzr9"] Oct 30 16:28:49.887928 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:28:49.887905 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-vmzr9"] Oct 30 16:28:50.595864 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:28:50.595833 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef0508a7-f3e9-4bb6-8353-f0c6b7e96640" path="/var/lib/kubelet/pods/ef0508a7-f3e9-4bb6-8353-f0c6b7e96640/volumes" Oct 30 16:29:18.838965 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:29:18.838931 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 16:29:23.726675 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:29:23.726641 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-sf7b2"] Oct 30 16:29:23.727053 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:29:23.726982 5052 cpu_manager.go:414] "containerMap: removing container" podUID="ef0508a7-f3e9-4bb6-8353-f0c6b7e96640" containerName="manager" Oct 30 16:29:23.727053 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:29:23.726993 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef0508a7-f3e9-4bb6-8353-f0c6b7e96640" containerName="manager" Oct 30 16:29:23.727123 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:29:23.727060 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef0508a7-f3e9-4bb6-8353-f0c6b7e96640" containerName="manager" Oct 30 16:29:23.730281 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:29:23.730263 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-sf7b2" Oct 30 16:29:23.738853 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:29:23.738831 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/81bdece5-3976-4b97-abc7-cf24ee38ac05-config-file\") pod \"limitador-limitador-74dfcfb894-sf7b2\" (UID: \"81bdece5-3976-4b97-abc7-cf24ee38ac05\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-sf7b2" Oct 30 16:29:23.738959 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:29:23.738882 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgm4g\" (UniqueName: \"kubernetes.io/projected/81bdece5-3976-4b97-abc7-cf24ee38ac05-kube-api-access-mgm4g\") pod \"limitador-limitador-74dfcfb894-sf7b2\" (UID: \"81bdece5-3976-4b97-abc7-cf24ee38ac05\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-sf7b2" Oct 30 16:29:23.751154 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:29:23.751131 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-sf7b2"] Oct 30 16:29:23.763565 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:29:23.763540 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Oct 30 16:29:23.763687 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:29:23.763609 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-vtlxz\"" Oct 30 16:29:23.840036 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:29:23.840009 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/81bdece5-3976-4b97-abc7-cf24ee38ac05-config-file\") pod \"limitador-limitador-74dfcfb894-sf7b2\" (UID: \"81bdece5-3976-4b97-abc7-cf24ee38ac05\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-sf7b2" Oct 30 16:29:23.840201 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:29:23.840068 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgm4g\" (UniqueName: \"kubernetes.io/projected/81bdece5-3976-4b97-abc7-cf24ee38ac05-kube-api-access-mgm4g\") pod \"limitador-limitador-74dfcfb894-sf7b2\" (UID: \"81bdece5-3976-4b97-abc7-cf24ee38ac05\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-sf7b2" Oct 30 16:29:23.840626 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:29:23.840609 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/81bdece5-3976-4b97-abc7-cf24ee38ac05-config-file\") pod \"limitador-limitador-74dfcfb894-sf7b2\" (UID: \"81bdece5-3976-4b97-abc7-cf24ee38ac05\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-sf7b2" Oct 30 16:29:23.850105 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:29:23.850081 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-sf7b2"] Oct 30 16:29:23.866992 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:29:23.866970 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgm4g\" (UniqueName: \"kubernetes.io/projected/81bdece5-3976-4b97-abc7-cf24ee38ac05-kube-api-access-mgm4g\") pod \"limitador-limitador-74dfcfb894-sf7b2\" (UID: \"81bdece5-3976-4b97-abc7-cf24ee38ac05\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-sf7b2" Oct 30 16:29:24.043342 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:29:24.043259 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-sf7b2" Oct 30 16:29:24.345314 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:29:24.345228 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-sf7b2"] Oct 30 16:29:24.351466 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:29:24.351439 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81bdece5_3976_4b97_abc7_cf24ee38ac05.slice/crio-d162a233be690d7f26e04f53eb784154f4b2cd58c3aafae4f0840bb553d154b6 WatchSource:0}: Error finding container d162a233be690d7f26e04f53eb784154f4b2cd58c3aafae4f0840bb553d154b6: Status 404 returned error can't find the container with id d162a233be690d7f26e04f53eb784154f4b2cd58c3aafae4f0840bb553d154b6 Oct 30 16:29:24.975962 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:29:24.975926 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-sf7b2" event={"ID":"81bdece5-3976-4b97-abc7-cf24ee38ac05","Type":"ContainerStarted","Data":"d162a233be690d7f26e04f53eb784154f4b2cd58c3aafae4f0840bb553d154b6"} Oct 30 16:29:25.035421 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:29:25.035327 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-fbmqw"] Oct 30 16:29:25.038965 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:29:25.038934 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cdwcj"] Oct 30 16:29:25.039086 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:29:25.039070 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-fbmqw" Oct 30 16:29:25.042668 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:29:25.042646 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-cdwcj" Oct 30 16:29:25.050794 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:29:25.050770 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbsvr\" (UniqueName: \"kubernetes.io/projected/f13a47a9-59c2-41a9-9748-20f135986be3-kube-api-access-cbsvr\") pod \"authorino-56c6c9ddd5-fbmqw\" (UID: \"f13a47a9-59c2-41a9-9748-20f135986be3\") " pod="kuadrant-system/authorino-56c6c9ddd5-fbmqw" Oct 30 16:29:25.057427 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:29:25.057366 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-fbmqw"] Oct 30 16:29:25.069513 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:29:25.069481 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-rsnhr\"" Oct 30 16:29:25.071004 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:29:25.070978 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cdwcj"] Oct 30 16:29:25.142282 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:29:25.142244 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cdwcj"] Oct 30 16:29:25.152037 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:29:25.152002 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbsvr\" (UniqueName: \"kubernetes.io/projected/f13a47a9-59c2-41a9-9748-20f135986be3-kube-api-access-cbsvr\") pod \"authorino-56c6c9ddd5-fbmqw\" (UID: \"f13a47a9-59c2-41a9-9748-20f135986be3\") " pod="kuadrant-system/authorino-56c6c9ddd5-fbmqw" Oct 30 16:29:25.152175 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:29:25.152047 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c1731802-685a-44c6-9700-efc3784bbbeb-config-file\") pod \"limitador-limitador-7c9b87d574-cdwcj\" (UID: \"c1731802-685a-44c6-9700-efc3784bbbeb\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-cdwcj" Oct 30 16:29:25.152175 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:29:25.152158 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xj65k\" (UniqueName: \"kubernetes.io/projected/c1731802-685a-44c6-9700-efc3784bbbeb-kube-api-access-xj65k\") pod \"limitador-limitador-7c9b87d574-cdwcj\" (UID: \"c1731802-685a-44c6-9700-efc3784bbbeb\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-cdwcj" Oct 30 16:29:25.177322 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:29:25.177298 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbsvr\" (UniqueName: \"kubernetes.io/projected/f13a47a9-59c2-41a9-9748-20f135986be3-kube-api-access-cbsvr\") pod \"authorino-56c6c9ddd5-fbmqw\" (UID: \"f13a47a9-59c2-41a9-9748-20f135986be3\") " pod="kuadrant-system/authorino-56c6c9ddd5-fbmqw" Oct 30 16:29:25.253125 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:29:25.253053 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c1731802-685a-44c6-9700-efc3784bbbeb-config-file\") pod \"limitador-limitador-7c9b87d574-cdwcj\" (UID: \"c1731802-685a-44c6-9700-efc3784bbbeb\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-cdwcj" Oct 30 16:29:25.253125 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:29:25.253124 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xj65k\" (UniqueName: \"kubernetes.io/projected/c1731802-685a-44c6-9700-efc3784bbbeb-kube-api-access-xj65k\") pod \"limitador-limitador-7c9b87d574-cdwcj\" (UID: \"c1731802-685a-44c6-9700-efc3784bbbeb\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-cdwcj" Oct 30 16:29:25.253827 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:29:25.253795 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c1731802-685a-44c6-9700-efc3784bbbeb-config-file\") pod \"limitador-limitador-7c9b87d574-cdwcj\" (UID: \"c1731802-685a-44c6-9700-efc3784bbbeb\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-cdwcj" Oct 30 16:29:25.278720 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:29:25.278696 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xj65k\" (UniqueName: \"kubernetes.io/projected/c1731802-685a-44c6-9700-efc3784bbbeb-kube-api-access-xj65k\") pod \"limitador-limitador-7c9b87d574-cdwcj\" (UID: \"c1731802-685a-44c6-9700-efc3784bbbeb\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-cdwcj" Oct 30 16:29:25.362309 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:29:25.362291 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-fbmqw" Oct 30 16:29:25.371432 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:29:25.371282 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-cdwcj" Oct 30 16:29:25.664901 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:29:25.664875 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-fbmqw"] Oct 30 16:29:25.666486 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:29:25.666460 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf13a47a9_59c2_41a9_9748_20f135986be3.slice/crio-6f042dcec42c7c3ad481f042f0dbbae31b8bfc1dd8e231a69c8e0e9a47bd5c39 WatchSource:0}: Error finding container 6f042dcec42c7c3ad481f042f0dbbae31b8bfc1dd8e231a69c8e0e9a47bd5c39: Status 404 returned error can't find the container with id 6f042dcec42c7c3ad481f042f0dbbae31b8bfc1dd8e231a69c8e0e9a47bd5c39 Oct 30 16:29:25.681293 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:29:25.681273 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cdwcj"] Oct 30 16:29:25.683026 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:29:25.683001 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1731802_685a_44c6_9700_efc3784bbbeb.slice/crio-ca4ee3e10e11caf4812c6bc9289238387fbaa32c655d428271bc4af975e4dec9 WatchSource:0}: Error finding container ca4ee3e10e11caf4812c6bc9289238387fbaa32c655d428271bc4af975e4dec9: Status 404 returned error can't find the container with id ca4ee3e10e11caf4812c6bc9289238387fbaa32c655d428271bc4af975e4dec9 Oct 30 16:29:25.982203 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:29:25.982122 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-fbmqw" event={"ID":"f13a47a9-59c2-41a9-9748-20f135986be3","Type":"ContainerStarted","Data":"6f042dcec42c7c3ad481f042f0dbbae31b8bfc1dd8e231a69c8e0e9a47bd5c39"} Oct 30 16:29:25.983571 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:29:25.983546 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-sf7b2" event={"ID":"81bdece5-3976-4b97-abc7-cf24ee38ac05","Type":"ContainerStarted","Data":"0e1eb42e2f13b7f9c880e6e63de06a138fa004b83ef4e669bed46c25fd6e8d7f"} Oct 30 16:29:25.983725 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:29:25.983644 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-sf7b2" Oct 30 16:29:25.984956 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:29:25.984933 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-cdwcj" event={"ID":"c1731802-685a-44c6-9700-efc3784bbbeb","Type":"ContainerStarted","Data":"b7810a4d450f1430b1ae2f96cd6e19f768848d26ff6eddf936824fe14ebac3b4"} Oct 30 16:29:25.985057 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:29:25.984968 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-cdwcj" event={"ID":"c1731802-685a-44c6-9700-efc3784bbbeb","Type":"ContainerStarted","Data":"ca4ee3e10e11caf4812c6bc9289238387fbaa32c655d428271bc4af975e4dec9"} Oct 30 16:29:25.985057 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:29:25.984992 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-cdwcj" Oct 30 16:29:26.028936 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:29:26.028892 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-sf7b2" podStartSLOduration=2.054148866 podStartE2EDuration="3.028879474s" totalImagesPullingTime="974.730608ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:29:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:29:26.027941615 +0000 UTC m=+521.555276833" watchObservedRunningTime="2026-10-30 16:29:26.028879474 +0000 UTC m=+521.556214692" Oct 30 16:29:26.074128 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:29:26.074080 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-cdwcj" podStartSLOduration=1.074062989 podStartE2EDuration="1.074062989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:29:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:29:26.072957488 +0000 UTC m=+521.600292706" watchObservedRunningTime="2026-10-30 16:29:26.074062989 +0000 UTC m=+521.601398202" Oct 30 16:29:27.995707 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:29:27.995666 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-fbmqw" event={"ID":"f13a47a9-59c2-41a9-9748-20f135986be3","Type":"ContainerStarted","Data":"e960cbdd13a5e6ac29a27002afe0685b64bf997dff2e1de2f6b589f2d7a85321"} Oct 30 16:29:28.038943 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:29:28.038891 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-fbmqw" podStartSLOduration=1.140463619 podStartE2EDuration="3.038877883s" totalImagesPullingTime="1.898414264s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:29:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:29:28.037468303 +0000 UTC m=+523.564803523" watchObservedRunningTime="2026-10-30 16:29:28.038877883 +0000 UTC m=+523.566213102" Oct 30 16:29:36.991061 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:29:36.991026 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-sf7b2" Oct 30 16:29:36.991511 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:29:36.991096 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-cdwcj" Oct 30 16:29:37.117720 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:29:37.117688 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-sf7b2"] Oct 30 16:29:37.117913 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:29:37.117889 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-sf7b2" podUID="81bdece5-3976-4b97-abc7-cf24ee38ac05" containerName="limitador" containerID="cri-o://0e1eb42e2f13b7f9c880e6e63de06a138fa004b83ef4e669bed46c25fd6e8d7f" gracePeriod=30 Oct 30 16:29:37.665625 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:29:37.665603 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-sf7b2" Oct 30 16:29:37.757031 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:29:37.756960 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81bdece5-3976-4b97-abc7-cf24ee38ac05-config-file\" (UniqueName: \"kubernetes.io/configmap/81bdece5-3976-4b97-abc7-cf24ee38ac05-config-file\") pod \"81bdece5-3976-4b97-abc7-cf24ee38ac05\" (UID: \"81bdece5-3976-4b97-abc7-cf24ee38ac05\") " Oct 30 16:29:37.757031 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:29:37.757017 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81bdece5-3976-4b97-abc7-cf24ee38ac05-kube-api-access-mgm4g\" (UniqueName: \"kubernetes.io/projected/81bdece5-3976-4b97-abc7-cf24ee38ac05-kube-api-access-mgm4g\") pod \"81bdece5-3976-4b97-abc7-cf24ee38ac05\" (UID: \"81bdece5-3976-4b97-abc7-cf24ee38ac05\") " Oct 30 16:29:37.757383 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:29:37.757353 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81bdece5-3976-4b97-abc7-cf24ee38ac05-config-file" pod "81bdece5-3976-4b97-abc7-cf24ee38ac05" (UID: "81bdece5-3976-4b97-abc7-cf24ee38ac05"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:29:37.759142 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:29:37.759116 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81bdece5-3976-4b97-abc7-cf24ee38ac05-kube-api-access-mgm4g" pod "81bdece5-3976-4b97-abc7-cf24ee38ac05" (UID: "81bdece5-3976-4b97-abc7-cf24ee38ac05"). InnerVolumeSpecName "kube-api-access-mgm4g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:29:37.858368 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:29:37.858339 5052 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/81bdece5-3976-4b97-abc7-cf24ee38ac05-config-file\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:29:37.858368 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:29:37.858364 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgm4g\" (UniqueName: \"kubernetes.io/projected/81bdece5-3976-4b97-abc7-cf24ee38ac05-kube-api-access-mgm4g\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:29:38.041457 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:29:38.041373 5052 generic.go:356] "Generic (PLEG): container finished" podID="81bdece5-3976-4b97-abc7-cf24ee38ac05" containerID="0e1eb42e2f13b7f9c880e6e63de06a138fa004b83ef4e669bed46c25fd6e8d7f" exitCode=0 Oct 30 16:29:38.041457 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:29:38.041424 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-sf7b2" Oct 30 16:29:38.041906 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:29:38.041455 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-sf7b2" event={"ID":"81bdece5-3976-4b97-abc7-cf24ee38ac05","Type":"ContainerDied","Data":"0e1eb42e2f13b7f9c880e6e63de06a138fa004b83ef4e669bed46c25fd6e8d7f"} Oct 30 16:29:38.041906 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:29:38.041513 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-sf7b2" event={"ID":"81bdece5-3976-4b97-abc7-cf24ee38ac05","Type":"ContainerDied","Data":"d162a233be690d7f26e04f53eb784154f4b2cd58c3aafae4f0840bb553d154b6"} Oct 30 16:29:38.041906 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:29:38.041537 5052 scope.go:122] "RemoveContainer" containerID="0e1eb42e2f13b7f9c880e6e63de06a138fa004b83ef4e669bed46c25fd6e8d7f" Oct 30 16:29:38.050363 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:29:38.050340 5052 scope.go:122] "RemoveContainer" containerID="0e1eb42e2f13b7f9c880e6e63de06a138fa004b83ef4e669bed46c25fd6e8d7f" Oct 30 16:29:38.050618 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:29:38.050596 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e1eb42e2f13b7f9c880e6e63de06a138fa004b83ef4e669bed46c25fd6e8d7f\": container with ID starting with 0e1eb42e2f13b7f9c880e6e63de06a138fa004b83ef4e669bed46c25fd6e8d7f not found: ID does not exist" containerID="0e1eb42e2f13b7f9c880e6e63de06a138fa004b83ef4e669bed46c25fd6e8d7f" Oct 30 16:29:38.050694 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:29:38.050626 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e1eb42e2f13b7f9c880e6e63de06a138fa004b83ef4e669bed46c25fd6e8d7f"} err="failed to get container status \"0e1eb42e2f13b7f9c880e6e63de06a138fa004b83ef4e669bed46c25fd6e8d7f\": rpc error: code = NotFound desc = could not find container \"0e1eb42e2f13b7f9c880e6e63de06a138fa004b83ef4e669bed46c25fd6e8d7f\": container with ID starting with 0e1eb42e2f13b7f9c880e6e63de06a138fa004b83ef4e669bed46c25fd6e8d7f not found: ID does not exist" Oct 30 16:29:38.097759 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:29:38.097734 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-sf7b2"] Oct 30 16:29:38.111428 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:29:38.111407 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-sf7b2"] Oct 30 16:29:38.595860 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:29:38.595831 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81bdece5-3976-4b97-abc7-cf24ee38ac05" path="/var/lib/kubelet/pods/81bdece5-3976-4b97-abc7-cf24ee38ac05/volumes" Oct 30 16:29:48.838171 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:29:48.838132 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 16:29:55.279285 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:29:55.279252 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-vwd6j"] Oct 30 16:29:55.279804 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:29:55.279638 5052 cpu_manager.go:414] "containerMap: removing container" podUID="81bdece5-3976-4b97-abc7-cf24ee38ac05" containerName="limitador" Oct 30 16:29:55.279804 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:29:55.279650 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81bdece5-3976-4b97-abc7-cf24ee38ac05" containerName="limitador" Oct 30 16:29:55.279804 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:29:55.279711 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="81bdece5-3976-4b97-abc7-cf24ee38ac05" containerName="limitador" Oct 30 16:29:55.284118 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:29:55.284098 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-vwd6j" Oct 30 16:29:55.301814 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:29:55.301790 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-vwd6j"] Oct 30 16:29:55.406442 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:29:55.406411 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pql98\" (UniqueName: \"kubernetes.io/projected/2edd0958-0bd8-4796-b727-81273b646d30-kube-api-access-pql98\") pod \"authorino-6f6944bcdd-vwd6j\" (UID: \"2edd0958-0bd8-4796-b727-81273b646d30\") " pod="kuadrant-system/authorino-6f6944bcdd-vwd6j" Oct 30 16:29:55.444603 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:29:55.444563 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-vwd6j"] Oct 30 16:29:55.444771 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:29:55.444750 5052 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-pql98], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6f6944bcdd-vwd6j" podUID="2edd0958-0bd8-4796-b727-81273b646d30" Oct 30 16:29:55.507910 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:29:55.507882 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pql98\" (UniqueName: \"kubernetes.io/projected/2edd0958-0bd8-4796-b727-81273b646d30-kube-api-access-pql98\") pod \"authorino-6f6944bcdd-vwd6j\" (UID: \"2edd0958-0bd8-4796-b727-81273b646d30\") " pod="kuadrant-system/authorino-6f6944bcdd-vwd6j" Oct 30 16:29:55.533008 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:29:55.532951 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pql98\" (UniqueName: \"kubernetes.io/projected/2edd0958-0bd8-4796-b727-81273b646d30-kube-api-access-pql98\") pod \"authorino-6f6944bcdd-vwd6j\" (UID: \"2edd0958-0bd8-4796-b727-81273b646d30\") " pod="kuadrant-system/authorino-6f6944bcdd-vwd6j" Oct 30 16:29:56.126540 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:29:56.126509 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-vwd6j" Oct 30 16:29:56.135091 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:29:56.135070 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-vwd6j" Oct 30 16:29:56.214469 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:29:56.214441 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2edd0958-0bd8-4796-b727-81273b646d30-kube-api-access-pql98\" (UniqueName: \"kubernetes.io/projected/2edd0958-0bd8-4796-b727-81273b646d30-kube-api-access-pql98\") pod \"2edd0958-0bd8-4796-b727-81273b646d30\" (UID: \"2edd0958-0bd8-4796-b727-81273b646d30\") " Oct 30 16:29:56.216558 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:29:56.216535 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2edd0958-0bd8-4796-b727-81273b646d30-kube-api-access-pql98" pod "2edd0958-0bd8-4796-b727-81273b646d30" (UID: "2edd0958-0bd8-4796-b727-81273b646d30"). InnerVolumeSpecName "kube-api-access-pql98". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:29:56.315212 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:29:56.315184 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pql98\" (UniqueName: \"kubernetes.io/projected/2edd0958-0bd8-4796-b727-81273b646d30-kube-api-access-pql98\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:29:57.131663 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:29:57.131633 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-vwd6j" Oct 30 16:29:57.211047 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:29:57.211016 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-vwd6j"] Oct 30 16:29:57.215185 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:29:57.215154 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-vwd6j"] Oct 30 16:29:57.502337 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:29:57.502301 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-fbmqw"] Oct 30 16:29:57.502729 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:29:57.502524 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-fbmqw" podUID="f13a47a9-59c2-41a9-9748-20f135986be3" containerName="authorino" containerID="cri-o://e960cbdd13a5e6ac29a27002afe0685b64bf997dff2e1de2f6b589f2d7a85321" gracePeriod=30 Oct 30 16:29:57.763264 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:29:57.763211 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-fbmqw" Oct 30 16:29:57.930246 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:29:57.930208 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f13a47a9-59c2-41a9-9748-20f135986be3-kube-api-access-cbsvr\" (UniqueName: \"kubernetes.io/projected/f13a47a9-59c2-41a9-9748-20f135986be3-kube-api-access-cbsvr\") pod \"f13a47a9-59c2-41a9-9748-20f135986be3\" (UID: \"f13a47a9-59c2-41a9-9748-20f135986be3\") " Oct 30 16:29:57.932264 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:29:57.932239 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f13a47a9-59c2-41a9-9748-20f135986be3-kube-api-access-cbsvr" pod "f13a47a9-59c2-41a9-9748-20f135986be3" (UID: "f13a47a9-59c2-41a9-9748-20f135986be3"). InnerVolumeSpecName "kube-api-access-cbsvr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:29:58.031012 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:29:58.030941 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbsvr\" (UniqueName: \"kubernetes.io/projected/f13a47a9-59c2-41a9-9748-20f135986be3-kube-api-access-cbsvr\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:29:58.136819 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:29:58.136789 5052 generic.go:356] "Generic (PLEG): container finished" podID="f13a47a9-59c2-41a9-9748-20f135986be3" containerID="e960cbdd13a5e6ac29a27002afe0685b64bf997dff2e1de2f6b589f2d7a85321" exitCode=0 Oct 30 16:29:58.136964 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:29:58.136835 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-fbmqw" Oct 30 16:29:58.136964 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:29:58.136874 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-fbmqw" event={"ID":"f13a47a9-59c2-41a9-9748-20f135986be3","Type":"ContainerDied","Data":"e960cbdd13a5e6ac29a27002afe0685b64bf997dff2e1de2f6b589f2d7a85321"} Oct 30 16:29:58.136964 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:29:58.136919 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-fbmqw" event={"ID":"f13a47a9-59c2-41a9-9748-20f135986be3","Type":"ContainerDied","Data":"6f042dcec42c7c3ad481f042f0dbbae31b8bfc1dd8e231a69c8e0e9a47bd5c39"} Oct 30 16:29:58.136964 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:29:58.136940 5052 scope.go:122] "RemoveContainer" containerID="e960cbdd13a5e6ac29a27002afe0685b64bf997dff2e1de2f6b589f2d7a85321" Oct 30 16:29:58.145675 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:29:58.145653 5052 scope.go:122] "RemoveContainer" containerID="e960cbdd13a5e6ac29a27002afe0685b64bf997dff2e1de2f6b589f2d7a85321" Oct 30 16:29:58.145956 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:29:58.145935 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e960cbdd13a5e6ac29a27002afe0685b64bf997dff2e1de2f6b589f2d7a85321\": container with ID starting with e960cbdd13a5e6ac29a27002afe0685b64bf997dff2e1de2f6b589f2d7a85321 not found: ID does not exist" containerID="e960cbdd13a5e6ac29a27002afe0685b64bf997dff2e1de2f6b589f2d7a85321" Oct 30 16:29:58.146029 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:29:58.145963 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e960cbdd13a5e6ac29a27002afe0685b64bf997dff2e1de2f6b589f2d7a85321"} err="failed to get container status \"e960cbdd13a5e6ac29a27002afe0685b64bf997dff2e1de2f6b589f2d7a85321\": rpc error: code = NotFound desc = could not find container \"e960cbdd13a5e6ac29a27002afe0685b64bf997dff2e1de2f6b589f2d7a85321\": container with ID starting with e960cbdd13a5e6ac29a27002afe0685b64bf997dff2e1de2f6b589f2d7a85321 not found: ID does not exist" Oct 30 16:29:58.191820 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:29:58.191798 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-fbmqw"] Oct 30 16:29:58.205546 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:29:58.205522 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-fbmqw"] Oct 30 16:29:58.595370 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:29:58.595338 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2edd0958-0bd8-4796-b727-81273b646d30" path="/var/lib/kubelet/pods/2edd0958-0bd8-4796-b727-81273b646d30/volumes" Oct 30 16:29:58.595778 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:29:58.595546 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f13a47a9-59c2-41a9-9748-20f135986be3" path="/var/lib/kubelet/pods/f13a47a9-59c2-41a9-9748-20f135986be3/volumes" Oct 30 16:30:15.419437 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:15.419369 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-577f596567-h5692"] Oct 30 16:30:15.419799 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:15.419787 5052 cpu_manager.go:414] "containerMap: removing container" podUID="f13a47a9-59c2-41a9-9748-20f135986be3" containerName="authorino" Oct 30 16:30:15.419847 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:15.419799 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f13a47a9-59c2-41a9-9748-20f135986be3" containerName="authorino" Oct 30 16:30:15.419882 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:15.419865 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="f13a47a9-59c2-41a9-9748-20f135986be3" containerName="authorino" Oct 30 16:30:15.422919 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:15.422901 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-577f596567-h5692" Oct 30 16:30:15.441703 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:15.441688 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 30 16:30:15.453275 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:15.453254 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-gtz2x\"" Oct 30 16:30:15.453395 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:15.453322 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 30 16:30:15.453395 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:15.453378 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 30 16:30:15.454795 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:15.454772 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-577f596567-h5692"] Oct 30 16:30:15.472323 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:15.472303 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/95d70dec-072d-4521-a554-fc167a1d8bd1-webhook-cert\") pod \"maas-controller-577f596567-h5692\" (UID: \"95d70dec-072d-4521-a554-fc167a1d8bd1\") " pod="opendatahub/maas-controller-577f596567-h5692" Oct 30 16:30:15.472420 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:15.472329 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4kh7\" (UniqueName: \"kubernetes.io/projected/95d70dec-072d-4521-a554-fc167a1d8bd1-kube-api-access-r4kh7\") pod \"maas-controller-577f596567-h5692\" (UID: \"95d70dec-072d-4521-a554-fc167a1d8bd1\") " pod="opendatahub/maas-controller-577f596567-h5692" Oct 30 16:30:15.472420 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:15.472376 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95d70dec-072d-4521-a554-fc167a1d8bd1-metrics-certs\") pod \"maas-controller-577f596567-h5692\" (UID: \"95d70dec-072d-4521-a554-fc167a1d8bd1\") " pod="opendatahub/maas-controller-577f596567-h5692" Oct 30 16:30:15.572811 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:15.572781 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95d70dec-072d-4521-a554-fc167a1d8bd1-metrics-certs\") pod \"maas-controller-577f596567-h5692\" (UID: \"95d70dec-072d-4521-a554-fc167a1d8bd1\") " pod="opendatahub/maas-controller-577f596567-h5692" Oct 30 16:30:15.572936 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:15.572833 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/95d70dec-072d-4521-a554-fc167a1d8bd1-webhook-cert\") pod \"maas-controller-577f596567-h5692\" (UID: \"95d70dec-072d-4521-a554-fc167a1d8bd1\") " pod="opendatahub/maas-controller-577f596567-h5692" Oct 30 16:30:15.573029 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:15.573015 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4kh7\" (UniqueName: \"kubernetes.io/projected/95d70dec-072d-4521-a554-fc167a1d8bd1-kube-api-access-r4kh7\") pod \"maas-controller-577f596567-h5692\" (UID: \"95d70dec-072d-4521-a554-fc167a1d8bd1\") " pod="opendatahub/maas-controller-577f596567-h5692" Oct 30 16:30:15.575320 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:15.575302 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/95d70dec-072d-4521-a554-fc167a1d8bd1-webhook-cert\") pod \"maas-controller-577f596567-h5692\" (UID: \"95d70dec-072d-4521-a554-fc167a1d8bd1\") " pod="opendatahub/maas-controller-577f596567-h5692" Oct 30 16:30:15.575437 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:15.575414 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95d70dec-072d-4521-a554-fc167a1d8bd1-metrics-certs\") pod \"maas-controller-577f596567-h5692\" (UID: \"95d70dec-072d-4521-a554-fc167a1d8bd1\") " pod="opendatahub/maas-controller-577f596567-h5692" Oct 30 16:30:15.605184 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:15.605167 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4kh7\" (UniqueName: \"kubernetes.io/projected/95d70dec-072d-4521-a554-fc167a1d8bd1-kube-api-access-r4kh7\") pod \"maas-controller-577f596567-h5692\" (UID: \"95d70dec-072d-4521-a554-fc167a1d8bd1\") " pod="opendatahub/maas-controller-577f596567-h5692" Oct 30 16:30:15.735469 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:15.735406 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-577f596567-h5692" Oct 30 16:30:15.952245 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:15.952215 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-575c7b5c5b-lh44m"] Oct 30 16:30:15.955864 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:15.955844 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-575c7b5c5b-lh44m" Oct 30 16:30:15.974992 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:15.974959 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c997fe94-1cf0-4316-826e-6bd921636715-webhook-cert\") pod \"maas-controller-575c7b5c5b-lh44m\" (UID: \"c997fe94-1cf0-4316-826e-6bd921636715\") " pod="opendatahub/maas-controller-575c7b5c5b-lh44m" Oct 30 16:30:15.975140 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:15.975068 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c997fe94-1cf0-4316-826e-6bd921636715-metrics-certs\") pod \"maas-controller-575c7b5c5b-lh44m\" (UID: \"c997fe94-1cf0-4316-826e-6bd921636715\") " pod="opendatahub/maas-controller-575c7b5c5b-lh44m" Oct 30 16:30:15.975140 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:15.975126 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9bvv\" (UniqueName: \"kubernetes.io/projected/c997fe94-1cf0-4316-826e-6bd921636715-kube-api-access-b9bvv\") pod \"maas-controller-575c7b5c5b-lh44m\" (UID: \"c997fe94-1cf0-4316-826e-6bd921636715\") " pod="opendatahub/maas-controller-575c7b5c5b-lh44m" Oct 30 16:30:15.988619 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:15.988527 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-575c7b5c5b-lh44m"] Oct 30 16:30:16.062667 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:16.062645 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-577f596567-h5692"] Oct 30 16:30:16.064474 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:30:16.064443 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95d70dec_072d_4521_a554_fc167a1d8bd1.slice/crio-1079ac575f81e64f99d0e6dfb9380bb3ea0f174b41d5ed4cb1946c9a185d01e0 WatchSource:0}: Error finding container 1079ac575f81e64f99d0e6dfb9380bb3ea0f174b41d5ed4cb1946c9a185d01e0: Status 404 returned error can't find the container with id 1079ac575f81e64f99d0e6dfb9380bb3ea0f174b41d5ed4cb1946c9a185d01e0 Oct 30 16:30:16.075804 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:16.075779 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c997fe94-1cf0-4316-826e-6bd921636715-metrics-certs\") pod \"maas-controller-575c7b5c5b-lh44m\" (UID: \"c997fe94-1cf0-4316-826e-6bd921636715\") " pod="opendatahub/maas-controller-575c7b5c5b-lh44m" Oct 30 16:30:16.075922 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:16.075819 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9bvv\" (UniqueName: \"kubernetes.io/projected/c997fe94-1cf0-4316-826e-6bd921636715-kube-api-access-b9bvv\") pod \"maas-controller-575c7b5c5b-lh44m\" (UID: \"c997fe94-1cf0-4316-826e-6bd921636715\") " pod="opendatahub/maas-controller-575c7b5c5b-lh44m" Oct 30 16:30:16.075922 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:16.075892 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c997fe94-1cf0-4316-826e-6bd921636715-webhook-cert\") pod \"maas-controller-575c7b5c5b-lh44m\" (UID: \"c997fe94-1cf0-4316-826e-6bd921636715\") " pod="opendatahub/maas-controller-575c7b5c5b-lh44m" Oct 30 16:30:16.078249 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:16.078229 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c997fe94-1cf0-4316-826e-6bd921636715-metrics-certs\") pod \"maas-controller-575c7b5c5b-lh44m\" (UID: \"c997fe94-1cf0-4316-826e-6bd921636715\") " pod="opendatahub/maas-controller-575c7b5c5b-lh44m" Oct 30 16:30:16.078334 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:16.078309 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c997fe94-1cf0-4316-826e-6bd921636715-webhook-cert\") pod \"maas-controller-575c7b5c5b-lh44m\" (UID: \"c997fe94-1cf0-4316-826e-6bd921636715\") " pod="opendatahub/maas-controller-575c7b5c5b-lh44m" Oct 30 16:30:16.100739 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:16.100716 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9bvv\" (UniqueName: \"kubernetes.io/projected/c997fe94-1cf0-4316-826e-6bd921636715-kube-api-access-b9bvv\") pod \"maas-controller-575c7b5c5b-lh44m\" (UID: \"c997fe94-1cf0-4316-826e-6bd921636715\") " pod="opendatahub/maas-controller-575c7b5c5b-lh44m" Oct 30 16:30:16.220455 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:16.220417 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-577f596567-h5692" event={"ID":"95d70dec-072d-4521-a554-fc167a1d8bd1","Type":"ContainerStarted","Data":"1079ac575f81e64f99d0e6dfb9380bb3ea0f174b41d5ed4cb1946c9a185d01e0"} Oct 30 16:30:16.268745 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:16.268672 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-575c7b5c5b-lh44m" Oct 30 16:30:16.573164 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:16.573081 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-575c7b5c5b-lh44m"] Oct 30 16:30:16.577697 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:30:16.577664 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc997fe94_1cf0_4316_826e_6bd921636715.slice/crio-cccb83bd0c96b4c272b1f8a8a44014a6b37d943804eb29120b9923c41fc88d6a WatchSource:0}: Error finding container cccb83bd0c96b4c272b1f8a8a44014a6b37d943804eb29120b9923c41fc88d6a: Status 404 returned error can't find the container with id cccb83bd0c96b4c272b1f8a8a44014a6b37d943804eb29120b9923c41fc88d6a Oct 30 16:30:17.228094 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:17.228015 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-575c7b5c5b-lh44m" event={"ID":"c997fe94-1cf0-4316-826e-6bd921636715","Type":"ContainerStarted","Data":"cccb83bd0c96b4c272b1f8a8a44014a6b37d943804eb29120b9923c41fc88d6a"} Oct 30 16:30:18.843800 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:30:18.843711 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 16:30:19.239890 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:19.239853 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-577f596567-h5692" event={"ID":"95d70dec-072d-4521-a554-fc167a1d8bd1","Type":"ContainerStarted","Data":"21814cf179775f76499c8fbaef357f6d30ef60f74e0d5d1d6487972a79193a6b"} Oct 30 16:30:19.240061 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:19.239965 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-577f596567-h5692" Oct 30 16:30:19.241333 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:19.241310 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-575c7b5c5b-lh44m" event={"ID":"c997fe94-1cf0-4316-826e-6bd921636715","Type":"ContainerStarted","Data":"5481a05961ec5bf038b9a3b2e17d1c5e41beb314db8388fb2cdc2cc8099dc91f"} Oct 30 16:30:19.241518 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:19.241448 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-575c7b5c5b-lh44m" Oct 30 16:30:19.286131 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:19.286088 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-577f596567-h5692" podStartSLOduration=1.60421573 podStartE2EDuration="4.286076413s" totalImagesPullingTime="2.681860683s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:30:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:30:19.285343772 +0000 UTC m=+574.812678991" watchObservedRunningTime="2026-10-30 16:30:19.286076413 +0000 UTC m=+574.813411632" Oct 30 16:30:19.332514 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:19.332464 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-575c7b5c5b-lh44m" podStartSLOduration=1.8215084799999999 podStartE2EDuration="4.332453113s" totalImagesPullingTime="2.510944633s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:30:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:30:19.331870716 +0000 UTC m=+574.859205934" watchObservedRunningTime="2026-10-30 16:30:19.332453113 +0000 UTC m=+574.859788333" Oct 30 16:30:22.519801 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:22.519764 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-4f74l"] Oct 30 16:30:22.523676 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:22.523652 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-4f74l" Oct 30 16:30:22.527945 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:22.527914 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/f3dcbff0-3582-4f17-8781-6aedfe47d607-proxy-source\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-4f74l\" (UID: \"f3dcbff0-3582-4f17-8781-6aedfe47d607\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-4f74l" Oct 30 16:30:22.528057 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:22.528031 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f3dcbff0-3582-4f17-8781-6aedfe47d607-service-ca\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-4f74l\" (UID: \"f3dcbff0-3582-4f17-8781-6aedfe47d607\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-4f74l" Oct 30 16:30:22.528101 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:22.528057 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jnxz\" (UniqueName: \"kubernetes.io/projected/f3dcbff0-3582-4f17-8781-6aedfe47d607-kube-api-access-9jnxz\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-4f74l\" (UID: \"f3dcbff0-3582-4f17-8781-6aedfe47d607\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-4f74l" Oct 30 16:30:22.528193 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:22.528176 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f3dcbff0-3582-4f17-8781-6aedfe47d607-tls-cert\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-4f74l\" (UID: \"f3dcbff0-3582-4f17-8781-6aedfe47d607\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-4f74l" Oct 30 16:30:22.557661 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:22.557633 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\"" Oct 30 16:30:22.557834 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:22.557695 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\"" Oct 30 16:30:22.557834 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:22.557719 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\"" Oct 30 16:30:22.558341 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:22.557758 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-qvlwc\"" Oct 30 16:30:22.559254 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:22.559234 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-4f74l"] Oct 30 16:30:22.629181 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:22.629122 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/f3dcbff0-3582-4f17-8781-6aedfe47d607-proxy-source\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-4f74l\" (UID: \"f3dcbff0-3582-4f17-8781-6aedfe47d607\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-4f74l" Oct 30 16:30:22.629345 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:22.629185 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f3dcbff0-3582-4f17-8781-6aedfe47d607-service-ca\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-4f74l\" (UID: \"f3dcbff0-3582-4f17-8781-6aedfe47d607\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-4f74l" Oct 30 16:30:22.629345 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:22.629210 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jnxz\" (UniqueName: \"kubernetes.io/projected/f3dcbff0-3582-4f17-8781-6aedfe47d607-kube-api-access-9jnxz\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-4f74l\" (UID: \"f3dcbff0-3582-4f17-8781-6aedfe47d607\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-4f74l" Oct 30 16:30:22.629345 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:30:22.629250 5052 configmap.go:193] Couldn't get configMap opendatahub/usage-tenancy-proxy-source: configmap "usage-tenancy-proxy-source" not found Oct 30 16:30:22.629345 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:22.629282 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f3dcbff0-3582-4f17-8781-6aedfe47d607-tls-cert\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-4f74l\" (UID: \"f3dcbff0-3582-4f17-8781-6aedfe47d607\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-4f74l" Oct 30 16:30:22.629345 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:30:22.629319 5052 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f3dcbff0-3582-4f17-8781-6aedfe47d607-proxy-source podName:f3dcbff0-3582-4f17-8781-6aedfe47d607 nodeName:}" failed. No retries permitted until 2026-10-30 16:30:23.129299277 +0000 UTC m=+578.656634485 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-source" (UniqueName: "kubernetes.io/configmap/f3dcbff0-3582-4f17-8781-6aedfe47d607-proxy-source") pod "usage-logs-tenancy-proxy-7d94f55f4b-4f74l" (UID: "f3dcbff0-3582-4f17-8781-6aedfe47d607") : configmap "usage-tenancy-proxy-source" not found Oct 30 16:30:22.629620 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:30:22.629395 5052 secret.go:189] Couldn't get secret opendatahub/usage-logs-tenancy-proxy-tls: secret "usage-logs-tenancy-proxy-tls" not found Oct 30 16:30:22.629620 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:30:22.629479 5052 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3dcbff0-3582-4f17-8781-6aedfe47d607-tls-cert podName:f3dcbff0-3582-4f17-8781-6aedfe47d607 nodeName:}" failed. No retries permitted until 2026-10-30 16:30:23.12946137 +0000 UTC m=+578.656796571 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-cert" (UniqueName: "kubernetes.io/secret/f3dcbff0-3582-4f17-8781-6aedfe47d607-tls-cert") pod "usage-logs-tenancy-proxy-7d94f55f4b-4f74l" (UID: "f3dcbff0-3582-4f17-8781-6aedfe47d607") : secret "usage-logs-tenancy-proxy-tls" not found Oct 30 16:30:22.629854 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:22.629834 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f3dcbff0-3582-4f17-8781-6aedfe47d607-service-ca\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-4f74l\" (UID: \"f3dcbff0-3582-4f17-8781-6aedfe47d607\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-4f74l" Oct 30 16:30:22.655348 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:22.655312 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jnxz\" (UniqueName: \"kubernetes.io/projected/f3dcbff0-3582-4f17-8781-6aedfe47d607-kube-api-access-9jnxz\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-4f74l\" (UID: \"f3dcbff0-3582-4f17-8781-6aedfe47d607\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-4f74l" Oct 30 16:30:23.134426 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:23.134389 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f3dcbff0-3582-4f17-8781-6aedfe47d607-tls-cert\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-4f74l\" (UID: \"f3dcbff0-3582-4f17-8781-6aedfe47d607\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-4f74l" Oct 30 16:30:23.134630 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:23.134454 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/f3dcbff0-3582-4f17-8781-6aedfe47d607-proxy-source\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-4f74l\" (UID: \"f3dcbff0-3582-4f17-8781-6aedfe47d607\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-4f74l" Oct 30 16:30:23.135209 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:23.135186 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/f3dcbff0-3582-4f17-8781-6aedfe47d607-proxy-source\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-4f74l\" (UID: \"f3dcbff0-3582-4f17-8781-6aedfe47d607\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-4f74l" Oct 30 16:30:23.136954 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:23.136932 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f3dcbff0-3582-4f17-8781-6aedfe47d607-tls-cert\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-4f74l\" (UID: \"f3dcbff0-3582-4f17-8781-6aedfe47d607\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-4f74l" Oct 30 16:30:23.435914 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:23.435884 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-4f74l" Oct 30 16:30:23.736522 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:23.736494 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-4f74l"] Oct 30 16:30:23.737687 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:30:23.737656 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3dcbff0_3582_4f17_8781_6aedfe47d607.slice/crio-4e723970d2600617da98f8dc8b898e6fad57da060f0b3c5aa4b09e6d95b157bf WatchSource:0}: Error finding container 4e723970d2600617da98f8dc8b898e6fad57da060f0b3c5aa4b09e6d95b157bf: Status 404 returned error can't find the container with id 4e723970d2600617da98f8dc8b898e6fad57da060f0b3c5aa4b09e6d95b157bf Oct 30 16:30:24.266700 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:24.266668 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-4f74l" event={"ID":"f3dcbff0-3582-4f17-8781-6aedfe47d607","Type":"ContainerStarted","Data":"4e723970d2600617da98f8dc8b898e6fad57da060f0b3c5aa4b09e6d95b157bf"} Oct 30 16:30:30.248535 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:30.248501 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-577f596567-h5692" Oct 30 16:30:30.249137 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:30.248605 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-575c7b5c5b-lh44m" Oct 30 16:30:30.375556 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:30.375514 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-577f596567-h5692"] Oct 30 16:30:30.375863 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:30.375830 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-577f596567-h5692" podUID="95d70dec-072d-4521-a554-fc167a1d8bd1" containerName="manager" containerID="cri-o://21814cf179775f76499c8fbaef357f6d30ef60f74e0d5d1d6487972a79193a6b" gracePeriod=30 Oct 30 16:30:30.642977 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:30.642947 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-577f596567-h5692" Oct 30 16:30:30.712988 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:30.712959 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95d70dec-072d-4521-a554-fc167a1d8bd1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/95d70dec-072d-4521-a554-fc167a1d8bd1-metrics-certs\") pod \"95d70dec-072d-4521-a554-fc167a1d8bd1\" (UID: \"95d70dec-072d-4521-a554-fc167a1d8bd1\") " Oct 30 16:30:30.712988 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:30.713000 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95d70dec-072d-4521-a554-fc167a1d8bd1-kube-api-access-r4kh7\" (UniqueName: \"kubernetes.io/projected/95d70dec-072d-4521-a554-fc167a1d8bd1-kube-api-access-r4kh7\") pod \"95d70dec-072d-4521-a554-fc167a1d8bd1\" (UID: \"95d70dec-072d-4521-a554-fc167a1d8bd1\") " Oct 30 16:30:30.713204 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:30.713066 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95d70dec-072d-4521-a554-fc167a1d8bd1-webhook-cert\" (UniqueName: \"kubernetes.io/secret/95d70dec-072d-4521-a554-fc167a1d8bd1-webhook-cert\") pod \"95d70dec-072d-4521-a554-fc167a1d8bd1\" (UID: \"95d70dec-072d-4521-a554-fc167a1d8bd1\") " Oct 30 16:30:30.715900 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:30.715850 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95d70dec-072d-4521-a554-fc167a1d8bd1-kube-api-access-r4kh7" pod "95d70dec-072d-4521-a554-fc167a1d8bd1" (UID: "95d70dec-072d-4521-a554-fc167a1d8bd1"). InnerVolumeSpecName "kube-api-access-r4kh7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:30:30.716026 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:30.715996 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95d70dec-072d-4521-a554-fc167a1d8bd1-webhook-cert" pod "95d70dec-072d-4521-a554-fc167a1d8bd1" (UID: "95d70dec-072d-4521-a554-fc167a1d8bd1"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:30:30.716188 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:30.716161 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95d70dec-072d-4521-a554-fc167a1d8bd1-metrics-certs" pod "95d70dec-072d-4521-a554-fc167a1d8bd1" (UID: "95d70dec-072d-4521-a554-fc167a1d8bd1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:30:30.814269 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:30.814186 5052 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95d70dec-072d-4521-a554-fc167a1d8bd1-metrics-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:30:30.814269 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:30.814218 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4kh7\" (UniqueName: \"kubernetes.io/projected/95d70dec-072d-4521-a554-fc167a1d8bd1-kube-api-access-r4kh7\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:30:30.814269 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:30.814228 5052 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/95d70dec-072d-4521-a554-fc167a1d8bd1-webhook-cert\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:30:31.304989 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:31.304954 5052 generic.go:356] "Generic (PLEG): container finished" podID="95d70dec-072d-4521-a554-fc167a1d8bd1" containerID="21814cf179775f76499c8fbaef357f6d30ef60f74e0d5d1d6487972a79193a6b" exitCode=0 Oct 30 16:30:31.305516 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:31.305018 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-577f596567-h5692" Oct 30 16:30:31.305516 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:31.305022 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-577f596567-h5692" event={"ID":"95d70dec-072d-4521-a554-fc167a1d8bd1","Type":"ContainerDied","Data":"21814cf179775f76499c8fbaef357f6d30ef60f74e0d5d1d6487972a79193a6b"} Oct 30 16:30:31.305516 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:31.305111 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-577f596567-h5692" event={"ID":"95d70dec-072d-4521-a554-fc167a1d8bd1","Type":"ContainerDied","Data":"1079ac575f81e64f99d0e6dfb9380bb3ea0f174b41d5ed4cb1946c9a185d01e0"} Oct 30 16:30:31.305516 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:31.305135 5052 scope.go:122] "RemoveContainer" containerID="21814cf179775f76499c8fbaef357f6d30ef60f74e0d5d1d6487972a79193a6b" Oct 30 16:30:31.315066 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:31.315049 5052 scope.go:122] "RemoveContainer" containerID="21814cf179775f76499c8fbaef357f6d30ef60f74e0d5d1d6487972a79193a6b" Oct 30 16:30:31.315336 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:30:31.315318 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21814cf179775f76499c8fbaef357f6d30ef60f74e0d5d1d6487972a79193a6b\": container with ID starting with 21814cf179775f76499c8fbaef357f6d30ef60f74e0d5d1d6487972a79193a6b not found: ID does not exist" containerID="21814cf179775f76499c8fbaef357f6d30ef60f74e0d5d1d6487972a79193a6b" Oct 30 16:30:31.315404 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:31.315342 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21814cf179775f76499c8fbaef357f6d30ef60f74e0d5d1d6487972a79193a6b"} err="failed to get container status \"21814cf179775f76499c8fbaef357f6d30ef60f74e0d5d1d6487972a79193a6b\": rpc error: code = NotFound desc = could not find container \"21814cf179775f76499c8fbaef357f6d30ef60f74e0d5d1d6487972a79193a6b\": container with ID starting with 21814cf179775f76499c8fbaef357f6d30ef60f74e0d5d1d6487972a79193a6b not found: ID does not exist" Oct 30 16:30:31.362684 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:31.362648 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-577f596567-h5692"] Oct 30 16:30:31.376229 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:31.376204 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-577f596567-h5692"] Oct 30 16:30:32.596407 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:32.596377 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95d70dec-072d-4521-a554-fc167a1d8bd1" path="/var/lib/kubelet/pods/95d70dec-072d-4521-a554-fc167a1d8bd1/volumes" Oct 30 16:30:35.327085 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:35.327043 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-4f74l" event={"ID":"f3dcbff0-3582-4f17-8781-6aedfe47d607","Type":"ContainerStarted","Data":"ebcf008b74dcf3899c80ba6d4c835ea5d07d192a898a047ae3e930481b05084d"} Oct 30 16:30:35.327514 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:35.327204 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-4f74l" Oct 30 16:30:35.373047 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:35.372996 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-4f74l" podStartSLOduration=2.684832642 podStartE2EDuration="13.37298118s" totalImagesPullingTime="10.688148538s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:30:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:30:35.371839774 +0000 UTC m=+590.899174993" watchObservedRunningTime="2026-10-30 16:30:35.37298118 +0000 UTC m=+590.900316403" Oct 30 16:30:44.648064 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:44.648037 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4wrf8_4f0f46f4-5cda-4d67-b6f3-47df06eb844c/console-operator/2.log" Oct 30 16:30:44.648498 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:44.648394 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4wrf8_4f0f46f4-5cda-4d67-b6f3-47df06eb844c/console-operator/2.log" Oct 30 16:30:46.341489 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:46.341459 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-4f74l" Oct 30 16:30:48.838456 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:30:48.838421 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 16:30:56.717772 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:56.717739 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-6xr5r"] Oct 30 16:30:56.718361 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:56.718270 5052 cpu_manager.go:414] "containerMap: removing container" podUID="95d70dec-072d-4521-a554-fc167a1d8bd1" containerName="manager" Oct 30 16:30:56.718361 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:56.718289 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95d70dec-072d-4521-a554-fc167a1d8bd1" containerName="manager" Oct 30 16:30:56.718480 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:56.718415 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="95d70dec-072d-4521-a554-fc167a1d8bd1" containerName="manager" Oct 30 16:30:56.890998 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:56.890968 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-6xr5r"] Oct 30 16:30:56.891157 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:56.891076 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-6xr5r" Oct 30 16:30:56.907829 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:56.907806 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:30:56.921817 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:56.921797 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 30 16:30:56.921931 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:56.921909 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-5jjc6\"" Oct 30 16:30:56.921997 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:56.921983 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:30:56.922067 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:56.922050 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 16:30:57.031906 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:57.031806 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fr2gv\" (UniqueName: \"kubernetes.io/projected/c03b7d08-8589-49f4-a662-c7bf1d9c8caf-kube-api-access-fr2gv\") pod \"maas-api-5c48877c75-6xr5r\" (UID: \"c03b7d08-8589-49f4-a662-c7bf1d9c8caf\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-6xr5r" Oct 30 16:30:57.032068 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:57.031934 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c03b7d08-8589-49f4-a662-c7bf1d9c8caf-openshift-service-ca\") pod \"maas-api-5c48877c75-6xr5r\" (UID: \"c03b7d08-8589-49f4-a662-c7bf1d9c8caf\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-6xr5r" Oct 30 16:30:57.032068 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:57.031967 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c03b7d08-8589-49f4-a662-c7bf1d9c8caf-metrics-certs\") pod \"maas-api-5c48877c75-6xr5r\" (UID: \"c03b7d08-8589-49f4-a662-c7bf1d9c8caf\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-6xr5r" Oct 30 16:30:57.032191 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:57.032114 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c03b7d08-8589-49f4-a662-c7bf1d9c8caf-maas-api-tls\") pod \"maas-api-5c48877c75-6xr5r\" (UID: \"c03b7d08-8589-49f4-a662-c7bf1d9c8caf\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-6xr5r" Oct 30 16:30:57.133251 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:57.133211 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c03b7d08-8589-49f4-a662-c7bf1d9c8caf-maas-api-tls\") pod \"maas-api-5c48877c75-6xr5r\" (UID: \"c03b7d08-8589-49f4-a662-c7bf1d9c8caf\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-6xr5r" Oct 30 16:30:57.133251 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:57.133256 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fr2gv\" (UniqueName: \"kubernetes.io/projected/c03b7d08-8589-49f4-a662-c7bf1d9c8caf-kube-api-access-fr2gv\") pod \"maas-api-5c48877c75-6xr5r\" (UID: \"c03b7d08-8589-49f4-a662-c7bf1d9c8caf\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-6xr5r" Oct 30 16:30:57.133498 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:57.133287 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c03b7d08-8589-49f4-a662-c7bf1d9c8caf-openshift-service-ca\") pod \"maas-api-5c48877c75-6xr5r\" (UID: \"c03b7d08-8589-49f4-a662-c7bf1d9c8caf\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-6xr5r" Oct 30 16:30:57.133498 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:57.133307 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c03b7d08-8589-49f4-a662-c7bf1d9c8caf-metrics-certs\") pod \"maas-api-5c48877c75-6xr5r\" (UID: \"c03b7d08-8589-49f4-a662-c7bf1d9c8caf\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-6xr5r" Oct 30 16:30:57.134021 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:57.134001 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c03b7d08-8589-49f4-a662-c7bf1d9c8caf-openshift-service-ca\") pod \"maas-api-5c48877c75-6xr5r\" (UID: \"c03b7d08-8589-49f4-a662-c7bf1d9c8caf\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-6xr5r" Oct 30 16:30:57.135793 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:57.135756 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c03b7d08-8589-49f4-a662-c7bf1d9c8caf-metrics-certs\") pod \"maas-api-5c48877c75-6xr5r\" (UID: \"c03b7d08-8589-49f4-a662-c7bf1d9c8caf\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-6xr5r" Oct 30 16:30:57.135880 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:57.135806 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c03b7d08-8589-49f4-a662-c7bf1d9c8caf-maas-api-tls\") pod \"maas-api-5c48877c75-6xr5r\" (UID: \"c03b7d08-8589-49f4-a662-c7bf1d9c8caf\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-6xr5r" Oct 30 16:30:57.158327 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:57.158301 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fr2gv\" (UniqueName: \"kubernetes.io/projected/c03b7d08-8589-49f4-a662-c7bf1d9c8caf-kube-api-access-fr2gv\") pod \"maas-api-5c48877c75-6xr5r\" (UID: \"c03b7d08-8589-49f4-a662-c7bf1d9c8caf\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-6xr5r" Oct 30 16:30:57.213487 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:57.213460 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-6xr5r" Oct 30 16:30:57.527458 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:57.527432 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-6xr5r"] Oct 30 16:30:57.528908 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:30:57.528883 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc03b7d08_8589_49f4_a662_c7bf1d9c8caf.slice/crio-1e8edf057a96b786e204885cc7258951310f8063b0131fcc42201807812841c1 WatchSource:0}: Error finding container 1e8edf057a96b786e204885cc7258951310f8063b0131fcc42201807812841c1: Status 404 returned error can't find the container with id 1e8edf057a96b786e204885cc7258951310f8063b0131fcc42201807812841c1 Oct 30 16:30:57.530225 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:57.530206 5052 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 16:30:58.438931 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:58.438896 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-6xr5r" event={"ID":"c03b7d08-8589-49f4-a662-c7bf1d9c8caf","Type":"ContainerStarted","Data":"1e8edf057a96b786e204885cc7258951310f8063b0131fcc42201807812841c1"} Oct 30 16:30:58.535357 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:30:58.535323 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cdwcj"] Oct 30 16:31:00.448842 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:31:00.448806 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-6xr5r" event={"ID":"c03b7d08-8589-49f4-a662-c7bf1d9c8caf","Type":"ContainerStarted","Data":"ac804409ebd126acbe5d4205c1d4973451a0f045cba8a290ff9dbc671a0bb7ac"} Oct 30 16:31:00.449236 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:31:00.448910 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5c48877c75-6xr5r" Oct 30 16:31:00.498202 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:31:00.498159 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5c48877c75-6xr5r" podStartSLOduration=2.622254492 podStartE2EDuration="4.498147617s" totalImagesPullingTime="1.875893125s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:30:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:31:00.497516466 +0000 UTC m=+616.024851709" watchObservedRunningTime="2026-10-30 16:31:00.498147617 +0000 UTC m=+616.025482836" Oct 30 16:31:06.458467 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:31:06.458436 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5c48877c75-6xr5r" Oct 30 16:31:18.838965 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:31:18.838920 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 16:31:48.838532 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:31:48.838453 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 16:31:59.100168 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:31:59.100135 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4p2lm"] Oct 30 16:31:59.103779 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:31:59.103753 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4p2lm" Oct 30 16:31:59.120704 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:31:59.120678 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 30 16:31:59.134836 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:31:59.134808 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 30 16:31:59.135034 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:31:59.135011 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Oct 30 16:31:59.135114 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:31:59.135091 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-w67vc\"" Oct 30 16:31:59.136542 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:31:59.136517 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4p2lm"] Oct 30 16:31:59.272827 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:31:59.272794 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4w8m\" (UniqueName: \"kubernetes.io/projected/6ba3d48f-8737-4e17-a90b-89c3f7b41c59-kube-api-access-k4w8m\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4p2lm\" (UID: \"6ba3d48f-8737-4e17-a90b-89c3f7b41c59\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4p2lm" Oct 30 16:31:59.273090 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:31:59.272847 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ba3d48f-8737-4e17-a90b-89c3f7b41c59-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4p2lm\" (UID: \"6ba3d48f-8737-4e17-a90b-89c3f7b41c59\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4p2lm" Oct 30 16:31:59.273090 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:31:59.272965 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ba3d48f-8737-4e17-a90b-89c3f7b41c59-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4p2lm\" (UID: \"6ba3d48f-8737-4e17-a90b-89c3f7b41c59\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4p2lm" Oct 30 16:31:59.273090 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:31:59.273001 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ba3d48f-8737-4e17-a90b-89c3f7b41c59-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4p2lm\" (UID: \"6ba3d48f-8737-4e17-a90b-89c3f7b41c59\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4p2lm" Oct 30 16:31:59.273090 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:31:59.273018 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ba3d48f-8737-4e17-a90b-89c3f7b41c59-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4p2lm\" (UID: \"6ba3d48f-8737-4e17-a90b-89c3f7b41c59\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4p2lm" Oct 30 16:31:59.273090 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:31:59.273034 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ba3d48f-8737-4e17-a90b-89c3f7b41c59-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4p2lm\" (UID: \"6ba3d48f-8737-4e17-a90b-89c3f7b41c59\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4p2lm" Oct 30 16:31:59.373830 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:31:59.373755 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4w8m\" (UniqueName: \"kubernetes.io/projected/6ba3d48f-8737-4e17-a90b-89c3f7b41c59-kube-api-access-k4w8m\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4p2lm\" (UID: \"6ba3d48f-8737-4e17-a90b-89c3f7b41c59\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4p2lm" Oct 30 16:31:59.373830 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:31:59.373797 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ba3d48f-8737-4e17-a90b-89c3f7b41c59-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4p2lm\" (UID: \"6ba3d48f-8737-4e17-a90b-89c3f7b41c59\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4p2lm" Oct 30 16:31:59.373830 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:31:59.373826 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ba3d48f-8737-4e17-a90b-89c3f7b41c59-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4p2lm\" (UID: \"6ba3d48f-8737-4e17-a90b-89c3f7b41c59\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4p2lm" Oct 30 16:31:59.374095 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:31:59.373846 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ba3d48f-8737-4e17-a90b-89c3f7b41c59-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4p2lm\" (UID: \"6ba3d48f-8737-4e17-a90b-89c3f7b41c59\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4p2lm" Oct 30 16:31:59.374095 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:31:59.373861 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ba3d48f-8737-4e17-a90b-89c3f7b41c59-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4p2lm\" (UID: \"6ba3d48f-8737-4e17-a90b-89c3f7b41c59\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4p2lm" Oct 30 16:31:59.374095 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:31:59.373876 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ba3d48f-8737-4e17-a90b-89c3f7b41c59-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4p2lm\" (UID: \"6ba3d48f-8737-4e17-a90b-89c3f7b41c59\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4p2lm" Oct 30 16:31:59.374210 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:31:59.374187 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ba3d48f-8737-4e17-a90b-89c3f7b41c59-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4p2lm\" (UID: \"6ba3d48f-8737-4e17-a90b-89c3f7b41c59\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4p2lm" Oct 30 16:31:59.374257 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:31:59.374235 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ba3d48f-8737-4e17-a90b-89c3f7b41c59-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4p2lm\" (UID: \"6ba3d48f-8737-4e17-a90b-89c3f7b41c59\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4p2lm" Oct 30 16:31:59.374355 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:31:59.374330 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ba3d48f-8737-4e17-a90b-89c3f7b41c59-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4p2lm\" (UID: \"6ba3d48f-8737-4e17-a90b-89c3f7b41c59\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4p2lm" Oct 30 16:31:59.376186 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:31:59.376160 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ba3d48f-8737-4e17-a90b-89c3f7b41c59-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4p2lm\" (UID: \"6ba3d48f-8737-4e17-a90b-89c3f7b41c59\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4p2lm" Oct 30 16:31:59.376422 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:31:59.376408 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ba3d48f-8737-4e17-a90b-89c3f7b41c59-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4p2lm\" (UID: \"6ba3d48f-8737-4e17-a90b-89c3f7b41c59\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4p2lm" Oct 30 16:31:59.398999 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:31:59.398977 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4w8m\" (UniqueName: \"kubernetes.io/projected/6ba3d48f-8737-4e17-a90b-89c3f7b41c59-kube-api-access-k4w8m\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4p2lm\" (UID: \"6ba3d48f-8737-4e17-a90b-89c3f7b41c59\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4p2lm" Oct 30 16:31:59.416919 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:31:59.416897 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4p2lm" Oct 30 16:31:59.718089 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:31:59.718066 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4p2lm"] Oct 30 16:31:59.720212 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:31:59.720184 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ba3d48f_8737_4e17_a90b_89c3f7b41c59.slice/crio-df59cbf19e673f07398519dabab460d6ccefd4de827e67b95dcb185ec156cad9 WatchSource:0}: Error finding container df59cbf19e673f07398519dabab460d6ccefd4de827e67b95dcb185ec156cad9: Status 404 returned error can't find the container with id df59cbf19e673f07398519dabab460d6ccefd4de827e67b95dcb185ec156cad9 Oct 30 16:32:00.365229 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:00.365190 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cdwcj"] Oct 30 16:32:00.711417 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:00.711380 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4p2lm" event={"ID":"6ba3d48f-8737-4e17-a90b-89c3f7b41c59","Type":"ContainerStarted","Data":"df59cbf19e673f07398519dabab460d6ccefd4de827e67b95dcb185ec156cad9"} Oct 30 16:32:01.718298 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:01.718262 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4p2lm" event={"ID":"6ba3d48f-8737-4e17-a90b-89c3f7b41c59","Type":"ContainerStarted","Data":"e55c1e9fc012c95439c7dc8a72ec645ca2123df633474ad3dacfafe7ed9c0125"} Oct 30 16:32:01.764395 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:01.764339 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4p2lm" podStartSLOduration=1.724334978 podStartE2EDuration="2.764325791s" totalImagesPullingTime="1.039990813s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:31:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:32:01.763885103 +0000 UTC m=+677.291220341" watchObservedRunningTime="2026-10-30 16:32:01.764325791 +0000 UTC m=+677.291661010" Oct 30 16:32:02.417833 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:02.417799 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4p2lm" Oct 30 16:32:03.095605 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:03.095552 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-62cvz"] Oct 30 16:32:03.099900 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:03.099876 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-62cvz" Oct 30 16:32:03.124761 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:03.124735 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-62cvz"] Oct 30 16:32:03.171999 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:03.171969 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Oct 30 16:32:03.212481 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:03.212455 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/730923b8-cd00-4831-91d2-b4b0071ae57b-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-62cvz\" (UID: \"730923b8-cd00-4831-91d2-b4b0071ae57b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-62cvz" Oct 30 16:32:03.212481 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:03.212483 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/730923b8-cd00-4831-91d2-b4b0071ae57b-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-62cvz\" (UID: \"730923b8-cd00-4831-91d2-b4b0071ae57b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-62cvz" Oct 30 16:32:03.212725 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:03.212539 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/730923b8-cd00-4831-91d2-b4b0071ae57b-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-62cvz\" (UID: \"730923b8-cd00-4831-91d2-b4b0071ae57b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-62cvz" Oct 30 16:32:03.212725 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:03.212566 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fg4qd\" (UniqueName: \"kubernetes.io/projected/730923b8-cd00-4831-91d2-b4b0071ae57b-kube-api-access-fg4qd\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-62cvz\" (UID: \"730923b8-cd00-4831-91d2-b4b0071ae57b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-62cvz" Oct 30 16:32:03.212725 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:03.212621 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/730923b8-cd00-4831-91d2-b4b0071ae57b-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-62cvz\" (UID: \"730923b8-cd00-4831-91d2-b4b0071ae57b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-62cvz" Oct 30 16:32:03.212885 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:03.212748 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/730923b8-cd00-4831-91d2-b4b0071ae57b-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-62cvz\" (UID: \"730923b8-cd00-4831-91d2-b4b0071ae57b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-62cvz" Oct 30 16:32:03.313411 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:03.313377 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/730923b8-cd00-4831-91d2-b4b0071ae57b-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-62cvz\" (UID: \"730923b8-cd00-4831-91d2-b4b0071ae57b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-62cvz" Oct 30 16:32:03.313564 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:03.313424 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/730923b8-cd00-4831-91d2-b4b0071ae57b-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-62cvz\" (UID: \"730923b8-cd00-4831-91d2-b4b0071ae57b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-62cvz" Oct 30 16:32:03.313564 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:03.313474 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/730923b8-cd00-4831-91d2-b4b0071ae57b-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-62cvz\" (UID: \"730923b8-cd00-4831-91d2-b4b0071ae57b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-62cvz" Oct 30 16:32:03.313564 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:03.313495 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/730923b8-cd00-4831-91d2-b4b0071ae57b-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-62cvz\" (UID: \"730923b8-cd00-4831-91d2-b4b0071ae57b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-62cvz" Oct 30 16:32:03.313564 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:03.313527 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/730923b8-cd00-4831-91d2-b4b0071ae57b-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-62cvz\" (UID: \"730923b8-cd00-4831-91d2-b4b0071ae57b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-62cvz" Oct 30 16:32:03.313796 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:03.313566 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fg4qd\" (UniqueName: \"kubernetes.io/projected/730923b8-cd00-4831-91d2-b4b0071ae57b-kube-api-access-fg4qd\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-62cvz\" (UID: \"730923b8-cd00-4831-91d2-b4b0071ae57b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-62cvz" Oct 30 16:32:03.314048 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:03.314000 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/730923b8-cd00-4831-91d2-b4b0071ae57b-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-62cvz\" (UID: \"730923b8-cd00-4831-91d2-b4b0071ae57b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-62cvz" Oct 30 16:32:03.314048 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:03.314009 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/730923b8-cd00-4831-91d2-b4b0071ae57b-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-62cvz\" (UID: \"730923b8-cd00-4831-91d2-b4b0071ae57b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-62cvz" Oct 30 16:32:03.314159 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:03.314063 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/730923b8-cd00-4831-91d2-b4b0071ae57b-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-62cvz\" (UID: \"730923b8-cd00-4831-91d2-b4b0071ae57b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-62cvz" Oct 30 16:32:03.315864 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:03.315838 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/730923b8-cd00-4831-91d2-b4b0071ae57b-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-62cvz\" (UID: \"730923b8-cd00-4831-91d2-b4b0071ae57b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-62cvz" Oct 30 16:32:03.316048 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:03.316031 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/730923b8-cd00-4831-91d2-b4b0071ae57b-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-62cvz\" (UID: \"730923b8-cd00-4831-91d2-b4b0071ae57b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-62cvz" Oct 30 16:32:03.338841 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:03.338820 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fg4qd\" (UniqueName: \"kubernetes.io/projected/730923b8-cd00-4831-91d2-b4b0071ae57b-kube-api-access-fg4qd\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-62cvz\" (UID: \"730923b8-cd00-4831-91d2-b4b0071ae57b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-62cvz" Oct 30 16:32:03.415629 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:03.415601 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-62cvz" Oct 30 16:32:03.720518 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:03.720461 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-62cvz"] Oct 30 16:32:03.722678 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:32:03.722649 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod730923b8_cd00_4831_91d2_b4b0071ae57b.slice/crio-82075342b6b6a0f115aa71847a0643fe06f115457e6fa5bbbe4d376610be6b2f WatchSource:0}: Error finding container 82075342b6b6a0f115aa71847a0643fe06f115457e6fa5bbbe4d376610be6b2f: Status 404 returned error can't find the container with id 82075342b6b6a0f115aa71847a0643fe06f115457e6fa5bbbe4d376610be6b2f Oct 30 16:32:04.417295 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:04.417265 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cdwcj"] Oct 30 16:32:04.732474 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:04.732381 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-62cvz" event={"ID":"730923b8-cd00-4831-91d2-b4b0071ae57b","Type":"ContainerStarted","Data":"c211b3468b0113ecdcf91defb0b6c12d4e7967b0c24a1ad80e27ca4c0fefc815"} Oct 30 16:32:04.732474 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:04.732440 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-62cvz" event={"ID":"730923b8-cd00-4831-91d2-b4b0071ae57b","Type":"ContainerStarted","Data":"82075342b6b6a0f115aa71847a0643fe06f115457e6fa5bbbe4d376610be6b2f"} Oct 30 16:32:04.780619 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:04.780550 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-62cvz" podStartSLOduration=1.588159384 podStartE2EDuration="1.780536881s" totalImagesPullingTime="192.377497ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:32:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:32:04.778795299 +0000 UTC m=+680.306130546" watchObservedRunningTime="2026-10-30 16:32:04.780536881 +0000 UTC m=+680.307872153" Oct 30 16:32:05.416057 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:05.416027 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-62cvz" Oct 30 16:32:08.545726 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:08.545697 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-lhs5r"] Oct 30 16:32:08.549646 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:08.549623 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lhs5r" Oct 30 16:32:08.570735 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:08.570706 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-lhs5r"] Oct 30 16:32:08.579851 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:08.579831 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Oct 30 16:32:08.659633 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:08.659598 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84448e7a-d354-4820-b6a9-df2183c1b9c2-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-lhs5r\" (UID: \"84448e7a-d354-4820-b6a9-df2183c1b9c2\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lhs5r" Oct 30 16:32:08.659802 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:08.659637 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84448e7a-d354-4820-b6a9-df2183c1b9c2-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-lhs5r\" (UID: \"84448e7a-d354-4820-b6a9-df2183c1b9c2\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lhs5r" Oct 30 16:32:08.659802 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:08.659718 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84448e7a-d354-4820-b6a9-df2183c1b9c2-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-lhs5r\" (UID: \"84448e7a-d354-4820-b6a9-df2183c1b9c2\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lhs5r" Oct 30 16:32:08.659914 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:08.659813 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84448e7a-d354-4820-b6a9-df2183c1b9c2-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-lhs5r\" (UID: \"84448e7a-d354-4820-b6a9-df2183c1b9c2\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lhs5r" Oct 30 16:32:08.659914 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:08.659850 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84448e7a-d354-4820-b6a9-df2183c1b9c2-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-lhs5r\" (UID: \"84448e7a-d354-4820-b6a9-df2183c1b9c2\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lhs5r" Oct 30 16:32:08.660021 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:08.659917 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbnps\" (UniqueName: \"kubernetes.io/projected/84448e7a-d354-4820-b6a9-df2183c1b9c2-kube-api-access-nbnps\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-lhs5r\" (UID: \"84448e7a-d354-4820-b6a9-df2183c1b9c2\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lhs5r" Oct 30 16:32:08.760753 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:08.760722 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84448e7a-d354-4820-b6a9-df2183c1b9c2-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-lhs5r\" (UID: \"84448e7a-d354-4820-b6a9-df2183c1b9c2\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lhs5r" Oct 30 16:32:08.761095 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:08.760770 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84448e7a-d354-4820-b6a9-df2183c1b9c2-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-lhs5r\" (UID: \"84448e7a-d354-4820-b6a9-df2183c1b9c2\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lhs5r" Oct 30 16:32:08.761095 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:08.760787 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84448e7a-d354-4820-b6a9-df2183c1b9c2-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-lhs5r\" (UID: \"84448e7a-d354-4820-b6a9-df2183c1b9c2\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lhs5r" Oct 30 16:32:08.761095 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:08.760836 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84448e7a-d354-4820-b6a9-df2183c1b9c2-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-lhs5r\" (UID: \"84448e7a-d354-4820-b6a9-df2183c1b9c2\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lhs5r" Oct 30 16:32:08.761095 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:08.760895 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84448e7a-d354-4820-b6a9-df2183c1b9c2-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-lhs5r\" (UID: \"84448e7a-d354-4820-b6a9-df2183c1b9c2\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lhs5r" Oct 30 16:32:08.761380 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:08.761251 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84448e7a-d354-4820-b6a9-df2183c1b9c2-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-lhs5r\" (UID: \"84448e7a-d354-4820-b6a9-df2183c1b9c2\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lhs5r" Oct 30 16:32:08.761380 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:08.761368 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbnps\" (UniqueName: \"kubernetes.io/projected/84448e7a-d354-4820-b6a9-df2183c1b9c2-kube-api-access-nbnps\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-lhs5r\" (UID: \"84448e7a-d354-4820-b6a9-df2183c1b9c2\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lhs5r" Oct 30 16:32:08.761464 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:08.761443 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84448e7a-d354-4820-b6a9-df2183c1b9c2-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-lhs5r\" (UID: \"84448e7a-d354-4820-b6a9-df2183c1b9c2\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lhs5r" Oct 30 16:32:08.761504 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:08.761459 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84448e7a-d354-4820-b6a9-df2183c1b9c2-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-lhs5r\" (UID: \"84448e7a-d354-4820-b6a9-df2183c1b9c2\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lhs5r" Oct 30 16:32:08.763257 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:08.763229 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84448e7a-d354-4820-b6a9-df2183c1b9c2-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-lhs5r\" (UID: \"84448e7a-d354-4820-b6a9-df2183c1b9c2\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lhs5r" Oct 30 16:32:08.763426 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:08.763408 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84448e7a-d354-4820-b6a9-df2183c1b9c2-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-lhs5r\" (UID: \"84448e7a-d354-4820-b6a9-df2183c1b9c2\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lhs5r" Oct 30 16:32:08.786281 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:08.786255 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbnps\" (UniqueName: \"kubernetes.io/projected/84448e7a-d354-4820-b6a9-df2183c1b9c2-kube-api-access-nbnps\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-lhs5r\" (UID: \"84448e7a-d354-4820-b6a9-df2183c1b9c2\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lhs5r" Oct 30 16:32:08.863255 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:08.863198 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lhs5r" Oct 30 16:32:09.008320 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:09.008289 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cdwcj"] Oct 30 16:32:09.164437 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:09.164407 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-lhs5r"] Oct 30 16:32:09.166757 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:32:09.166725 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84448e7a_d354_4820_b6a9_df2183c1b9c2.slice/crio-0f3e58efcd3ffdb764f07b7b0965274dd0a4d3649d91aac4135f9905476bcdfc WatchSource:0}: Error finding container 0f3e58efcd3ffdb764f07b7b0965274dd0a4d3649d91aac4135f9905476bcdfc: Status 404 returned error can't find the container with id 0f3e58efcd3ffdb764f07b7b0965274dd0a4d3649d91aac4135f9905476bcdfc Oct 30 16:32:09.417815 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:09.417787 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4p2lm" Oct 30 16:32:09.431010 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:09.430991 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4p2lm" Oct 30 16:32:09.755871 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:09.755771 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lhs5r" event={"ID":"84448e7a-d354-4820-b6a9-df2183c1b9c2","Type":"ContainerStarted","Data":"bfb63b34b4bff5afc17d151960e9fe146f50c677401cfbe87c2536df6a24237d"} Oct 30 16:32:09.755871 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:09.755827 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lhs5r" event={"ID":"84448e7a-d354-4820-b6a9-df2183c1b9c2","Type":"ContainerStarted","Data":"0f3e58efcd3ffdb764f07b7b0965274dd0a4d3649d91aac4135f9905476bcdfc"} Oct 30 16:32:09.766902 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:09.766877 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4p2lm" Oct 30 16:32:09.802422 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:09.802364 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lhs5r" podStartSLOduration=1.600230176 podStartE2EDuration="1.802350043s" totalImagesPullingTime="202.119867ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:32:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:32:09.801784579 +0000 UTC m=+685.329119799" watchObservedRunningTime="2026-10-30 16:32:09.802350043 +0000 UTC m=+685.329685258" Oct 30 16:32:09.864343 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:09.864315 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lhs5r" Oct 30 16:32:10.239499 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:10.239462 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89845t"] Oct 30 16:32:10.243456 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:10.243433 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89845t" Oct 30 16:32:10.264985 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:10.264958 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89845t"] Oct 30 16:32:10.273779 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:10.273755 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2e67399-be2a-4721-a2ca-2128d7dfa09a-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89845t\" (UID: \"c2e67399-be2a-4721-a2ca-2128d7dfa09a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89845t" Oct 30 16:32:10.273904 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:10.273785 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltmhs\" (UniqueName: \"kubernetes.io/projected/c2e67399-be2a-4721-a2ca-2128d7dfa09a-kube-api-access-ltmhs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89845t\" (UID: \"c2e67399-be2a-4721-a2ca-2128d7dfa09a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89845t" Oct 30 16:32:10.273985 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:10.273925 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2e67399-be2a-4721-a2ca-2128d7dfa09a-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89845t\" (UID: \"c2e67399-be2a-4721-a2ca-2128d7dfa09a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89845t" Oct 30 16:32:10.273985 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:10.273972 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2e67399-be2a-4721-a2ca-2128d7dfa09a-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89845t\" (UID: \"c2e67399-be2a-4721-a2ca-2128d7dfa09a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89845t" Oct 30 16:32:10.274116 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:10.274013 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2e67399-be2a-4721-a2ca-2128d7dfa09a-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89845t\" (UID: \"c2e67399-be2a-4721-a2ca-2128d7dfa09a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89845t" Oct 30 16:32:10.274173 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:10.274128 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2e67399-be2a-4721-a2ca-2128d7dfa09a-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89845t\" (UID: \"c2e67399-be2a-4721-a2ca-2128d7dfa09a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89845t" Oct 30 16:32:10.276801 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:10.276784 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Oct 30 16:32:10.375124 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:10.375094 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2e67399-be2a-4721-a2ca-2128d7dfa09a-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89845t\" (UID: \"c2e67399-be2a-4721-a2ca-2128d7dfa09a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89845t" Oct 30 16:32:10.375294 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:10.375133 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2e67399-be2a-4721-a2ca-2128d7dfa09a-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89845t\" (UID: \"c2e67399-be2a-4721-a2ca-2128d7dfa09a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89845t" Oct 30 16:32:10.375294 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:10.375168 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2e67399-be2a-4721-a2ca-2128d7dfa09a-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89845t\" (UID: \"c2e67399-be2a-4721-a2ca-2128d7dfa09a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89845t" Oct 30 16:32:10.375294 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:10.375217 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2e67399-be2a-4721-a2ca-2128d7dfa09a-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89845t\" (UID: \"c2e67399-be2a-4721-a2ca-2128d7dfa09a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89845t" Oct 30 16:32:10.375294 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:10.375276 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2e67399-be2a-4721-a2ca-2128d7dfa09a-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89845t\" (UID: \"c2e67399-be2a-4721-a2ca-2128d7dfa09a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89845t" Oct 30 16:32:10.375518 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:10.375301 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltmhs\" (UniqueName: \"kubernetes.io/projected/c2e67399-be2a-4721-a2ca-2128d7dfa09a-kube-api-access-ltmhs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89845t\" (UID: \"c2e67399-be2a-4721-a2ca-2128d7dfa09a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89845t" Oct 30 16:32:10.375571 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:10.375534 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2e67399-be2a-4721-a2ca-2128d7dfa09a-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89845t\" (UID: \"c2e67399-be2a-4721-a2ca-2128d7dfa09a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89845t" Oct 30 16:32:10.375685 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:10.375656 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2e67399-be2a-4721-a2ca-2128d7dfa09a-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89845t\" (UID: \"c2e67399-be2a-4721-a2ca-2128d7dfa09a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89845t" Oct 30 16:32:10.375896 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:10.375875 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2e67399-be2a-4721-a2ca-2128d7dfa09a-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89845t\" (UID: \"c2e67399-be2a-4721-a2ca-2128d7dfa09a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89845t" Oct 30 16:32:10.377591 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:10.377560 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2e67399-be2a-4721-a2ca-2128d7dfa09a-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89845t\" (UID: \"c2e67399-be2a-4721-a2ca-2128d7dfa09a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89845t" Oct 30 16:32:10.377933 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:10.377915 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2e67399-be2a-4721-a2ca-2128d7dfa09a-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89845t\" (UID: \"c2e67399-be2a-4721-a2ca-2128d7dfa09a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89845t" Oct 30 16:32:10.400537 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:10.400508 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltmhs\" (UniqueName: \"kubernetes.io/projected/c2e67399-be2a-4721-a2ca-2128d7dfa09a-kube-api-access-ltmhs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89845t\" (UID: \"c2e67399-be2a-4721-a2ca-2128d7dfa09a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89845t" Oct 30 16:32:10.558027 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:10.557942 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89845t" Oct 30 16:32:10.858229 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:10.858194 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89845t"] Oct 30 16:32:10.859204 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:32:10.859179 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2e67399_be2a_4721_a2ca_2128d7dfa09a.slice/crio-ee838f01497e82bb8e79eb80487055e9298a68488e2e92f38f6c29fe3d76d677 WatchSource:0}: Error finding container ee838f01497e82bb8e79eb80487055e9298a68488e2e92f38f6c29fe3d76d677: Status 404 returned error can't find the container with id ee838f01497e82bb8e79eb80487055e9298a68488e2e92f38f6c29fe3d76d677 Oct 30 16:32:11.767327 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:11.767292 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89845t" event={"ID":"c2e67399-be2a-4721-a2ca-2128d7dfa09a","Type":"ContainerStarted","Data":"2088f6fc7a65abb39bee9687acf78141bdef18e7a54fcbeddc54f1eeb746d027"} Oct 30 16:32:11.767327 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:11.767343 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89845t" event={"ID":"c2e67399-be2a-4721-a2ca-2128d7dfa09a","Type":"ContainerStarted","Data":"ee838f01497e82bb8e79eb80487055e9298a68488e2e92f38f6c29fe3d76d677"} Oct 30 16:32:11.814984 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:11.814937 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89845t" podStartSLOduration=1.615354725 podStartE2EDuration="1.814923024s" totalImagesPullingTime="199.568299ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:32:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:32:11.813239777 +0000 UTC m=+687.340574997" watchObservedRunningTime="2026-10-30 16:32:11.814923024 +0000 UTC m=+687.342258243" Oct 30 16:32:12.558085 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:12.558053 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89845t" Oct 30 16:32:13.416251 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:13.416224 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-62cvz" Oct 30 16:32:13.429335 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:13.429308 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-62cvz" Oct 30 16:32:13.787501 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:13.787427 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-62cvz" Oct 30 16:32:15.181917 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:15.181884 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cdwcj"] Oct 30 16:32:18.838606 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:32:18.838556 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 16:32:18.863970 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:18.863938 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lhs5r" Oct 30 16:32:18.876836 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:18.876815 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lhs5r" Oct 30 16:32:19.816818 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:19.816792 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lhs5r" Oct 30 16:32:20.558147 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:20.558112 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89845t" Oct 30 16:32:20.570837 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:20.570812 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89845t" Oct 30 16:32:20.821146 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:20.821056 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89845t" Oct 30 16:32:27.247948 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:27.247914 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5ddb48fb-kmgjr"] Oct 30 16:32:27.251685 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:27.251669 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5ddb48fb-kmgjr" Oct 30 16:32:27.270946 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:27.270921 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5ddb48fb-kmgjr"] Oct 30 16:32:27.282509 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:27.282482 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-rsnhr\"" Oct 30 16:32:27.282689 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:27.282670 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Oct 30 16:32:27.322910 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:27.322889 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/25d85829-1725-4150-9c59-09c80da4dcab-tls-cert\") pod \"authorino-5ddb48fb-kmgjr\" (UID: \"25d85829-1725-4150-9c59-09c80da4dcab\") " pod="kuadrant-system/authorino-5ddb48fb-kmgjr" Oct 30 16:32:27.323007 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:27.322945 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sdtv\" (UniqueName: \"kubernetes.io/projected/25d85829-1725-4150-9c59-09c80da4dcab-kube-api-access-5sdtv\") pod \"authorino-5ddb48fb-kmgjr\" (UID: \"25d85829-1725-4150-9c59-09c80da4dcab\") " pod="kuadrant-system/authorino-5ddb48fb-kmgjr" Oct 30 16:32:27.423811 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:27.423785 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sdtv\" (UniqueName: \"kubernetes.io/projected/25d85829-1725-4150-9c59-09c80da4dcab-kube-api-access-5sdtv\") pod \"authorino-5ddb48fb-kmgjr\" (UID: \"25d85829-1725-4150-9c59-09c80da4dcab\") " pod="kuadrant-system/authorino-5ddb48fb-kmgjr" Oct 30 16:32:27.423931 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:27.423855 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/25d85829-1725-4150-9c59-09c80da4dcab-tls-cert\") pod \"authorino-5ddb48fb-kmgjr\" (UID: \"25d85829-1725-4150-9c59-09c80da4dcab\") " pod="kuadrant-system/authorino-5ddb48fb-kmgjr" Oct 30 16:32:27.426155 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:27.426134 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/25d85829-1725-4150-9c59-09c80da4dcab-tls-cert\") pod \"authorino-5ddb48fb-kmgjr\" (UID: \"25d85829-1725-4150-9c59-09c80da4dcab\") " pod="kuadrant-system/authorino-5ddb48fb-kmgjr" Oct 30 16:32:27.448516 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:27.448488 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sdtv\" (UniqueName: \"kubernetes.io/projected/25d85829-1725-4150-9c59-09c80da4dcab-kube-api-access-5sdtv\") pod \"authorino-5ddb48fb-kmgjr\" (UID: \"25d85829-1725-4150-9c59-09c80da4dcab\") " pod="kuadrant-system/authorino-5ddb48fb-kmgjr" Oct 30 16:32:27.570208 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:27.570141 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5ddb48fb-kmgjr" Oct 30 16:32:27.868656 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:27.868633 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5ddb48fb-kmgjr"] Oct 30 16:32:27.870841 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:32:27.870813 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod25d85829_1725_4150_9c59_09c80da4dcab.slice/crio-10f086ac239ceb82ef8e25f0d35a6af0639508cb04e069b03ec6fcbfddbd5dd0 WatchSource:0}: Error finding container 10f086ac239ceb82ef8e25f0d35a6af0639508cb04e069b03ec6fcbfddbd5dd0: Status 404 returned error can't find the container with id 10f086ac239ceb82ef8e25f0d35a6af0639508cb04e069b03ec6fcbfddbd5dd0 Oct 30 16:32:28.850275 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:28.850193 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5ddb48fb-kmgjr" event={"ID":"25d85829-1725-4150-9c59-09c80da4dcab","Type":"ContainerStarted","Data":"b182fbc23565610f44f1bfd8a6423d6a7399a0b473ec2499680117e2a69b6cbf"} Oct 30 16:32:28.850275 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:28.850232 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5ddb48fb-kmgjr" event={"ID":"25d85829-1725-4150-9c59-09c80da4dcab","Type":"ContainerStarted","Data":"10f086ac239ceb82ef8e25f0d35a6af0639508cb04e069b03ec6fcbfddbd5dd0"} Oct 30 16:32:28.900771 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:28.900706 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-5ddb48fb-kmgjr" podStartSLOduration=1.304454121 podStartE2EDuration="1.900686527s" totalImagesPullingTime="596.232406ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:32:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:32:28.897683498 +0000 UTC m=+704.425018717" watchObservedRunningTime="2026-10-30 16:32:28.900686527 +0000 UTC m=+704.428021762" Oct 30 16:32:31.756599 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:31.755993 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5cz66"] Oct 30 16:32:31.765181 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:31.764530 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5cz66" Oct 30 16:32:31.843604 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:31.840438 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5cz66"] Oct 30 16:32:31.856852 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:31.856817 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-nztr5\"" Oct 30 16:32:31.877100 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:31.877059 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f59239c4-202e-45d5-9942-26fb0410d813-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-5cz66\" (UID: \"f59239c4-202e-45d5-9942-26fb0410d813\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5cz66" Oct 30 16:32:31.877260 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:31.877125 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f59239c4-202e-45d5-9942-26fb0410d813-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-5cz66\" (UID: \"f59239c4-202e-45d5-9942-26fb0410d813\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5cz66" Oct 30 16:32:31.877260 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:31.877194 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f59239c4-202e-45d5-9942-26fb0410d813-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-5cz66\" (UID: \"f59239c4-202e-45d5-9942-26fb0410d813\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5cz66" Oct 30 16:32:31.877391 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:31.877294 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f59239c4-202e-45d5-9942-26fb0410d813-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-5cz66\" (UID: \"f59239c4-202e-45d5-9942-26fb0410d813\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5cz66" Oct 30 16:32:31.877445 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:31.877403 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f59239c4-202e-45d5-9942-26fb0410d813-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-5cz66\" (UID: \"f59239c4-202e-45d5-9942-26fb0410d813\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5cz66" Oct 30 16:32:31.877492 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:31.877475 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f59239c4-202e-45d5-9942-26fb0410d813-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-5cz66\" (UID: \"f59239c4-202e-45d5-9942-26fb0410d813\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5cz66" Oct 30 16:32:31.877542 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:31.877508 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f59239c4-202e-45d5-9942-26fb0410d813-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-5cz66\" (UID: \"f59239c4-202e-45d5-9942-26fb0410d813\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5cz66" Oct 30 16:32:31.877663 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:31.877560 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f59239c4-202e-45d5-9942-26fb0410d813-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-5cz66\" (UID: \"f59239c4-202e-45d5-9942-26fb0410d813\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5cz66" Oct 30 16:32:31.877663 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:31.877616 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57xfz\" (UniqueName: \"kubernetes.io/projected/f59239c4-202e-45d5-9942-26fb0410d813-kube-api-access-57xfz\") pod \"maas-default-gateway-openshift-default-76f4876667-5cz66\" (UID: \"f59239c4-202e-45d5-9942-26fb0410d813\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5cz66" Oct 30 16:32:31.978730 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:31.978696 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f59239c4-202e-45d5-9942-26fb0410d813-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-5cz66\" (UID: \"f59239c4-202e-45d5-9942-26fb0410d813\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5cz66" Oct 30 16:32:31.978898 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:31.978740 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f59239c4-202e-45d5-9942-26fb0410d813-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-5cz66\" (UID: \"f59239c4-202e-45d5-9942-26fb0410d813\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5cz66" Oct 30 16:32:31.978898 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:31.978759 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f59239c4-202e-45d5-9942-26fb0410d813-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-5cz66\" (UID: \"f59239c4-202e-45d5-9942-26fb0410d813\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5cz66" Oct 30 16:32:31.978898 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:31.978872 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f59239c4-202e-45d5-9942-26fb0410d813-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-5cz66\" (UID: \"f59239c4-202e-45d5-9942-26fb0410d813\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5cz66" Oct 30 16:32:31.979095 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:31.978910 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57xfz\" (UniqueName: \"kubernetes.io/projected/f59239c4-202e-45d5-9942-26fb0410d813-kube-api-access-57xfz\") pod \"maas-default-gateway-openshift-default-76f4876667-5cz66\" (UID: \"f59239c4-202e-45d5-9942-26fb0410d813\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5cz66" Oct 30 16:32:31.979095 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:31.978975 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f59239c4-202e-45d5-9942-26fb0410d813-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-5cz66\" (UID: \"f59239c4-202e-45d5-9942-26fb0410d813\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5cz66" Oct 30 16:32:31.979095 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:31.979021 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f59239c4-202e-45d5-9942-26fb0410d813-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-5cz66\" (UID: \"f59239c4-202e-45d5-9942-26fb0410d813\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5cz66" Oct 30 16:32:31.979095 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:31.979043 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f59239c4-202e-45d5-9942-26fb0410d813-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-5cz66\" (UID: \"f59239c4-202e-45d5-9942-26fb0410d813\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5cz66" Oct 30 16:32:31.979095 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:31.979071 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f59239c4-202e-45d5-9942-26fb0410d813-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-5cz66\" (UID: \"f59239c4-202e-45d5-9942-26fb0410d813\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5cz66" Oct 30 16:32:31.979411 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:31.979179 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f59239c4-202e-45d5-9942-26fb0410d813-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-5cz66\" (UID: \"f59239c4-202e-45d5-9942-26fb0410d813\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5cz66" Oct 30 16:32:31.979411 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:31.979222 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f59239c4-202e-45d5-9942-26fb0410d813-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-5cz66\" (UID: \"f59239c4-202e-45d5-9942-26fb0410d813\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5cz66" Oct 30 16:32:31.979517 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:31.979429 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f59239c4-202e-45d5-9942-26fb0410d813-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-5cz66\" (UID: \"f59239c4-202e-45d5-9942-26fb0410d813\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5cz66" Oct 30 16:32:31.979517 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:31.979500 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f59239c4-202e-45d5-9942-26fb0410d813-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-5cz66\" (UID: \"f59239c4-202e-45d5-9942-26fb0410d813\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5cz66" Oct 30 16:32:31.979801 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:31.979785 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f59239c4-202e-45d5-9942-26fb0410d813-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-5cz66\" (UID: \"f59239c4-202e-45d5-9942-26fb0410d813\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5cz66" Oct 30 16:32:31.981167 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:31.981145 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f59239c4-202e-45d5-9942-26fb0410d813-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-5cz66\" (UID: \"f59239c4-202e-45d5-9942-26fb0410d813\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5cz66" Oct 30 16:32:31.981964 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:31.981942 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f59239c4-202e-45d5-9942-26fb0410d813-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-5cz66\" (UID: \"f59239c4-202e-45d5-9942-26fb0410d813\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5cz66" Oct 30 16:32:32.001939 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:32.001917 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f59239c4-202e-45d5-9942-26fb0410d813-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-5cz66\" (UID: \"f59239c4-202e-45d5-9942-26fb0410d813\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5cz66" Oct 30 16:32:32.002160 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:32.002138 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57xfz\" (UniqueName: \"kubernetes.io/projected/f59239c4-202e-45d5-9942-26fb0410d813-kube-api-access-57xfz\") pod \"maas-default-gateway-openshift-default-76f4876667-5cz66\" (UID: \"f59239c4-202e-45d5-9942-26fb0410d813\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5cz66" Oct 30 16:32:32.118812 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:32.118734 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5cz66" Oct 30 16:32:32.447656 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:32.447624 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5cz66"] Oct 30 16:32:32.451203 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:32:32.451170 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf59239c4_202e_45d5_9942_26fb0410d813.slice/crio-34fc023fffb89f50d0121b3f0857f2a193e37eb0b350c4158ec4466ba2df3bdf WatchSource:0}: Error finding container 34fc023fffb89f50d0121b3f0857f2a193e37eb0b350c4158ec4466ba2df3bdf: Status 404 returned error can't find the container with id 34fc023fffb89f50d0121b3f0857f2a193e37eb0b350c4158ec4466ba2df3bdf Oct 30 16:32:32.872221 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:32.872141 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5cz66" event={"ID":"f59239c4-202e-45d5-9942-26fb0410d813","Type":"ContainerStarted","Data":"34fc023fffb89f50d0121b3f0857f2a193e37eb0b350c4158ec4466ba2df3bdf"} Oct 30 16:32:35.028710 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:35.028678 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:32:35.029050 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:35.028746 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:32:35.029050 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:35.028776 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:32:35.891197 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:35.891153 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5cz66" event={"ID":"f59239c4-202e-45d5-9942-26fb0410d813","Type":"ContainerStarted","Data":"5153a71e73a3e6ac24744eac1a5a2deee8a609fcd75de487da50e3064a05cece"} Oct 30 16:32:35.965277 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:35.965228 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5cz66" podStartSLOduration=2.391148019 podStartE2EDuration="4.965215879s" totalImagesPullingTime="2.57406786s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:32:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:32:35.953146872 +0000 UTC m=+711.480482091" watchObservedRunningTime="2026-10-30 16:32:35.965215879 +0000 UTC m=+711.492551099" Oct 30 16:32:36.118942 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:36.118908 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5cz66" Oct 30 16:32:36.120597 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:36.120563 5052 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-5cz66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Oct 30 16:32:36.120727 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:36.120624 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5cz66" podUID="f59239c4-202e-45d5-9942-26fb0410d813" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Oct 30 16:32:37.119879 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:37.119846 5052 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-5cz66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Oct 30 16:32:37.120263 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:37.119899 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5cz66" podUID="f59239c4-202e-45d5-9942-26fb0410d813" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Oct 30 16:32:38.119282 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:38.119247 5052 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-5cz66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Oct 30 16:32:38.119460 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:38.119296 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5cz66" podUID="f59239c4-202e-45d5-9942-26fb0410d813" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Oct 30 16:32:39.119183 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:39.119145 5052 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-5cz66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Oct 30 16:32:39.119662 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:39.119197 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5cz66" podUID="f59239c4-202e-45d5-9942-26fb0410d813" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Oct 30 16:32:40.119269 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:40.119235 5052 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-5cz66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Oct 30 16:32:40.119685 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:40.119297 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5cz66" podUID="f59239c4-202e-45d5-9942-26fb0410d813" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Oct 30 16:32:40.837969 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:40.837828 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-4ptgl"] Oct 30 16:32:40.842293 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:40.842266 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5f78687d5b-4ptgl" Oct 30 16:32:40.864114 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:40.864085 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-4ptgl"] Oct 30 16:32:40.880920 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:40.880892 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Oct 30 16:32:40.881088 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:40.881039 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-xwsp9\"" Oct 30 16:32:40.986291 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:40.986253 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9k7j\" (UniqueName: \"kubernetes.io/projected/973d1ea4-3981-4350-b719-edb7911c23ab-kube-api-access-k9k7j\") pod \"payload-processing-5f78687d5b-4ptgl\" (UID: \"973d1ea4-3981-4350-b719-edb7911c23ab\") " pod="openshift-ingress/payload-processing-5f78687d5b-4ptgl" Oct 30 16:32:40.986500 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:40.986438 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/973d1ea4-3981-4350-b719-edb7911c23ab-plugins-config-volume\") pod \"payload-processing-5f78687d5b-4ptgl\" (UID: \"973d1ea4-3981-4350-b719-edb7911c23ab\") " pod="openshift-ingress/payload-processing-5f78687d5b-4ptgl" Oct 30 16:32:41.087086 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:41.087052 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/973d1ea4-3981-4350-b719-edb7911c23ab-plugins-config-volume\") pod \"payload-processing-5f78687d5b-4ptgl\" (UID: \"973d1ea4-3981-4350-b719-edb7911c23ab\") " pod="openshift-ingress/payload-processing-5f78687d5b-4ptgl" Oct 30 16:32:41.087239 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:41.087098 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9k7j\" (UniqueName: \"kubernetes.io/projected/973d1ea4-3981-4350-b719-edb7911c23ab-kube-api-access-k9k7j\") pod \"payload-processing-5f78687d5b-4ptgl\" (UID: \"973d1ea4-3981-4350-b719-edb7911c23ab\") " pod="openshift-ingress/payload-processing-5f78687d5b-4ptgl" Oct 30 16:32:41.087872 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:41.087850 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/973d1ea4-3981-4350-b719-edb7911c23ab-plugins-config-volume\") pod \"payload-processing-5f78687d5b-4ptgl\" (UID: \"973d1ea4-3981-4350-b719-edb7911c23ab\") " pod="openshift-ingress/payload-processing-5f78687d5b-4ptgl" Oct 30 16:32:41.110720 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:41.110651 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9k7j\" (UniqueName: \"kubernetes.io/projected/973d1ea4-3981-4350-b719-edb7911c23ab-kube-api-access-k9k7j\") pod \"payload-processing-5f78687d5b-4ptgl\" (UID: \"973d1ea4-3981-4350-b719-edb7911c23ab\") " pod="openshift-ingress/payload-processing-5f78687d5b-4ptgl" Oct 30 16:32:41.119565 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:41.119543 5052 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-5cz66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Oct 30 16:32:41.119900 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:41.119621 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5cz66" podUID="f59239c4-202e-45d5-9942-26fb0410d813" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Oct 30 16:32:41.168971 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:41.168943 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5f78687d5b-4ptgl" Oct 30 16:32:41.473478 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:41.473452 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-4ptgl"] Oct 30 16:32:41.476219 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:32:41.476184 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod973d1ea4_3981_4350_b719_edb7911c23ab.slice/crio-544d242ce967b096ea67f768b5996c1bd9294a48f460234beba03f6b1b5be39f WatchSource:0}: Error finding container 544d242ce967b096ea67f768b5996c1bd9294a48f460234beba03f6b1b5be39f: Status 404 returned error can't find the container with id 544d242ce967b096ea67f768b5996c1bd9294a48f460234beba03f6b1b5be39f Oct 30 16:32:41.922117 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:41.922081 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5f78687d5b-4ptgl" event={"ID":"973d1ea4-3981-4350-b719-edb7911c23ab","Type":"ContainerStarted","Data":"544d242ce967b096ea67f768b5996c1bd9294a48f460234beba03f6b1b5be39f"} Oct 30 16:32:42.119754 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:42.119703 5052 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-5cz66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Oct 30 16:32:42.120204 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:42.119780 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5cz66" podUID="f59239c4-202e-45d5-9942-26fb0410d813" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Oct 30 16:32:43.933850 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:43.933813 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5f78687d5b-4ptgl" event={"ID":"973d1ea4-3981-4350-b719-edb7911c23ab","Type":"ContainerStarted","Data":"4ed8b62d3afb7986d5b02b8718b346e99d3c53998f94cc66a26915abcf9eccff"} Oct 30 16:32:43.934345 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:43.933951 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5f78687d5b-4ptgl" Oct 30 16:32:43.974776 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:43.974728 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5f78687d5b-4ptgl" podStartSLOduration=2.000650029 podStartE2EDuration="3.97471635s" totalImagesPullingTime="1.974066321s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:32:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:32:43.973402749 +0000 UTC m=+719.500737967" watchObservedRunningTime="2026-10-30 16:32:43.97471635 +0000 UTC m=+719.502051570" Oct 30 16:32:44.120573 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:44.120540 5052 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-5cz66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:32:44.120741 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:44.120621 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5cz66" podUID="f59239c4-202e-45d5-9942-26fb0410d813" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:32:45.161781 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:45.161737 5052 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-5cz66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:32:45.162311 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:45.161804 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5cz66" podUID="f59239c4-202e-45d5-9942-26fb0410d813" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:32:45.728877 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:45.728846 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5cz66" Oct 30 16:32:45.729267 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:45.729242 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5cz66" Oct 30 16:32:45.729940 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:45.729919 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5cz66" Oct 30 16:32:46.714070 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:46.714024 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-t8b5p"] Oct 30 16:32:46.717720 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:46.717698 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-t8b5p" Oct 30 16:32:46.739055 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:46.739026 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcc7c9e1-99ea-4b97-9e95-2e23a00ac096-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-t8b5p\" (UID: \"fcc7c9e1-99ea-4b97-9e95-2e23a00ac096\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-t8b5p" Oct 30 16:32:46.739243 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:46.739084 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcc7c9e1-99ea-4b97-9e95-2e23a00ac096-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-t8b5p\" (UID: \"fcc7c9e1-99ea-4b97-9e95-2e23a00ac096\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-t8b5p" Oct 30 16:32:46.739243 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:46.739128 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcc7c9e1-99ea-4b97-9e95-2e23a00ac096-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-t8b5p\" (UID: \"fcc7c9e1-99ea-4b97-9e95-2e23a00ac096\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-t8b5p" Oct 30 16:32:46.739243 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:46.739170 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcc7c9e1-99ea-4b97-9e95-2e23a00ac096-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-t8b5p\" (UID: \"fcc7c9e1-99ea-4b97-9e95-2e23a00ac096\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-t8b5p" Oct 30 16:32:46.739243 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:46.739220 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvwlt\" (UniqueName: \"kubernetes.io/projected/fcc7c9e1-99ea-4b97-9e95-2e23a00ac096-kube-api-access-vvwlt\") pod \"maas-default-gateway-openshift-default-76f4876667-t8b5p\" (UID: \"fcc7c9e1-99ea-4b97-9e95-2e23a00ac096\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-t8b5p" Oct 30 16:32:46.739478 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:46.739254 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcc7c9e1-99ea-4b97-9e95-2e23a00ac096-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-t8b5p\" (UID: \"fcc7c9e1-99ea-4b97-9e95-2e23a00ac096\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-t8b5p" Oct 30 16:32:46.739478 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:46.739337 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcc7c9e1-99ea-4b97-9e95-2e23a00ac096-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-t8b5p\" (UID: \"fcc7c9e1-99ea-4b97-9e95-2e23a00ac096\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-t8b5p" Oct 30 16:32:46.739628 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:46.739497 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcc7c9e1-99ea-4b97-9e95-2e23a00ac096-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-t8b5p\" (UID: \"fcc7c9e1-99ea-4b97-9e95-2e23a00ac096\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-t8b5p" Oct 30 16:32:46.739628 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:46.739530 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcc7c9e1-99ea-4b97-9e95-2e23a00ac096-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-t8b5p\" (UID: \"fcc7c9e1-99ea-4b97-9e95-2e23a00ac096\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-t8b5p" Oct 30 16:32:46.740349 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:46.740325 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-t8b5p"] Oct 30 16:32:46.840451 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:46.840405 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcc7c9e1-99ea-4b97-9e95-2e23a00ac096-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-t8b5p\" (UID: \"fcc7c9e1-99ea-4b97-9e95-2e23a00ac096\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-t8b5p" Oct 30 16:32:46.840451 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:46.840457 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcc7c9e1-99ea-4b97-9e95-2e23a00ac096-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-t8b5p\" (UID: \"fcc7c9e1-99ea-4b97-9e95-2e23a00ac096\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-t8b5p" Oct 30 16:32:46.840735 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:46.840482 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcc7c9e1-99ea-4b97-9e95-2e23a00ac096-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-t8b5p\" (UID: \"fcc7c9e1-99ea-4b97-9e95-2e23a00ac096\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-t8b5p" Oct 30 16:32:46.840735 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:46.840508 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvwlt\" (UniqueName: \"kubernetes.io/projected/fcc7c9e1-99ea-4b97-9e95-2e23a00ac096-kube-api-access-vvwlt\") pod \"maas-default-gateway-openshift-default-76f4876667-t8b5p\" (UID: \"fcc7c9e1-99ea-4b97-9e95-2e23a00ac096\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-t8b5p" Oct 30 16:32:46.840735 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:46.840562 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcc7c9e1-99ea-4b97-9e95-2e23a00ac096-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-t8b5p\" (UID: \"fcc7c9e1-99ea-4b97-9e95-2e23a00ac096\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-t8b5p" Oct 30 16:32:46.840735 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:46.840638 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcc7c9e1-99ea-4b97-9e95-2e23a00ac096-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-t8b5p\" (UID: \"fcc7c9e1-99ea-4b97-9e95-2e23a00ac096\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-t8b5p" Oct 30 16:32:46.840735 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:46.840695 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcc7c9e1-99ea-4b97-9e95-2e23a00ac096-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-t8b5p\" (UID: \"fcc7c9e1-99ea-4b97-9e95-2e23a00ac096\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-t8b5p" Oct 30 16:32:46.840735 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:46.840727 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcc7c9e1-99ea-4b97-9e95-2e23a00ac096-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-t8b5p\" (UID: \"fcc7c9e1-99ea-4b97-9e95-2e23a00ac096\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-t8b5p" Oct 30 16:32:46.841043 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:46.840783 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcc7c9e1-99ea-4b97-9e95-2e23a00ac096-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-t8b5p\" (UID: \"fcc7c9e1-99ea-4b97-9e95-2e23a00ac096\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-t8b5p" Oct 30 16:32:46.841043 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:46.840893 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcc7c9e1-99ea-4b97-9e95-2e23a00ac096-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-t8b5p\" (UID: \"fcc7c9e1-99ea-4b97-9e95-2e23a00ac096\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-t8b5p" Oct 30 16:32:46.841043 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:46.840911 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcc7c9e1-99ea-4b97-9e95-2e23a00ac096-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-t8b5p\" (UID: \"fcc7c9e1-99ea-4b97-9e95-2e23a00ac096\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-t8b5p" Oct 30 16:32:46.841043 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:46.840965 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcc7c9e1-99ea-4b97-9e95-2e23a00ac096-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-t8b5p\" (UID: \"fcc7c9e1-99ea-4b97-9e95-2e23a00ac096\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-t8b5p" Oct 30 16:32:46.841043 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:46.841008 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcc7c9e1-99ea-4b97-9e95-2e23a00ac096-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-t8b5p\" (UID: \"fcc7c9e1-99ea-4b97-9e95-2e23a00ac096\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-t8b5p" Oct 30 16:32:46.841457 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:46.841436 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcc7c9e1-99ea-4b97-9e95-2e23a00ac096-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-t8b5p\" (UID: \"fcc7c9e1-99ea-4b97-9e95-2e23a00ac096\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-t8b5p" Oct 30 16:32:46.842856 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:46.842832 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcc7c9e1-99ea-4b97-9e95-2e23a00ac096-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-t8b5p\" (UID: \"fcc7c9e1-99ea-4b97-9e95-2e23a00ac096\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-t8b5p" Oct 30 16:32:46.843090 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:46.843073 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcc7c9e1-99ea-4b97-9e95-2e23a00ac096-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-t8b5p\" (UID: \"fcc7c9e1-99ea-4b97-9e95-2e23a00ac096\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-t8b5p" Oct 30 16:32:46.863668 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:46.863644 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcc7c9e1-99ea-4b97-9e95-2e23a00ac096-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-t8b5p\" (UID: \"fcc7c9e1-99ea-4b97-9e95-2e23a00ac096\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-t8b5p" Oct 30 16:32:46.863816 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:46.863800 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvwlt\" (UniqueName: \"kubernetes.io/projected/fcc7c9e1-99ea-4b97-9e95-2e23a00ac096-kube-api-access-vvwlt\") pod \"maas-default-gateway-openshift-default-76f4876667-t8b5p\" (UID: \"fcc7c9e1-99ea-4b97-9e95-2e23a00ac096\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-t8b5p" Oct 30 16:32:47.037846 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:47.037769 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-t8b5p" Oct 30 16:32:47.354715 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:47.354632 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-t8b5p"] Oct 30 16:32:47.358387 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:32:47.358355 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcc7c9e1_99ea_4b97_9e95_2e23a00ac096.slice/crio-655d7856bc618f3258c537f8a41c33477fffaf1929539f36b24b3b3721562c54 WatchSource:0}: Error finding container 655d7856bc618f3258c537f8a41c33477fffaf1929539f36b24b3b3721562c54: Status 404 returned error can't find the container with id 655d7856bc618f3258c537f8a41c33477fffaf1929539f36b24b3b3721562c54 Oct 30 16:32:47.360492 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:47.360461 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:32:47.360600 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:47.360527 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:32:47.360600 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:47.360559 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:32:47.958825 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:47.958791 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-t8b5p" event={"ID":"fcc7c9e1-99ea-4b97-9e95-2e23a00ac096","Type":"ContainerStarted","Data":"9e7b48641705d27c2cb8662a4c19de28ca2c5163584c0c95c4f608e52147f282"} Oct 30 16:32:47.959242 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:47.958842 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-t8b5p" event={"ID":"fcc7c9e1-99ea-4b97-9e95-2e23a00ac096","Type":"ContainerStarted","Data":"655d7856bc618f3258c537f8a41c33477fffaf1929539f36b24b3b3721562c54"} Oct 30 16:32:48.029018 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:48.028761 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-t8b5p" podStartSLOduration=2.028744853 podStartE2EDuration="2.028744853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:32:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:32:48.015092016 +0000 UTC m=+723.542427230" watchObservedRunningTime="2026-10-30 16:32:48.028744853 +0000 UTC m=+723.556080073" Oct 30 16:32:48.039086 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:48.039051 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-t8b5p" Oct 30 16:32:48.047544 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:48.047508 5052 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-t8b5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Oct 30 16:32:48.047882 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:48.047850 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-t8b5p" podUID="fcc7c9e1-99ea-4b97-9e95-2e23a00ac096" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Oct 30 16:32:48.864118 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:32:48.864078 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 16:32:49.049950 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:49.049914 5052 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-t8b5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Oct 30 16:32:49.050550 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:49.050510 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-t8b5p" podUID="fcc7c9e1-99ea-4b97-9e95-2e23a00ac096" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Oct 30 16:32:50.046606 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:50.043960 5052 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-t8b5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Oct 30 16:32:50.046606 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:50.044020 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-t8b5p" podUID="fcc7c9e1-99ea-4b97-9e95-2e23a00ac096" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Oct 30 16:32:51.043604 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:51.039941 5052 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-t8b5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Oct 30 16:32:51.043604 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:51.040002 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-t8b5p" podUID="fcc7c9e1-99ea-4b97-9e95-2e23a00ac096" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Oct 30 16:32:52.046011 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:52.045976 5052 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-t8b5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Oct 30 16:32:52.046641 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:52.046574 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-t8b5p" podUID="fcc7c9e1-99ea-4b97-9e95-2e23a00ac096" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Oct 30 16:32:53.046603 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:53.043052 5052 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-t8b5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Oct 30 16:32:53.046603 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:53.043124 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-t8b5p" podUID="fcc7c9e1-99ea-4b97-9e95-2e23a00ac096" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Oct 30 16:32:54.039107 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:54.039070 5052 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-t8b5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Oct 30 16:32:54.039290 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:54.039137 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-t8b5p" podUID="fcc7c9e1-99ea-4b97-9e95-2e23a00ac096" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Oct 30 16:32:54.943433 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:54.943405 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5f78687d5b-4ptgl" Oct 30 16:32:55.613761 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:55.613732 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-t8b5p" Oct 30 16:32:55.614105 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:55.614076 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-t8b5p" Oct 30 16:32:55.614741 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:32:55.614725 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-t8b5p" Oct 30 16:33:18.840321 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:33:18.840280 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 16:33:48.839505 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:33:48.839467 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 16:34:03.094618 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:03.094564 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-46brl"] Oct 30 16:34:03.099871 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:03.099838 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-46brl" Oct 30 16:34:03.123065 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:03.123039 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-46brl"] Oct 30 16:34:03.131472 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:03.131429 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-0b2348-openshift-default-dockercfg-86t2v\"" Oct 30 16:34:03.274680 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:03.274631 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b8235a9-5724-4354-ad7b-eaef26b3966f-workload-socket\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-46brl\" (UID: \"4b8235a9-5724-4354-ad7b-eaef26b3966f\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-46brl" Oct 30 16:34:03.274866 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:03.274774 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b8235a9-5724-4354-ad7b-eaef26b3966f-workload-certs\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-46brl\" (UID: \"4b8235a9-5724-4354-ad7b-eaef26b3966f\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-46brl" Oct 30 16:34:03.274943 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:03.274917 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b8235a9-5724-4354-ad7b-eaef26b3966f-istiod-ca-cert\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-46brl\" (UID: \"4b8235a9-5724-4354-ad7b-eaef26b3966f\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-46brl" Oct 30 16:34:03.275303 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:03.275267 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b8235a9-5724-4354-ad7b-eaef26b3966f-credential-socket\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-46brl\" (UID: \"4b8235a9-5724-4354-ad7b-eaef26b3966f\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-46brl" Oct 30 16:34:03.275372 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:03.275338 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b8235a9-5724-4354-ad7b-eaef26b3966f-istio-envoy\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-46brl\" (UID: \"4b8235a9-5724-4354-ad7b-eaef26b3966f\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-46brl" Oct 30 16:34:03.275428 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:03.275369 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b8235a9-5724-4354-ad7b-eaef26b3966f-istio-token\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-46brl\" (UID: \"4b8235a9-5724-4354-ad7b-eaef26b3966f\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-46brl" Oct 30 16:34:03.275428 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:03.275404 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w24fw\" (UniqueName: \"kubernetes.io/projected/4b8235a9-5724-4354-ad7b-eaef26b3966f-kube-api-access-w24fw\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-46brl\" (UID: \"4b8235a9-5724-4354-ad7b-eaef26b3966f\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-46brl" Oct 30 16:34:03.275531 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:03.275477 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b8235a9-5724-4354-ad7b-eaef26b3966f-istio-data\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-46brl\" (UID: \"4b8235a9-5724-4354-ad7b-eaef26b3966f\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-46brl" Oct 30 16:34:03.275531 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:03.275517 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b8235a9-5724-4354-ad7b-eaef26b3966f-istio-podinfo\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-46brl\" (UID: \"4b8235a9-5724-4354-ad7b-eaef26b3966f\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-46brl" Oct 30 16:34:03.358784 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:03.358702 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-hff9w"] Oct 30 16:34:03.362756 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:03.362731 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-hff9w" Oct 30 16:34:03.378190 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:03.378159 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b8235a9-5724-4354-ad7b-eaef26b3966f-workload-socket\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-46brl\" (UID: \"4b8235a9-5724-4354-ad7b-eaef26b3966f\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-46brl" Oct 30 16:34:03.378397 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:03.378361 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b8235a9-5724-4354-ad7b-eaef26b3966f-workload-certs\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-46brl\" (UID: \"4b8235a9-5724-4354-ad7b-eaef26b3966f\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-46brl" Oct 30 16:34:03.378509 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:03.378457 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b8235a9-5724-4354-ad7b-eaef26b3966f-istiod-ca-cert\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-46brl\" (UID: \"4b8235a9-5724-4354-ad7b-eaef26b3966f\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-46brl" Oct 30 16:34:03.378509 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:03.378501 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b8235a9-5724-4354-ad7b-eaef26b3966f-credential-socket\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-46brl\" (UID: \"4b8235a9-5724-4354-ad7b-eaef26b3966f\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-46brl" Oct 30 16:34:03.378640 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:03.378557 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b8235a9-5724-4354-ad7b-eaef26b3966f-istio-envoy\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-46brl\" (UID: \"4b8235a9-5724-4354-ad7b-eaef26b3966f\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-46brl" Oct 30 16:34:03.378640 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:03.378614 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b8235a9-5724-4354-ad7b-eaef26b3966f-istio-token\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-46brl\" (UID: \"4b8235a9-5724-4354-ad7b-eaef26b3966f\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-46brl" Oct 30 16:34:03.380997 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:03.378658 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w24fw\" (UniqueName: \"kubernetes.io/projected/4b8235a9-5724-4354-ad7b-eaef26b3966f-kube-api-access-w24fw\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-46brl\" (UID: \"4b8235a9-5724-4354-ad7b-eaef26b3966f\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-46brl" Oct 30 16:34:03.380997 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:03.378696 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b8235a9-5724-4354-ad7b-eaef26b3966f-istio-data\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-46brl\" (UID: \"4b8235a9-5724-4354-ad7b-eaef26b3966f\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-46brl" Oct 30 16:34:03.380997 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:03.378736 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b8235a9-5724-4354-ad7b-eaef26b3966f-istio-podinfo\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-46brl\" (UID: \"4b8235a9-5724-4354-ad7b-eaef26b3966f\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-46brl" Oct 30 16:34:03.380997 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:03.379208 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b8235a9-5724-4354-ad7b-eaef26b3966f-credential-socket\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-46brl\" (UID: \"4b8235a9-5724-4354-ad7b-eaef26b3966f\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-46brl" Oct 30 16:34:03.380997 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:03.379449 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b8235a9-5724-4354-ad7b-eaef26b3966f-workload-socket\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-46brl\" (UID: \"4b8235a9-5724-4354-ad7b-eaef26b3966f\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-46brl" Oct 30 16:34:03.380997 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:03.379702 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b8235a9-5724-4354-ad7b-eaef26b3966f-workload-certs\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-46brl\" (UID: \"4b8235a9-5724-4354-ad7b-eaef26b3966f\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-46brl" Oct 30 16:34:03.380997 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:03.380277 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b8235a9-5724-4354-ad7b-eaef26b3966f-istiod-ca-cert\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-46brl\" (UID: \"4b8235a9-5724-4354-ad7b-eaef26b3966f\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-46brl" Oct 30 16:34:03.390904 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:03.381507 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b8235a9-5724-4354-ad7b-eaef26b3966f-istio-data\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-46brl\" (UID: \"4b8235a9-5724-4354-ad7b-eaef26b3966f\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-46brl" Oct 30 16:34:03.390904 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:03.381896 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b8235a9-5724-4354-ad7b-eaef26b3966f-istio-podinfo\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-46brl\" (UID: \"4b8235a9-5724-4354-ad7b-eaef26b3966f\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-46brl" Oct 30 16:34:03.390904 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:03.389239 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-hff9w"] Oct 30 16:34:03.394890 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:03.394864 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b8235a9-5724-4354-ad7b-eaef26b3966f-istio-envoy\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-46brl\" (UID: \"4b8235a9-5724-4354-ad7b-eaef26b3966f\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-46brl" Oct 30 16:34:03.408622 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:03.408595 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b8235a9-5724-4354-ad7b-eaef26b3966f-istio-token\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-46brl\" (UID: \"4b8235a9-5724-4354-ad7b-eaef26b3966f\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-46brl" Oct 30 16:34:03.416919 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:03.416895 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w24fw\" (UniqueName: \"kubernetes.io/projected/4b8235a9-5724-4354-ad7b-eaef26b3966f-kube-api-access-w24fw\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-46brl\" (UID: \"4b8235a9-5724-4354-ad7b-eaef26b3966f\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-46brl" Oct 30 16:34:03.418313 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:03.418274 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-46brl" Oct 30 16:34:03.444066 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:03.444029 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-967fac-openshift-default-dockercfg-lfjgc\"" Oct 30 16:34:03.480185 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:03.480155 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/559db955-2645-4b32-9070-e87d7bcb485e-istio-token\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-hff9w\" (UID: \"559db955-2645-4b32-9070-e87d7bcb485e\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-hff9w" Oct 30 16:34:03.480312 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:03.480206 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/559db955-2645-4b32-9070-e87d7bcb485e-istio-podinfo\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-hff9w\" (UID: \"559db955-2645-4b32-9070-e87d7bcb485e\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-hff9w" Oct 30 16:34:03.480312 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:03.480289 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/559db955-2645-4b32-9070-e87d7bcb485e-credential-socket\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-hff9w\" (UID: \"559db955-2645-4b32-9070-e87d7bcb485e\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-hff9w" Oct 30 16:34:03.480447 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:03.480326 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/559db955-2645-4b32-9070-e87d7bcb485e-istio-envoy\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-hff9w\" (UID: \"559db955-2645-4b32-9070-e87d7bcb485e\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-hff9w" Oct 30 16:34:03.480447 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:03.480386 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/559db955-2645-4b32-9070-e87d7bcb485e-workload-certs\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-hff9w\" (UID: \"559db955-2645-4b32-9070-e87d7bcb485e\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-hff9w" Oct 30 16:34:03.480447 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:03.480409 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/559db955-2645-4b32-9070-e87d7bcb485e-istiod-ca-cert\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-hff9w\" (UID: \"559db955-2645-4b32-9070-e87d7bcb485e\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-hff9w" Oct 30 16:34:03.480630 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:03.480478 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/559db955-2645-4b32-9070-e87d7bcb485e-workload-socket\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-hff9w\" (UID: \"559db955-2645-4b32-9070-e87d7bcb485e\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-hff9w" Oct 30 16:34:03.480630 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:03.480503 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/559db955-2645-4b32-9070-e87d7bcb485e-istio-data\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-hff9w\" (UID: \"559db955-2645-4b32-9070-e87d7bcb485e\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-hff9w" Oct 30 16:34:03.480630 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:03.480555 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzs6s\" (UniqueName: \"kubernetes.io/projected/559db955-2645-4b32-9070-e87d7bcb485e-kube-api-access-dzs6s\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-hff9w\" (UID: \"559db955-2645-4b32-9070-e87d7bcb485e\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-hff9w" Oct 30 16:34:03.581635 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:03.581566 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/559db955-2645-4b32-9070-e87d7bcb485e-workload-socket\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-hff9w\" (UID: \"559db955-2645-4b32-9070-e87d7bcb485e\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-hff9w" Oct 30 16:34:03.581829 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:03.581651 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/559db955-2645-4b32-9070-e87d7bcb485e-istio-data\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-hff9w\" (UID: \"559db955-2645-4b32-9070-e87d7bcb485e\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-hff9w" Oct 30 16:34:03.581829 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:03.581704 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzs6s\" (UniqueName: \"kubernetes.io/projected/559db955-2645-4b32-9070-e87d7bcb485e-kube-api-access-dzs6s\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-hff9w\" (UID: \"559db955-2645-4b32-9070-e87d7bcb485e\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-hff9w" Oct 30 16:34:03.582271 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:03.582249 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/559db955-2645-4b32-9070-e87d7bcb485e-istio-token\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-hff9w\" (UID: \"559db955-2645-4b32-9070-e87d7bcb485e\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-hff9w" Oct 30 16:34:03.582334 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:03.582159 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/559db955-2645-4b32-9070-e87d7bcb485e-workload-socket\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-hff9w\" (UID: \"559db955-2645-4b32-9070-e87d7bcb485e\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-hff9w" Oct 30 16:34:03.582334 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:03.582288 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/559db955-2645-4b32-9070-e87d7bcb485e-istio-podinfo\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-hff9w\" (UID: \"559db955-2645-4b32-9070-e87d7bcb485e\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-hff9w" Oct 30 16:34:03.582334 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:03.582327 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/559db955-2645-4b32-9070-e87d7bcb485e-istio-data\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-hff9w\" (UID: \"559db955-2645-4b32-9070-e87d7bcb485e\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-hff9w" Oct 30 16:34:03.582481 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:03.582402 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/559db955-2645-4b32-9070-e87d7bcb485e-credential-socket\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-hff9w\" (UID: \"559db955-2645-4b32-9070-e87d7bcb485e\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-hff9w" Oct 30 16:34:03.582481 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:03.582444 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/559db955-2645-4b32-9070-e87d7bcb485e-istio-envoy\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-hff9w\" (UID: \"559db955-2645-4b32-9070-e87d7bcb485e\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-hff9w" Oct 30 16:34:03.582599 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:03.582499 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/559db955-2645-4b32-9070-e87d7bcb485e-workload-certs\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-hff9w\" (UID: \"559db955-2645-4b32-9070-e87d7bcb485e\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-hff9w" Oct 30 16:34:03.582599 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:03.582526 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/559db955-2645-4b32-9070-e87d7bcb485e-istiod-ca-cert\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-hff9w\" (UID: \"559db955-2645-4b32-9070-e87d7bcb485e\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-hff9w" Oct 30 16:34:03.583148 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:03.583104 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/559db955-2645-4b32-9070-e87d7bcb485e-workload-certs\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-hff9w\" (UID: \"559db955-2645-4b32-9070-e87d7bcb485e\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-hff9w" Oct 30 16:34:03.583284 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:03.583240 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/559db955-2645-4b32-9070-e87d7bcb485e-credential-socket\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-hff9w\" (UID: \"559db955-2645-4b32-9070-e87d7bcb485e\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-hff9w" Oct 30 16:34:03.583394 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:03.583364 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/559db955-2645-4b32-9070-e87d7bcb485e-istiod-ca-cert\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-hff9w\" (UID: \"559db955-2645-4b32-9070-e87d7bcb485e\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-hff9w" Oct 30 16:34:03.584971 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:03.584947 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/559db955-2645-4b32-9070-e87d7bcb485e-istio-envoy\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-hff9w\" (UID: \"559db955-2645-4b32-9070-e87d7bcb485e\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-hff9w" Oct 30 16:34:03.586563 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:03.586542 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/559db955-2645-4b32-9070-e87d7bcb485e-istio-podinfo\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-hff9w\" (UID: \"559db955-2645-4b32-9070-e87d7bcb485e\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-hff9w" Oct 30 16:34:03.606040 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:03.606021 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzs6s\" (UniqueName: \"kubernetes.io/projected/559db955-2645-4b32-9070-e87d7bcb485e-kube-api-access-dzs6s\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-hff9w\" (UID: \"559db955-2645-4b32-9070-e87d7bcb485e\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-hff9w" Oct 30 16:34:03.606143 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:03.606084 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/559db955-2645-4b32-9070-e87d7bcb485e-istio-token\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-hff9w\" (UID: \"559db955-2645-4b32-9070-e87d7bcb485e\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-hff9w" Oct 30 16:34:03.691484 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:03.691460 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-hff9w" Oct 30 16:34:03.754033 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:03.753984 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-46brl"] Oct 30 16:34:03.784391 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:03.784349 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:34:03.784543 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:03.784519 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:34:03.784646 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:03.784624 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:34:04.042745 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:04.042655 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-hff9w"] Oct 30 16:34:04.046550 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:34:04.046505 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod559db955_2645_4b32_9070_e87d7bcb485e.slice/crio-5fa9cfd7dc3091f4bdec10986926c81c7fe12ae7ccd7b56dd71ac53d9de06d30 WatchSource:0}: Error finding container 5fa9cfd7dc3091f4bdec10986926c81c7fe12ae7ccd7b56dd71ac53d9de06d30: Status 404 returned error can't find the container with id 5fa9cfd7dc3091f4bdec10986926c81c7fe12ae7ccd7b56dd71ac53d9de06d30 Oct 30 16:34:04.049373 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:04.049341 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:34:04.049462 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:04.049413 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:34:04.049462 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:04.049439 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:34:04.330498 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:04.330391 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-46brl" event={"ID":"4b8235a9-5724-4354-ad7b-eaef26b3966f","Type":"ContainerStarted","Data":"71f6f0d2b5d722966a3978154591e1372551da04dc4e5d1c33602851e57f1f3c"} Oct 30 16:34:04.330498 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:04.330476 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-46brl" event={"ID":"4b8235a9-5724-4354-ad7b-eaef26b3966f","Type":"ContainerStarted","Data":"116f02411087ca7344713291a12ea7df517231d5d0f667bfa85825ab2c0e949b"} Oct 30 16:34:04.332310 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:04.332270 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-hff9w" event={"ID":"559db955-2645-4b32-9070-e87d7bcb485e","Type":"ContainerStarted","Data":"1d5a748362b69bcf0ccd10cfb10f69389cbb92e9c484daf9086340a58d426295"} Oct 30 16:34:04.332458 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:04.332335 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-hff9w" event={"ID":"559db955-2645-4b32-9070-e87d7bcb485e","Type":"ContainerStarted","Data":"5fa9cfd7dc3091f4bdec10986926c81c7fe12ae7ccd7b56dd71ac53d9de06d30"} Oct 30 16:34:04.375870 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:04.375817 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-46brl" podStartSLOduration=1.375803393 podStartE2EDuration="1.375803393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:34:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:34:04.37481853 +0000 UTC m=+799.902153750" watchObservedRunningTime="2026-10-30 16:34:04.375803393 +0000 UTC m=+799.903138670" Oct 30 16:34:04.420576 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:04.420523 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-46brl" Oct 30 16:34:04.425636 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:04.425614 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-46brl" Oct 30 16:34:04.429991 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:04.429952 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-hff9w" podStartSLOduration=1.429939444 podStartE2EDuration="1.429939444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:34:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:34:04.428758239 +0000 UTC m=+799.956093475" watchObservedRunningTime="2026-10-30 16:34:04.429939444 +0000 UTC m=+799.957274663" Oct 30 16:34:04.693991 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:04.693953 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-hff9w" Oct 30 16:34:05.337638 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:05.337605 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-46brl" Oct 30 16:34:05.338496 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:05.338477 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-46brl" Oct 30 16:34:05.697644 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:05.697616 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-hff9w" Oct 30 16:34:06.342816 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:06.342781 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-hff9w" Oct 30 16:34:06.343601 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:06.343562 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-hff9w" Oct 30 16:34:08.555734 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:08.555705 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-263f00-64bd44f46f-9qwbt"] Oct 30 16:34:08.559200 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:08.559184 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-263f00-64bd44f46f-9qwbt" Oct 30 16:34:08.584773 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:08.584752 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-263f00-serving-cert\"" Oct 30 16:34:08.589809 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:08.589786 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-263f00-64bd44f46f-9qwbt"] Oct 30 16:34:08.638125 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:08.638090 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldwzb\" (UniqueName: \"kubernetes.io/projected/e98c90ff-9a9a-4e40-aeba-556d98be49a2-kube-api-access-ldwzb\") pod \"maas-api-e2e-worker-w4-263f00-64bd44f46f-9qwbt\" (UID: \"e98c90ff-9a9a-4e40-aeba-556d98be49a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-263f00-64bd44f46f-9qwbt" Oct 30 16:34:08.638379 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:08.638352 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e98c90ff-9a9a-4e40-aeba-556d98be49a2-maas-api-tls\") pod \"maas-api-e2e-worker-w4-263f00-64bd44f46f-9qwbt\" (UID: \"e98c90ff-9a9a-4e40-aeba-556d98be49a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-263f00-64bd44f46f-9qwbt" Oct 30 16:34:08.638487 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:08.638468 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e98c90ff-9a9a-4e40-aeba-556d98be49a2-metrics-certs\") pod \"maas-api-e2e-worker-w4-263f00-64bd44f46f-9qwbt\" (UID: \"e98c90ff-9a9a-4e40-aeba-556d98be49a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-263f00-64bd44f46f-9qwbt" Oct 30 16:34:08.638524 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:08.638503 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e98c90ff-9a9a-4e40-aeba-556d98be49a2-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-263f00-64bd44f46f-9qwbt\" (UID: \"e98c90ff-9a9a-4e40-aeba-556d98be49a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-263f00-64bd44f46f-9qwbt" Oct 30 16:34:08.739638 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:08.739567 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e98c90ff-9a9a-4e40-aeba-556d98be49a2-maas-api-tls\") pod \"maas-api-e2e-worker-w4-263f00-64bd44f46f-9qwbt\" (UID: \"e98c90ff-9a9a-4e40-aeba-556d98be49a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-263f00-64bd44f46f-9qwbt" Oct 30 16:34:08.739823 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:08.739690 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e98c90ff-9a9a-4e40-aeba-556d98be49a2-metrics-certs\") pod \"maas-api-e2e-worker-w4-263f00-64bd44f46f-9qwbt\" (UID: \"e98c90ff-9a9a-4e40-aeba-556d98be49a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-263f00-64bd44f46f-9qwbt" Oct 30 16:34:08.739823 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:08.739722 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e98c90ff-9a9a-4e40-aeba-556d98be49a2-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-263f00-64bd44f46f-9qwbt\" (UID: \"e98c90ff-9a9a-4e40-aeba-556d98be49a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-263f00-64bd44f46f-9qwbt" Oct 30 16:34:08.739823 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:08.739817 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldwzb\" (UniqueName: \"kubernetes.io/projected/e98c90ff-9a9a-4e40-aeba-556d98be49a2-kube-api-access-ldwzb\") pod \"maas-api-e2e-worker-w4-263f00-64bd44f46f-9qwbt\" (UID: \"e98c90ff-9a9a-4e40-aeba-556d98be49a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-263f00-64bd44f46f-9qwbt" Oct 30 16:34:08.740764 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:08.740726 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e98c90ff-9a9a-4e40-aeba-556d98be49a2-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-263f00-64bd44f46f-9qwbt\" (UID: \"e98c90ff-9a9a-4e40-aeba-556d98be49a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-263f00-64bd44f46f-9qwbt" Oct 30 16:34:08.742958 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:08.742928 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e98c90ff-9a9a-4e40-aeba-556d98be49a2-maas-api-tls\") pod \"maas-api-e2e-worker-w4-263f00-64bd44f46f-9qwbt\" (UID: \"e98c90ff-9a9a-4e40-aeba-556d98be49a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-263f00-64bd44f46f-9qwbt" Oct 30 16:34:08.743081 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:08.743037 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e98c90ff-9a9a-4e40-aeba-556d98be49a2-metrics-certs\") pod \"maas-api-e2e-worker-w4-263f00-64bd44f46f-9qwbt\" (UID: \"e98c90ff-9a9a-4e40-aeba-556d98be49a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-263f00-64bd44f46f-9qwbt" Oct 30 16:34:08.765379 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:08.765349 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldwzb\" (UniqueName: \"kubernetes.io/projected/e98c90ff-9a9a-4e40-aeba-556d98be49a2-kube-api-access-ldwzb\") pod \"maas-api-e2e-worker-w4-263f00-64bd44f46f-9qwbt\" (UID: \"e98c90ff-9a9a-4e40-aeba-556d98be49a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-263f00-64bd44f46f-9qwbt" Oct 30 16:34:08.872054 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:08.871966 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-263f00-64bd44f46f-9qwbt" Oct 30 16:34:08.989606 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:08.986843 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-263f00-544fdb49d4-57fv4"] Oct 30 16:34:09.014604 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:08.992863 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-263f00-544fdb49d4-57fv4" Oct 30 16:34:09.024825 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:09.024792 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-263f00\"" Oct 30 16:34:09.025721 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:09.025591 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-263f00-544fdb49d4-57fv4"] Oct 30 16:34:09.025936 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:09.025913 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-263f00-dockercfg-95v8l\"" Oct 30 16:34:09.042352 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:09.042318 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ba3e877-170c-44be-aad2-c96e7cf6b99f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-263f00-544fdb49d4-57fv4\" (UID: \"2ba3e877-170c-44be-aad2-c96e7cf6b99f\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-263f00-544fdb49d4-57fv4" Oct 30 16:34:09.042496 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:09.042379 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ls97l\" (UniqueName: \"kubernetes.io/projected/2ba3e877-170c-44be-aad2-c96e7cf6b99f-kube-api-access-ls97l\") pod \"payload-processing-e2e-worker-w4-263f00-544fdb49d4-57fv4\" (UID: \"2ba3e877-170c-44be-aad2-c96e7cf6b99f\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-263f00-544fdb49d4-57fv4" Oct 30 16:34:09.143387 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:09.143322 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ba3e877-170c-44be-aad2-c96e7cf6b99f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-263f00-544fdb49d4-57fv4\" (UID: \"2ba3e877-170c-44be-aad2-c96e7cf6b99f\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-263f00-544fdb49d4-57fv4" Oct 30 16:34:09.143387 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:09.143360 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ls97l\" (UniqueName: \"kubernetes.io/projected/2ba3e877-170c-44be-aad2-c96e7cf6b99f-kube-api-access-ls97l\") pod \"payload-processing-e2e-worker-w4-263f00-544fdb49d4-57fv4\" (UID: \"2ba3e877-170c-44be-aad2-c96e7cf6b99f\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-263f00-544fdb49d4-57fv4" Oct 30 16:34:09.144000 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:09.143981 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ba3e877-170c-44be-aad2-c96e7cf6b99f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-263f00-544fdb49d4-57fv4\" (UID: \"2ba3e877-170c-44be-aad2-c96e7cf6b99f\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-263f00-544fdb49d4-57fv4" Oct 30 16:34:09.166434 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:09.166408 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ls97l\" (UniqueName: \"kubernetes.io/projected/2ba3e877-170c-44be-aad2-c96e7cf6b99f-kube-api-access-ls97l\") pod \"payload-processing-e2e-worker-w4-263f00-544fdb49d4-57fv4\" (UID: \"2ba3e877-170c-44be-aad2-c96e7cf6b99f\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-263f00-544fdb49d4-57fv4" Oct 30 16:34:09.179100 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:09.179073 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-263f00-64bd44f46f-9qwbt"] Oct 30 16:34:09.183063 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:34:09.183040 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode98c90ff_9a9a_4e40_aeba_556d98be49a2.slice/crio-ee1618a69c3e552e086beb91f1389e4ea4f30388a1d692dd84d495b7bd1b800a WatchSource:0}: Error finding container ee1618a69c3e552e086beb91f1389e4ea4f30388a1d692dd84d495b7bd1b800a: Status 404 returned error can't find the container with id ee1618a69c3e552e086beb91f1389e4ea4f30388a1d692dd84d495b7bd1b800a Oct 30 16:34:09.329997 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:09.329967 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-263f00-544fdb49d4-57fv4" Oct 30 16:34:09.360130 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:09.360101 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-263f00-64bd44f46f-9qwbt" event={"ID":"e98c90ff-9a9a-4e40-aeba-556d98be49a2","Type":"ContainerStarted","Data":"ee1618a69c3e552e086beb91f1389e4ea4f30388a1d692dd84d495b7bd1b800a"} Oct 30 16:34:09.632809 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:09.632782 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-263f00-544fdb49d4-57fv4"] Oct 30 16:34:09.636115 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:34:09.636091 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ba3e877_170c_44be_aad2_c96e7cf6b99f.slice/crio-b19c9aff620122371e56b2fbb621f4c71b97fcb62d436f89140a65880acf8639 WatchSource:0}: Error finding container b19c9aff620122371e56b2fbb621f4c71b97fcb62d436f89140a65880acf8639: Status 404 returned error can't find the container with id b19c9aff620122371e56b2fbb621f4c71b97fcb62d436f89140a65880acf8639 Oct 30 16:34:10.367625 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:10.367566 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-263f00-64bd44f46f-9qwbt" event={"ID":"e98c90ff-9a9a-4e40-aeba-556d98be49a2","Type":"ContainerStarted","Data":"9ebcf9cd8aecab35e3019f3833c85e00d7e3c5ad9da1109061d418a396ef186c"} Oct 30 16:34:10.367836 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:10.367657 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-263f00-64bd44f46f-9qwbt" Oct 30 16:34:10.369095 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:10.369070 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-263f00-544fdb49d4-57fv4" event={"ID":"2ba3e877-170c-44be-aad2-c96e7cf6b99f","Type":"ContainerStarted","Data":"f187b659f6e6225e2b24ce32a0e56339a688c86e53bba2910fbbf8b4d4f779fb"} Oct 30 16:34:10.369207 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:10.369116 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-263f00-544fdb49d4-57fv4" Oct 30 16:34:10.369207 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:10.369133 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-263f00-544fdb49d4-57fv4" event={"ID":"2ba3e877-170c-44be-aad2-c96e7cf6b99f","Type":"ContainerStarted","Data":"b19c9aff620122371e56b2fbb621f4c71b97fcb62d436f89140a65880acf8639"} Oct 30 16:34:10.411835 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:10.411771 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-263f00-64bd44f46f-9qwbt" podStartSLOduration=2.074513114 podStartE2EDuration="2.41175468s" totalImagesPullingTime="337.241566ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:34:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:34:10.409289663 +0000 UTC m=+805.936624882" watchObservedRunningTime="2026-10-30 16:34:10.41175468 +0000 UTC m=+805.939089901" Oct 30 16:34:10.436413 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:10.436378 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-3e2f96-57dd88bc59-l2k57"] Oct 30 16:34:10.440473 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:10.440447 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3e2f96-57dd88bc59-l2k57" Oct 30 16:34:10.453606 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:10.453561 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7adb9b2-ac3c-440c-8338-f0a4375625d3-metrics-certs\") pod \"maas-api-e2e-worker-w0-3e2f96-57dd88bc59-l2k57\" (UID: \"d7adb9b2-ac3c-440c-8338-f0a4375625d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3e2f96-57dd88bc59-l2k57" Oct 30 16:34:10.453764 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:10.453721 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvz7h\" (UniqueName: \"kubernetes.io/projected/d7adb9b2-ac3c-440c-8338-f0a4375625d3-kube-api-access-mvz7h\") pod \"maas-api-e2e-worker-w0-3e2f96-57dd88bc59-l2k57\" (UID: \"d7adb9b2-ac3c-440c-8338-f0a4375625d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3e2f96-57dd88bc59-l2k57" Oct 30 16:34:10.453875 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:10.453856 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7adb9b2-ac3c-440c-8338-f0a4375625d3-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-3e2f96-57dd88bc59-l2k57\" (UID: \"d7adb9b2-ac3c-440c-8338-f0a4375625d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3e2f96-57dd88bc59-l2k57" Oct 30 16:34:10.454252 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:10.454229 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7adb9b2-ac3c-440c-8338-f0a4375625d3-maas-api-tls\") pod \"maas-api-e2e-worker-w0-3e2f96-57dd88bc59-l2k57\" (UID: \"d7adb9b2-ac3c-440c-8338-f0a4375625d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3e2f96-57dd88bc59-l2k57" Oct 30 16:34:10.456894 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:10.456854 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-263f00-544fdb49d4-57fv4" podStartSLOduration=2.456842652 podStartE2EDuration="2.456842652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:34:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:34:10.454613747 +0000 UTC m=+805.981948971" watchObservedRunningTime="2026-10-30 16:34:10.456842652 +0000 UTC m=+805.984177870" Oct 30 16:34:10.463882 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:10.463852 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-3e2f96-57dd88bc59-l2k57"] Oct 30 16:34:10.494941 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:10.494797 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-3e2f96-serving-cert\"" Oct 30 16:34:10.555959 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:10.555927 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7adb9b2-ac3c-440c-8338-f0a4375625d3-metrics-certs\") pod \"maas-api-e2e-worker-w0-3e2f96-57dd88bc59-l2k57\" (UID: \"d7adb9b2-ac3c-440c-8338-f0a4375625d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3e2f96-57dd88bc59-l2k57" Oct 30 16:34:10.556169 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:10.556018 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvz7h\" (UniqueName: \"kubernetes.io/projected/d7adb9b2-ac3c-440c-8338-f0a4375625d3-kube-api-access-mvz7h\") pod \"maas-api-e2e-worker-w0-3e2f96-57dd88bc59-l2k57\" (UID: \"d7adb9b2-ac3c-440c-8338-f0a4375625d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3e2f96-57dd88bc59-l2k57" Oct 30 16:34:10.556169 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:10.556073 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7adb9b2-ac3c-440c-8338-f0a4375625d3-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-3e2f96-57dd88bc59-l2k57\" (UID: \"d7adb9b2-ac3c-440c-8338-f0a4375625d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3e2f96-57dd88bc59-l2k57" Oct 30 16:34:10.556169 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:10.556155 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7adb9b2-ac3c-440c-8338-f0a4375625d3-maas-api-tls\") pod \"maas-api-e2e-worker-w0-3e2f96-57dd88bc59-l2k57\" (UID: \"d7adb9b2-ac3c-440c-8338-f0a4375625d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3e2f96-57dd88bc59-l2k57" Oct 30 16:34:10.556331 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:34:10.556283 5052 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-3e2f96-serving-cert: secret "maas-api-e2e-worker-w0-3e2f96-serving-cert" not found Oct 30 16:34:10.556374 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:34:10.556350 5052 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7adb9b2-ac3c-440c-8338-f0a4375625d3-maas-api-tls podName:d7adb9b2-ac3c-440c-8338-f0a4375625d3 nodeName:}" failed. No retries permitted until 2026-10-30 16:34:11.056330396 +0000 UTC m=+806.583665607 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d7adb9b2-ac3c-440c-8338-f0a4375625d3-maas-api-tls") pod "maas-api-e2e-worker-w0-3e2f96-57dd88bc59-l2k57" (UID: "d7adb9b2-ac3c-440c-8338-f0a4375625d3") : secret "maas-api-e2e-worker-w0-3e2f96-serving-cert" not found Oct 30 16:34:10.556964 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:10.556940 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7adb9b2-ac3c-440c-8338-f0a4375625d3-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-3e2f96-57dd88bc59-l2k57\" (UID: \"d7adb9b2-ac3c-440c-8338-f0a4375625d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3e2f96-57dd88bc59-l2k57" Oct 30 16:34:10.558464 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:10.558421 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7adb9b2-ac3c-440c-8338-f0a4375625d3-metrics-certs\") pod \"maas-api-e2e-worker-w0-3e2f96-57dd88bc59-l2k57\" (UID: \"d7adb9b2-ac3c-440c-8338-f0a4375625d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3e2f96-57dd88bc59-l2k57" Oct 30 16:34:10.580628 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:10.580600 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvz7h\" (UniqueName: \"kubernetes.io/projected/d7adb9b2-ac3c-440c-8338-f0a4375625d3-kube-api-access-mvz7h\") pod \"maas-api-e2e-worker-w0-3e2f96-57dd88bc59-l2k57\" (UID: \"d7adb9b2-ac3c-440c-8338-f0a4375625d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3e2f96-57dd88bc59-l2k57" Oct 30 16:34:11.060768 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:11.060731 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7adb9b2-ac3c-440c-8338-f0a4375625d3-maas-api-tls\") pod \"maas-api-e2e-worker-w0-3e2f96-57dd88bc59-l2k57\" (UID: \"d7adb9b2-ac3c-440c-8338-f0a4375625d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3e2f96-57dd88bc59-l2k57" Oct 30 16:34:11.063107 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:11.063080 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7adb9b2-ac3c-440c-8338-f0a4375625d3-maas-api-tls\") pod \"maas-api-e2e-worker-w0-3e2f96-57dd88bc59-l2k57\" (UID: \"d7adb9b2-ac3c-440c-8338-f0a4375625d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3e2f96-57dd88bc59-l2k57" Oct 30 16:34:11.354484 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:11.354391 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3e2f96-57dd88bc59-l2k57" Oct 30 16:34:11.681605 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:11.681551 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-3e2f96-57dd88bc59-l2k57"] Oct 30 16:34:11.685304 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:34:11.685273 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7adb9b2_ac3c_440c_8338_f0a4375625d3.slice/crio-6a2dc837f0d90dc65fa4d50d8df81b581b5c7a4f2078c864fb2682a8505b7724 WatchSource:0}: Error finding container 6a2dc837f0d90dc65fa4d50d8df81b581b5c7a4f2078c864fb2682a8505b7724: Status 404 returned error can't find the container with id 6a2dc837f0d90dc65fa4d50d8df81b581b5c7a4f2078c864fb2682a8505b7724 Oct 30 16:34:12.142699 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:12.142666 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-750dcb-56df54cc5-nsm9f"] Oct 30 16:34:12.146720 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:12.146696 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-750dcb-56df54cc5-nsm9f" Oct 30 16:34:12.174139 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:12.172994 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsslt\" (UniqueName: \"kubernetes.io/projected/d56798b3-ff19-4644-8d23-2967f523427d-kube-api-access-vsslt\") pod \"maas-api-e2e-worker-w2-750dcb-56df54cc5-nsm9f\" (UID: \"d56798b3-ff19-4644-8d23-2967f523427d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-750dcb-56df54cc5-nsm9f" Oct 30 16:34:12.174139 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:12.173038 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d56798b3-ff19-4644-8d23-2967f523427d-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-750dcb-56df54cc5-nsm9f\" (UID: \"d56798b3-ff19-4644-8d23-2967f523427d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-750dcb-56df54cc5-nsm9f" Oct 30 16:34:12.174139 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:12.173195 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d56798b3-ff19-4644-8d23-2967f523427d-maas-api-tls\") pod \"maas-api-e2e-worker-w2-750dcb-56df54cc5-nsm9f\" (UID: \"d56798b3-ff19-4644-8d23-2967f523427d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-750dcb-56df54cc5-nsm9f" Oct 30 16:34:12.174139 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:12.173232 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d56798b3-ff19-4644-8d23-2967f523427d-metrics-certs\") pod \"maas-api-e2e-worker-w2-750dcb-56df54cc5-nsm9f\" (UID: \"d56798b3-ff19-4644-8d23-2967f523427d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-750dcb-56df54cc5-nsm9f" Oct 30 16:34:12.178377 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:12.178349 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-750dcb-serving-cert\"" Oct 30 16:34:12.179256 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:12.179227 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-750dcb-56df54cc5-nsm9f"] Oct 30 16:34:12.216541 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:12.216511 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cdwcj"] Oct 30 16:34:12.274771 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:12.274736 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsslt\" (UniqueName: \"kubernetes.io/projected/d56798b3-ff19-4644-8d23-2967f523427d-kube-api-access-vsslt\") pod \"maas-api-e2e-worker-w2-750dcb-56df54cc5-nsm9f\" (UID: \"d56798b3-ff19-4644-8d23-2967f523427d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-750dcb-56df54cc5-nsm9f" Oct 30 16:34:12.274953 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:12.274780 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d56798b3-ff19-4644-8d23-2967f523427d-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-750dcb-56df54cc5-nsm9f\" (UID: \"d56798b3-ff19-4644-8d23-2967f523427d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-750dcb-56df54cc5-nsm9f" Oct 30 16:34:12.274953 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:12.274847 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d56798b3-ff19-4644-8d23-2967f523427d-maas-api-tls\") pod \"maas-api-e2e-worker-w2-750dcb-56df54cc5-nsm9f\" (UID: \"d56798b3-ff19-4644-8d23-2967f523427d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-750dcb-56df54cc5-nsm9f" Oct 30 16:34:12.275074 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:34:12.274975 5052 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w2-750dcb-serving-cert: secret "maas-api-e2e-worker-w2-750dcb-serving-cert" not found Oct 30 16:34:12.275074 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:34:12.275037 5052 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d56798b3-ff19-4644-8d23-2967f523427d-maas-api-tls podName:d56798b3-ff19-4644-8d23-2967f523427d nodeName:}" failed. No retries permitted until 2026-10-30 16:34:12.775015594 +0000 UTC m=+808.302350793 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d56798b3-ff19-4644-8d23-2967f523427d-maas-api-tls") pod "maas-api-e2e-worker-w2-750dcb-56df54cc5-nsm9f" (UID: "d56798b3-ff19-4644-8d23-2967f523427d") : secret "maas-api-e2e-worker-w2-750dcb-serving-cert" not found Oct 30 16:34:12.275256 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:12.275237 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d56798b3-ff19-4644-8d23-2967f523427d-metrics-certs\") pod \"maas-api-e2e-worker-w2-750dcb-56df54cc5-nsm9f\" (UID: \"d56798b3-ff19-4644-8d23-2967f523427d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-750dcb-56df54cc5-nsm9f" Oct 30 16:34:12.275630 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:12.275570 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d56798b3-ff19-4644-8d23-2967f523427d-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-750dcb-56df54cc5-nsm9f\" (UID: \"d56798b3-ff19-4644-8d23-2967f523427d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-750dcb-56df54cc5-nsm9f" Oct 30 16:34:12.278166 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:12.278112 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d56798b3-ff19-4644-8d23-2967f523427d-metrics-certs\") pod \"maas-api-e2e-worker-w2-750dcb-56df54cc5-nsm9f\" (UID: \"d56798b3-ff19-4644-8d23-2967f523427d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-750dcb-56df54cc5-nsm9f" Oct 30 16:34:12.297908 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:12.297884 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsslt\" (UniqueName: \"kubernetes.io/projected/d56798b3-ff19-4644-8d23-2967f523427d-kube-api-access-vsslt\") pod \"maas-api-e2e-worker-w2-750dcb-56df54cc5-nsm9f\" (UID: \"d56798b3-ff19-4644-8d23-2967f523427d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-750dcb-56df54cc5-nsm9f" Oct 30 16:34:12.388659 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:12.388621 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3e2f96-57dd88bc59-l2k57" event={"ID":"d7adb9b2-ac3c-440c-8338-f0a4375625d3","Type":"ContainerStarted","Data":"4ea7a2413f58480d7336dbb505e6335cf54cb4993d1cc08035b42ce867665604"} Oct 30 16:34:12.388834 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:12.388693 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3e2f96-57dd88bc59-l2k57" event={"ID":"d7adb9b2-ac3c-440c-8338-f0a4375625d3","Type":"ContainerStarted","Data":"6a2dc837f0d90dc65fa4d50d8df81b581b5c7a4f2078c864fb2682a8505b7724"} Oct 30 16:34:12.388834 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:12.388725 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3e2f96-57dd88bc59-l2k57" Oct 30 16:34:12.781743 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:12.781558 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d56798b3-ff19-4644-8d23-2967f523427d-maas-api-tls\") pod \"maas-api-e2e-worker-w2-750dcb-56df54cc5-nsm9f\" (UID: \"d56798b3-ff19-4644-8d23-2967f523427d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-750dcb-56df54cc5-nsm9f" Oct 30 16:34:12.784867 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:12.784830 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d56798b3-ff19-4644-8d23-2967f523427d-maas-api-tls\") pod \"maas-api-e2e-worker-w2-750dcb-56df54cc5-nsm9f\" (UID: \"d56798b3-ff19-4644-8d23-2967f523427d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-750dcb-56df54cc5-nsm9f" Oct 30 16:34:13.062206 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:13.062125 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-750dcb-56df54cc5-nsm9f" Oct 30 16:34:13.363921 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:13.363823 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3e2f96-57dd88bc59-l2k57" podStartSLOduration=3.023252623 podStartE2EDuration="3.363808109s" totalImagesPullingTime="340.555486ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:34:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:34:12.438950616 +0000 UTC m=+807.966285835" watchObservedRunningTime="2026-10-30 16:34:13.363808109 +0000 UTC m=+808.891143324" Oct 30 16:34:13.364341 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:13.364087 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-750dcb-56df54cc5-nsm9f"] Oct 30 16:34:13.368101 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:34:13.368069 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd56798b3_ff19_4644_8d23_2967f523427d.slice/crio-ce9db742e719d23196e32f7f014af9f14e87aee1c6296cbddaf122be17d8030d WatchSource:0}: Error finding container ce9db742e719d23196e32f7f014af9f14e87aee1c6296cbddaf122be17d8030d: Status 404 returned error can't find the container with id ce9db742e719d23196e32f7f014af9f14e87aee1c6296cbddaf122be17d8030d Oct 30 16:34:13.396032 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:13.395990 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-750dcb-56df54cc5-nsm9f" event={"ID":"d56798b3-ff19-4644-8d23-2967f523427d","Type":"ContainerStarted","Data":"ce9db742e719d23196e32f7f014af9f14e87aee1c6296cbddaf122be17d8030d"} Oct 30 16:34:14.028913 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:14.028878 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-0b2348-777785f486-tcq45"] Oct 30 16:34:14.032850 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:14.032828 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-0b2348-777785f486-tcq45" Oct 30 16:34:14.059195 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:14.059119 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-0b2348-serving-cert\"" Oct 30 16:34:14.062112 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:14.062087 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-0b2348-777785f486-tcq45"] Oct 30 16:34:14.098711 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:14.098682 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f58433e-03f7-4e7c-95b5-3d9871cce2f5-metrics-certs\") pod \"maas-api-e2e-worker-w3-0b2348-777785f486-tcq45\" (UID: \"0f58433e-03f7-4e7c-95b5-3d9871cce2f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-0b2348-777785f486-tcq45" Oct 30 16:34:14.098884 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:14.098785 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f58433e-03f7-4e7c-95b5-3d9871cce2f5-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-0b2348-777785f486-tcq45\" (UID: \"0f58433e-03f7-4e7c-95b5-3d9871cce2f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-0b2348-777785f486-tcq45" Oct 30 16:34:14.098948 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:14.098877 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f58433e-03f7-4e7c-95b5-3d9871cce2f5-maas-api-tls\") pod \"maas-api-e2e-worker-w3-0b2348-777785f486-tcq45\" (UID: \"0f58433e-03f7-4e7c-95b5-3d9871cce2f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-0b2348-777785f486-tcq45" Oct 30 16:34:14.098948 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:14.098912 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xg5jp\" (UniqueName: \"kubernetes.io/projected/0f58433e-03f7-4e7c-95b5-3d9871cce2f5-kube-api-access-xg5jp\") pod \"maas-api-e2e-worker-w3-0b2348-777785f486-tcq45\" (UID: \"0f58433e-03f7-4e7c-95b5-3d9871cce2f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-0b2348-777785f486-tcq45" Oct 30 16:34:14.199658 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:14.199625 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f58433e-03f7-4e7c-95b5-3d9871cce2f5-maas-api-tls\") pod \"maas-api-e2e-worker-w3-0b2348-777785f486-tcq45\" (UID: \"0f58433e-03f7-4e7c-95b5-3d9871cce2f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-0b2348-777785f486-tcq45" Oct 30 16:34:14.199658 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:14.199663 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xg5jp\" (UniqueName: \"kubernetes.io/projected/0f58433e-03f7-4e7c-95b5-3d9871cce2f5-kube-api-access-xg5jp\") pod \"maas-api-e2e-worker-w3-0b2348-777785f486-tcq45\" (UID: \"0f58433e-03f7-4e7c-95b5-3d9871cce2f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-0b2348-777785f486-tcq45" Oct 30 16:34:14.199863 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:14.199723 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f58433e-03f7-4e7c-95b5-3d9871cce2f5-metrics-certs\") pod \"maas-api-e2e-worker-w3-0b2348-777785f486-tcq45\" (UID: \"0f58433e-03f7-4e7c-95b5-3d9871cce2f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-0b2348-777785f486-tcq45" Oct 30 16:34:14.199863 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:14.199786 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f58433e-03f7-4e7c-95b5-3d9871cce2f5-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-0b2348-777785f486-tcq45\" (UID: \"0f58433e-03f7-4e7c-95b5-3d9871cce2f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-0b2348-777785f486-tcq45" Oct 30 16:34:14.200445 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:14.200421 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f58433e-03f7-4e7c-95b5-3d9871cce2f5-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-0b2348-777785f486-tcq45\" (UID: \"0f58433e-03f7-4e7c-95b5-3d9871cce2f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-0b2348-777785f486-tcq45" Oct 30 16:34:14.202102 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:14.202075 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f58433e-03f7-4e7c-95b5-3d9871cce2f5-maas-api-tls\") pod \"maas-api-e2e-worker-w3-0b2348-777785f486-tcq45\" (UID: \"0f58433e-03f7-4e7c-95b5-3d9871cce2f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-0b2348-777785f486-tcq45" Oct 30 16:34:14.202176 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:14.202149 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f58433e-03f7-4e7c-95b5-3d9871cce2f5-metrics-certs\") pod \"maas-api-e2e-worker-w3-0b2348-777785f486-tcq45\" (UID: \"0f58433e-03f7-4e7c-95b5-3d9871cce2f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-0b2348-777785f486-tcq45" Oct 30 16:34:14.222523 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:14.222498 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xg5jp\" (UniqueName: \"kubernetes.io/projected/0f58433e-03f7-4e7c-95b5-3d9871cce2f5-kube-api-access-xg5jp\") pod \"maas-api-e2e-worker-w3-0b2348-777785f486-tcq45\" (UID: \"0f58433e-03f7-4e7c-95b5-3d9871cce2f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-0b2348-777785f486-tcq45" Oct 30 16:34:14.345776 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:14.345695 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-0b2348-777785f486-tcq45" Oct 30 16:34:14.409032 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:14.408999 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-750dcb-56df54cc5-nsm9f" event={"ID":"d56798b3-ff19-4644-8d23-2967f523427d","Type":"ContainerStarted","Data":"cb10f1d06b15e55269145534c2d31d96dd48ea2c8968b2c868250b4727c84c0b"} Oct 30 16:34:14.409895 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:14.409872 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-750dcb-56df54cc5-nsm9f" Oct 30 16:34:14.453045 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:14.452996 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-750dcb-56df54cc5-nsm9f" podStartSLOduration=2.022400542 podStartE2EDuration="2.45298177s" totalImagesPullingTime="430.581228ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:34:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:34:14.452280582 +0000 UTC m=+809.979615800" watchObservedRunningTime="2026-10-30 16:34:14.45298177 +0000 UTC m=+809.980316989" Oct 30 16:34:14.655881 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:14.655853 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-0b2348-777785f486-tcq45"] Oct 30 16:34:14.659732 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:34:14.659698 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f58433e_03f7_4e7c_95b5_3d9871cce2f5.slice/crio-2b7b4fd283778928c9dd8c3273219592816128353d42322f09d6eec6a9d43c25 WatchSource:0}: Error finding container 2b7b4fd283778928c9dd8c3273219592816128353d42322f09d6eec6a9d43c25: Status 404 returned error can't find the container with id 2b7b4fd283778928c9dd8c3273219592816128353d42322f09d6eec6a9d43c25 Oct 30 16:34:15.418234 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:15.418197 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-0b2348-777785f486-tcq45" event={"ID":"0f58433e-03f7-4e7c-95b5-3d9871cce2f5","Type":"ContainerStarted","Data":"3203f0db909bf86e8cc44e6502b4bf7bd64847ae8d534960420e6ad6e531c69c"} Oct 30 16:34:15.419064 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:15.418263 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-0b2348-777785f486-tcq45" event={"ID":"0f58433e-03f7-4e7c-95b5-3d9871cce2f5","Type":"ContainerStarted","Data":"2b7b4fd283778928c9dd8c3273219592816128353d42322f09d6eec6a9d43c25"} Oct 30 16:34:15.419064 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:15.418920 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-0b2348-777785f486-tcq45" Oct 30 16:34:16.388075 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:16.388044 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-263f00-64bd44f46f-9qwbt" Oct 30 16:34:16.435640 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:16.435553 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-0b2348-777785f486-tcq45" podStartSLOduration=2.105201937 podStartE2EDuration="2.435534192s" totalImagesPullingTime="330.332255ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:34:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:34:15.471872839 +0000 UTC m=+810.999208058" watchObservedRunningTime="2026-10-30 16:34:16.435534192 +0000 UTC m=+811.962869412" Oct 30 16:34:18.401779 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:18.401750 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3e2f96-57dd88bc59-l2k57" Oct 30 16:34:18.839986 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:34:18.839896 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 16:34:19.651963 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:19.651926 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cdwcj"] Oct 30 16:34:21.383808 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:21.383780 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-263f00-544fdb49d4-57fv4" Oct 30 16:34:21.447785 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:21.447753 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-0b2348-777785f486-tcq45" Oct 30 16:34:21.449117 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:21.449095 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-750dcb-56df54cc5-nsm9f" Oct 30 16:34:26.277790 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:26.277753 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-967fac-7cbc897887-drl9k"] Oct 30 16:34:26.282012 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:26.281991 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-967fac-7cbc897887-drl9k" Oct 30 16:34:26.310962 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:26.310943 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w5-967fac-serving-cert\"" Oct 30 16:34:26.313732 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:26.313707 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-967fac-7cbc897887-drl9k"] Oct 30 16:34:26.359946 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:26.359908 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jcwt\" (UniqueName: \"kubernetes.io/projected/051f70f5-3c9d-44e7-9d47-d5b21feeee6e-kube-api-access-2jcwt\") pod \"maas-api-e2e-worker-w5-967fac-7cbc897887-drl9k\" (UID: \"051f70f5-3c9d-44e7-9d47-d5b21feeee6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-967fac-7cbc897887-drl9k" Oct 30 16:34:26.360102 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:26.359958 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/051f70f5-3c9d-44e7-9d47-d5b21feeee6e-maas-api-tls\") pod \"maas-api-e2e-worker-w5-967fac-7cbc897887-drl9k\" (UID: \"051f70f5-3c9d-44e7-9d47-d5b21feeee6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-967fac-7cbc897887-drl9k" Oct 30 16:34:26.360102 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:26.360003 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/051f70f5-3c9d-44e7-9d47-d5b21feeee6e-metrics-certs\") pod \"maas-api-e2e-worker-w5-967fac-7cbc897887-drl9k\" (UID: \"051f70f5-3c9d-44e7-9d47-d5b21feeee6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-967fac-7cbc897887-drl9k" Oct 30 16:34:26.360102 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:26.360022 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/051f70f5-3c9d-44e7-9d47-d5b21feeee6e-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-967fac-7cbc897887-drl9k\" (UID: \"051f70f5-3c9d-44e7-9d47-d5b21feeee6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-967fac-7cbc897887-drl9k" Oct 30 16:34:26.473603 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:34:26.462205 5052 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w5-967fac-serving-cert: secret "maas-api-e2e-worker-w5-967fac-serving-cert" not found Oct 30 16:34:26.473603 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:26.462891 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/051f70f5-3c9d-44e7-9d47-d5b21feeee6e-maas-api-tls\") pod \"maas-api-e2e-worker-w5-967fac-7cbc897887-drl9k\" (UID: \"051f70f5-3c9d-44e7-9d47-d5b21feeee6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-967fac-7cbc897887-drl9k" Oct 30 16:34:26.473603 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:26.463007 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/051f70f5-3c9d-44e7-9d47-d5b21feeee6e-metrics-certs\") pod \"maas-api-e2e-worker-w5-967fac-7cbc897887-drl9k\" (UID: \"051f70f5-3c9d-44e7-9d47-d5b21feeee6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-967fac-7cbc897887-drl9k" Oct 30 16:34:26.473603 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:26.463051 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/051f70f5-3c9d-44e7-9d47-d5b21feeee6e-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-967fac-7cbc897887-drl9k\" (UID: \"051f70f5-3c9d-44e7-9d47-d5b21feeee6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-967fac-7cbc897887-drl9k" Oct 30 16:34:26.473603 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:26.463181 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jcwt\" (UniqueName: \"kubernetes.io/projected/051f70f5-3c9d-44e7-9d47-d5b21feeee6e-kube-api-access-2jcwt\") pod \"maas-api-e2e-worker-w5-967fac-7cbc897887-drl9k\" (UID: \"051f70f5-3c9d-44e7-9d47-d5b21feeee6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-967fac-7cbc897887-drl9k" Oct 30 16:34:26.473603 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:34:26.463508 5052 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/051f70f5-3c9d-44e7-9d47-d5b21feeee6e-maas-api-tls podName:051f70f5-3c9d-44e7-9d47-d5b21feeee6e nodeName:}" failed. No retries permitted until 2026-10-30 16:34:26.963485538 +0000 UTC m=+822.490820745 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/051f70f5-3c9d-44e7-9d47-d5b21feeee6e-maas-api-tls") pod "maas-api-e2e-worker-w5-967fac-7cbc897887-drl9k" (UID: "051f70f5-3c9d-44e7-9d47-d5b21feeee6e") : secret "maas-api-e2e-worker-w5-967fac-serving-cert" not found Oct 30 16:34:26.473603 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:26.464458 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/051f70f5-3c9d-44e7-9d47-d5b21feeee6e-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-967fac-7cbc897887-drl9k\" (UID: \"051f70f5-3c9d-44e7-9d47-d5b21feeee6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-967fac-7cbc897887-drl9k" Oct 30 16:34:26.473603 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:26.473331 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/051f70f5-3c9d-44e7-9d47-d5b21feeee6e-metrics-certs\") pod \"maas-api-e2e-worker-w5-967fac-7cbc897887-drl9k\" (UID: \"051f70f5-3c9d-44e7-9d47-d5b21feeee6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-967fac-7cbc897887-drl9k" Oct 30 16:34:26.489574 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:26.489509 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jcwt\" (UniqueName: \"kubernetes.io/projected/051f70f5-3c9d-44e7-9d47-d5b21feeee6e-kube-api-access-2jcwt\") pod \"maas-api-e2e-worker-w5-967fac-7cbc897887-drl9k\" (UID: \"051f70f5-3c9d-44e7-9d47-d5b21feeee6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-967fac-7cbc897887-drl9k" Oct 30 16:34:26.969014 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:26.968976 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/051f70f5-3c9d-44e7-9d47-d5b21feeee6e-maas-api-tls\") pod \"maas-api-e2e-worker-w5-967fac-7cbc897887-drl9k\" (UID: \"051f70f5-3c9d-44e7-9d47-d5b21feeee6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-967fac-7cbc897887-drl9k" Oct 30 16:34:26.971781 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:26.971756 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/051f70f5-3c9d-44e7-9d47-d5b21feeee6e-maas-api-tls\") pod \"maas-api-e2e-worker-w5-967fac-7cbc897887-drl9k\" (UID: \"051f70f5-3c9d-44e7-9d47-d5b21feeee6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-967fac-7cbc897887-drl9k" Oct 30 16:34:27.194785 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:27.194759 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-967fac-7cbc897887-drl9k" Oct 30 16:34:27.501221 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:27.501195 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-967fac-7cbc897887-drl9k"] Oct 30 16:34:27.504056 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:34:27.504006 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod051f70f5_3c9d_44e7_9d47_d5b21feeee6e.slice/crio-6cfd84e4a9facfa944db9a50732fd292c3ca0c8fb2416c348b3b42b272e4c036 WatchSource:0}: Error finding container 6cfd84e4a9facfa944db9a50732fd292c3ca0c8fb2416c348b3b42b272e4c036: Status 404 returned error can't find the container with id 6cfd84e4a9facfa944db9a50732fd292c3ca0c8fb2416c348b3b42b272e4c036 Oct 30 16:34:28.516984 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:28.516951 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-967fac-7cbc897887-drl9k" event={"ID":"051f70f5-3c9d-44e7-9d47-d5b21feeee6e","Type":"ContainerStarted","Data":"870984cc4d77133579753418ba7ac24b263679c34e5c59b9255a908e704fbb3b"} Oct 30 16:34:28.517386 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:28.517007 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-967fac-7cbc897887-drl9k" event={"ID":"051f70f5-3c9d-44e7-9d47-d5b21feeee6e","Type":"ContainerStarted","Data":"6cfd84e4a9facfa944db9a50732fd292c3ca0c8fb2416c348b3b42b272e4c036"} Oct 30 16:34:28.517386 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:28.517086 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-967fac-7cbc897887-drl9k" Oct 30 16:34:28.560915 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:28.560866 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-967fac-7cbc897887-drl9k" podStartSLOduration=2.19184026 podStartE2EDuration="2.560852994s" totalImagesPullingTime="369.012734ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:34:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:34:28.5585227 +0000 UTC m=+824.085857921" watchObservedRunningTime="2026-10-30 16:34:28.560852994 +0000 UTC m=+824.088188212" Oct 30 16:34:29.262555 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:29.262169 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cdwcj"] Oct 30 16:34:29.722391 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:29.722356 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-a8c22d-7ff9d946b7-xqlzn"] Oct 30 16:34:29.727513 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:29.727488 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a8c22d-7ff9d946b7-xqlzn" Oct 30 16:34:29.802639 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:29.776402 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-a8c22d-serving-cert\"" Oct 30 16:34:29.802639 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:29.796755 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-a8c22d-7ff9d946b7-xqlzn"] Oct 30 16:34:29.805207 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:29.805078 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e68b78bc-50ac-43ee-8354-a11b921e2713-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-a8c22d-7ff9d946b7-xqlzn\" (UID: \"e68b78bc-50ac-43ee-8354-a11b921e2713\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a8c22d-7ff9d946b7-xqlzn" Oct 30 16:34:29.805207 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:29.805181 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e68b78bc-50ac-43ee-8354-a11b921e2713-maas-api-tls\") pod \"maas-api-e2e-worker-w1-a8c22d-7ff9d946b7-xqlzn\" (UID: \"e68b78bc-50ac-43ee-8354-a11b921e2713\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a8c22d-7ff9d946b7-xqlzn" Oct 30 16:34:29.805466 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:29.805335 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e68b78bc-50ac-43ee-8354-a11b921e2713-metrics-certs\") pod \"maas-api-e2e-worker-w1-a8c22d-7ff9d946b7-xqlzn\" (UID: \"e68b78bc-50ac-43ee-8354-a11b921e2713\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a8c22d-7ff9d946b7-xqlzn" Oct 30 16:34:29.805466 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:29.805385 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwvvh\" (UniqueName: \"kubernetes.io/projected/e68b78bc-50ac-43ee-8354-a11b921e2713-kube-api-access-dwvvh\") pod \"maas-api-e2e-worker-w1-a8c22d-7ff9d946b7-xqlzn\" (UID: \"e68b78bc-50ac-43ee-8354-a11b921e2713\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a8c22d-7ff9d946b7-xqlzn" Oct 30 16:34:29.906959 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:29.906911 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e68b78bc-50ac-43ee-8354-a11b921e2713-metrics-certs\") pod \"maas-api-e2e-worker-w1-a8c22d-7ff9d946b7-xqlzn\" (UID: \"e68b78bc-50ac-43ee-8354-a11b921e2713\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a8c22d-7ff9d946b7-xqlzn" Oct 30 16:34:29.907156 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:29.906965 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwvvh\" (UniqueName: \"kubernetes.io/projected/e68b78bc-50ac-43ee-8354-a11b921e2713-kube-api-access-dwvvh\") pod \"maas-api-e2e-worker-w1-a8c22d-7ff9d946b7-xqlzn\" (UID: \"e68b78bc-50ac-43ee-8354-a11b921e2713\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a8c22d-7ff9d946b7-xqlzn" Oct 30 16:34:29.907156 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:29.906998 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e68b78bc-50ac-43ee-8354-a11b921e2713-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-a8c22d-7ff9d946b7-xqlzn\" (UID: \"e68b78bc-50ac-43ee-8354-a11b921e2713\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a8c22d-7ff9d946b7-xqlzn" Oct 30 16:34:29.907156 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:29.907058 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e68b78bc-50ac-43ee-8354-a11b921e2713-maas-api-tls\") pod \"maas-api-e2e-worker-w1-a8c22d-7ff9d946b7-xqlzn\" (UID: \"e68b78bc-50ac-43ee-8354-a11b921e2713\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a8c22d-7ff9d946b7-xqlzn" Oct 30 16:34:29.907336 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:34:29.907206 5052 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w1-a8c22d-serving-cert: secret "maas-api-e2e-worker-w1-a8c22d-serving-cert" not found Oct 30 16:34:29.907336 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:34:29.907258 5052 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e68b78bc-50ac-43ee-8354-a11b921e2713-maas-api-tls podName:e68b78bc-50ac-43ee-8354-a11b921e2713 nodeName:}" failed. No retries permitted until 2026-10-30 16:34:30.407242641 +0000 UTC m=+825.934577852 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e68b78bc-50ac-43ee-8354-a11b921e2713-maas-api-tls") pod "maas-api-e2e-worker-w1-a8c22d-7ff9d946b7-xqlzn" (UID: "e68b78bc-50ac-43ee-8354-a11b921e2713") : secret "maas-api-e2e-worker-w1-a8c22d-serving-cert" not found Oct 30 16:34:29.909262 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:29.909230 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e68b78bc-50ac-43ee-8354-a11b921e2713-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-a8c22d-7ff9d946b7-xqlzn\" (UID: \"e68b78bc-50ac-43ee-8354-a11b921e2713\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a8c22d-7ff9d946b7-xqlzn" Oct 30 16:34:29.918324 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:29.918293 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e68b78bc-50ac-43ee-8354-a11b921e2713-metrics-certs\") pod \"maas-api-e2e-worker-w1-a8c22d-7ff9d946b7-xqlzn\" (UID: \"e68b78bc-50ac-43ee-8354-a11b921e2713\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a8c22d-7ff9d946b7-xqlzn" Oct 30 16:34:29.953392 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:29.953341 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwvvh\" (UniqueName: \"kubernetes.io/projected/e68b78bc-50ac-43ee-8354-a11b921e2713-kube-api-access-dwvvh\") pod \"maas-api-e2e-worker-w1-a8c22d-7ff9d946b7-xqlzn\" (UID: \"e68b78bc-50ac-43ee-8354-a11b921e2713\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a8c22d-7ff9d946b7-xqlzn" Oct 30 16:34:30.415562 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:30.415517 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e68b78bc-50ac-43ee-8354-a11b921e2713-maas-api-tls\") pod \"maas-api-e2e-worker-w1-a8c22d-7ff9d946b7-xqlzn\" (UID: \"e68b78bc-50ac-43ee-8354-a11b921e2713\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a8c22d-7ff9d946b7-xqlzn" Oct 30 16:34:30.418070 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:30.418036 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e68b78bc-50ac-43ee-8354-a11b921e2713-maas-api-tls\") pod \"maas-api-e2e-worker-w1-a8c22d-7ff9d946b7-xqlzn\" (UID: \"e68b78bc-50ac-43ee-8354-a11b921e2713\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a8c22d-7ff9d946b7-xqlzn" Oct 30 16:34:30.560541 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:30.560459 5052 generic.go:356] "Generic (PLEG): container finished" podID="4b8235a9-5724-4354-ad7b-eaef26b3966f" containerID="71f6f0d2b5d722966a3978154591e1372551da04dc4e5d1c33602851e57f1f3c" exitCode=0 Oct 30 16:34:30.560740 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:30.560535 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-46brl" event={"ID":"4b8235a9-5724-4354-ad7b-eaef26b3966f","Type":"ContainerDied","Data":"71f6f0d2b5d722966a3978154591e1372551da04dc4e5d1c33602851e57f1f3c"} Oct 30 16:34:30.560978 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:30.560960 5052 scope.go:122] "RemoveContainer" containerID="71f6f0d2b5d722966a3978154591e1372551da04dc4e5d1c33602851e57f1f3c" Oct 30 16:34:30.562382 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:30.562353 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:34:30.562504 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:30.562438 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:34:30.562504 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:30.562483 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:34:30.696303 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:30.696209 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a8c22d-7ff9d946b7-xqlzn" Oct 30 16:34:31.077401 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:31.077326 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-a8c22d-7ff9d946b7-xqlzn"] Oct 30 16:34:31.080639 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:34:31.080611 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode68b78bc_50ac_43ee_8354_a11b921e2713.slice/crio-a3e426afe8275ced5dc11eeacf8af02324bb2f87fbd4c33696ee6f8413359041 WatchSource:0}: Error finding container a3e426afe8275ced5dc11eeacf8af02324bb2f87fbd4c33696ee6f8413359041: Status 404 returned error can't find the container with id a3e426afe8275ced5dc11eeacf8af02324bb2f87fbd4c33696ee6f8413359041 Oct 30 16:34:31.590287 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:31.590240 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-46brl" event={"ID":"4b8235a9-5724-4354-ad7b-eaef26b3966f","Type":"ContainerStarted","Data":"06eae2719515f348df2fe76d5997e5a8ea46ffb83a0d24cce2f7885dfa88b985"} Oct 30 16:34:31.592171 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:31.592133 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a8c22d-7ff9d946b7-xqlzn" event={"ID":"e68b78bc-50ac-43ee-8354-a11b921e2713","Type":"ContainerStarted","Data":"a3e426afe8275ced5dc11eeacf8af02324bb2f87fbd4c33696ee6f8413359041"} Oct 30 16:34:32.421185 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:32.421149 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-46brl" Oct 30 16:34:32.422152 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:32.422128 5052 patch_prober.go:29] interesting pod/e2e-worker-w3-0b2348-openshift-default-85679bc47c-46brl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Oct 30 16:34:32.422275 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:32.422182 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-46brl" podUID="4b8235a9-5724-4354-ad7b-eaef26b3966f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Oct 30 16:34:32.598872 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:32.598833 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a8c22d-7ff9d946b7-xqlzn" event={"ID":"e68b78bc-50ac-43ee-8354-a11b921e2713","Type":"ContainerStarted","Data":"ae5569953f52787236943cca4f21042f58288bb70da33906cca7983c628910e1"} Oct 30 16:34:32.599074 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:32.599060 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a8c22d-7ff9d946b7-xqlzn" Oct 30 16:34:32.649602 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:32.649021 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a8c22d-7ff9d946b7-xqlzn" podStartSLOduration=3.229048693 podStartE2EDuration="3.649002541s" totalImagesPullingTime="419.953848ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:34:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:34:32.647227286 +0000 UTC m=+828.174562505" watchObservedRunningTime="2026-10-30 16:34:32.649002541 +0000 UTC m=+828.176337761" Oct 30 16:34:33.421478 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:33.421445 5052 patch_prober.go:29] interesting pod/e2e-worker-w3-0b2348-openshift-default-85679bc47c-46brl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Oct 30 16:34:33.422013 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:33.421495 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-46brl" podUID="4b8235a9-5724-4354-ad7b-eaef26b3966f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Oct 30 16:34:34.421733 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:34.421700 5052 patch_prober.go:29] interesting pod/e2e-worker-w3-0b2348-openshift-default-85679bc47c-46brl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Oct 30 16:34:34.422147 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:34.421751 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-46brl" podUID="4b8235a9-5724-4354-ad7b-eaef26b3966f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Oct 30 16:34:34.580131 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:34.580100 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-967fac-7cbc897887-drl9k" Oct 30 16:34:35.338725 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:35.338685 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-46brl" Oct 30 16:34:35.421392 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:35.421349 5052 patch_prober.go:29] interesting pod/e2e-worker-w3-0b2348-openshift-default-85679bc47c-46brl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Oct 30 16:34:35.421572 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:35.421417 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-46brl" podUID="4b8235a9-5724-4354-ad7b-eaef26b3966f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Oct 30 16:34:36.421153 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:36.420743 5052 patch_prober.go:29] interesting pod/e2e-worker-w3-0b2348-openshift-default-85679bc47c-46brl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Oct 30 16:34:36.421153 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:36.420800 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-46brl" podUID="4b8235a9-5724-4354-ad7b-eaef26b3966f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Oct 30 16:34:36.995098 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:36.995052 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cdwcj"] Oct 30 16:34:37.392185 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:37.391749 5052 patch_prober.go:29] interesting pod/e2e-worker-w5-967fac-openshift-default-dffc67487-hff9w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:34:37.392185 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:37.391820 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-hff9w" podUID="559db955-2645-4b32-9070-e87d7bcb485e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:34:37.440913 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:37.440876 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-46brl" Oct 30 16:34:37.442908 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:37.442881 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-46brl" Oct 30 16:34:38.609972 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:38.609945 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a8c22d-7ff9d946b7-xqlzn" Oct 30 16:34:39.642434 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:39.642340 5052 generic.go:356] "Generic (PLEG): container finished" podID="559db955-2645-4b32-9070-e87d7bcb485e" containerID="1d5a748362b69bcf0ccd10cfb10f69389cbb92e9c484daf9086340a58d426295" exitCode=0 Oct 30 16:34:39.642434 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:39.642414 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-hff9w" event={"ID":"559db955-2645-4b32-9070-e87d7bcb485e","Type":"ContainerDied","Data":"1d5a748362b69bcf0ccd10cfb10f69389cbb92e9c484daf9086340a58d426295"} Oct 30 16:34:39.642958 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:39.642798 5052 scope.go:122] "RemoveContainer" containerID="1d5a748362b69bcf0ccd10cfb10f69389cbb92e9c484daf9086340a58d426295" Oct 30 16:34:39.643997 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:39.643969 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:34:39.644108 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:39.644049 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:34:39.644108 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:39.644099 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:34:40.663185 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:40.663148 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-hff9w" event={"ID":"559db955-2645-4b32-9070-e87d7bcb485e","Type":"ContainerStarted","Data":"cf59e7c517f3ab79b6bb2d3c5016ce9ada14cc4a77b33e89e5c0c5271f581a54"} Oct 30 16:34:40.692645 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:40.692607 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-hff9w" Oct 30 16:34:40.692864 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:40.692840 5052 patch_prober.go:29] interesting pod/e2e-worker-w5-967fac-openshift-default-dffc67487-hff9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Oct 30 16:34:40.692921 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:40.692898 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-hff9w" podUID="559db955-2645-4b32-9070-e87d7bcb485e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Oct 30 16:34:41.691819 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:41.691784 5052 patch_prober.go:29] interesting pod/e2e-worker-w5-967fac-openshift-default-dffc67487-hff9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Oct 30 16:34:41.692212 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:41.691830 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-hff9w" podUID="559db955-2645-4b32-9070-e87d7bcb485e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Oct 30 16:34:42.692244 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:42.692212 5052 patch_prober.go:29] interesting pod/e2e-worker-w5-967fac-openshift-default-dffc67487-hff9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Oct 30 16:34:42.692642 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:42.692259 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-hff9w" podUID="559db955-2645-4b32-9070-e87d7bcb485e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Oct 30 16:34:43.692249 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:43.692216 5052 patch_prober.go:29] interesting pod/e2e-worker-w5-967fac-openshift-default-dffc67487-hff9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Oct 30 16:34:43.692694 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:43.692264 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-hff9w" podUID="559db955-2645-4b32-9070-e87d7bcb485e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Oct 30 16:34:44.515434 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:44.515399 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cdwcj"] Oct 30 16:34:44.691896 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:44.691861 5052 patch_prober.go:29] interesting pod/e2e-worker-w5-967fac-openshift-default-dffc67487-hff9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Oct 30 16:34:44.692061 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:44.691921 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-hff9w" podUID="559db955-2645-4b32-9070-e87d7bcb485e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Oct 30 16:34:45.999049 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:45.999013 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-hff9w" Oct 30 16:34:45.999608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:45.999558 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-hff9w" Oct 30 16:34:46.000360 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:46.000335 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-hff9w" Oct 30 16:34:48.840957 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:34:48.840919 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 16:34:56.330639 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:34:56.330598 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cdwcj"] Oct 30 16:35:03.736859 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:03.736810 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-f7blz"] Oct 30 16:35:03.743854 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:03.743828 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-f7blz" Oct 30 16:35:03.759079 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:03.759054 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-263f00-openshift-default-dockercfg-nr2tc\"" Oct 30 16:35:03.774601 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:03.774547 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-f7blz"] Oct 30 16:35:03.816927 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:03.816897 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1-istio-data\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-f7blz\" (UID: \"1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-f7blz" Oct 30 16:35:03.817065 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:03.816962 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1-istiod-ca-cert\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-f7blz\" (UID: \"1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-f7blz" Oct 30 16:35:03.817065 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:03.817002 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1-istio-podinfo\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-f7blz\" (UID: \"1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-f7blz" Oct 30 16:35:03.817149 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:03.817078 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1-credential-socket\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-f7blz\" (UID: \"1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-f7blz" Oct 30 16:35:03.817149 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:03.817116 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1-workload-socket\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-f7blz\" (UID: \"1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-f7blz" Oct 30 16:35:03.817241 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:03.817150 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1-istio-envoy\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-f7blz\" (UID: \"1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-f7blz" Oct 30 16:35:03.817241 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:03.817208 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1-workload-certs\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-f7blz\" (UID: \"1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-f7blz" Oct 30 16:35:03.817349 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:03.817303 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1-istio-token\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-f7blz\" (UID: \"1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-f7blz" Oct 30 16:35:03.817405 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:03.817345 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m54j8\" (UniqueName: \"kubernetes.io/projected/1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1-kube-api-access-m54j8\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-f7blz\" (UID: \"1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-f7blz" Oct 30 16:35:03.918353 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:03.918326 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1-istio-data\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-f7blz\" (UID: \"1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-f7blz" Oct 30 16:35:03.918505 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:03.918368 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1-istiod-ca-cert\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-f7blz\" (UID: \"1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-f7blz" Oct 30 16:35:03.918505 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:03.918389 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1-istio-podinfo\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-f7blz\" (UID: \"1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-f7blz" Oct 30 16:35:03.918505 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:03.918412 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1-credential-socket\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-f7blz\" (UID: \"1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-f7blz" Oct 30 16:35:03.918505 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:03.918436 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1-workload-socket\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-f7blz\" (UID: \"1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-f7blz" Oct 30 16:35:03.918505 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:03.918452 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1-istio-envoy\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-f7blz\" (UID: \"1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-f7blz" Oct 30 16:35:03.918505 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:03.918474 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1-workload-certs\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-f7blz\" (UID: \"1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-f7blz" Oct 30 16:35:03.918505 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:03.918499 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1-istio-token\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-f7blz\" (UID: \"1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-f7blz" Oct 30 16:35:03.918922 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:03.918518 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m54j8\" (UniqueName: \"kubernetes.io/projected/1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1-kube-api-access-m54j8\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-f7blz\" (UID: \"1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-f7blz" Oct 30 16:35:03.918983 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:03.918915 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1-istio-data\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-f7blz\" (UID: \"1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-f7blz" Oct 30 16:35:03.918983 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:03.918936 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1-workload-certs\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-f7blz\" (UID: \"1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-f7blz" Oct 30 16:35:03.919055 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:03.918984 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1-credential-socket\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-f7blz\" (UID: \"1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-f7blz" Oct 30 16:35:03.919191 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:03.919162 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1-istiod-ca-cert\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-f7blz\" (UID: \"1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-f7blz" Oct 30 16:35:03.919191 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:03.919178 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1-workload-socket\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-f7blz\" (UID: \"1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-f7blz" Oct 30 16:35:03.920907 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:03.920884 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1-istio-envoy\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-f7blz\" (UID: \"1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-f7blz" Oct 30 16:35:03.921148 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:03.921131 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1-istio-podinfo\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-f7blz\" (UID: \"1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-f7blz" Oct 30 16:35:03.944786 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:03.944757 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m54j8\" (UniqueName: \"kubernetes.io/projected/1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1-kube-api-access-m54j8\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-f7blz\" (UID: \"1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-f7blz" Oct 30 16:35:03.944870 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:03.944759 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1-istio-token\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-f7blz\" (UID: \"1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-f7blz" Oct 30 16:35:04.057201 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:04.057126 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-f7blz" Oct 30 16:35:04.375664 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:04.375459 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-f7blz"] Oct 30 16:35:04.378546 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:35:04.378521 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a6bcabe_663d_46c8_8c70_9b7ab0bd5db1.slice/crio-e4aea41d7b4555220ff6a2eb543f644dc67a5504e4e2f88d8ee48ce83615b16d WatchSource:0}: Error finding container e4aea41d7b4555220ff6a2eb543f644dc67a5504e4e2f88d8ee48ce83615b16d: Status 404 returned error can't find the container with id e4aea41d7b4555220ff6a2eb543f644dc67a5504e4e2f88d8ee48ce83615b16d Oct 30 16:35:04.380784 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:04.380756 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:35:04.380845 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:04.380817 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:35:04.380883 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:04.380844 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:35:04.798787 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:04.798731 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-f7blz" event={"ID":"1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1","Type":"ContainerStarted","Data":"2d72c42e5bc4e3f1dab3f51c29d6ffad1ad86844ed26aeace4fd738a49c2bfd7"} Oct 30 16:35:04.799209 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:04.798799 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-f7blz" event={"ID":"1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1","Type":"ContainerStarted","Data":"e4aea41d7b4555220ff6a2eb543f644dc67a5504e4e2f88d8ee48ce83615b16d"} Oct 30 16:35:04.868905 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:04.868851 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-f7blz" podStartSLOduration=1.868835625 podStartE2EDuration="1.868835625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:35:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:35:04.856817193 +0000 UTC m=+860.384152412" watchObservedRunningTime="2026-10-30 16:35:04.868835625 +0000 UTC m=+860.396170843" Oct 30 16:35:05.058091 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:05.057983 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-f7blz" Oct 30 16:35:05.059226 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:05.059198 5052 patch_prober.go:29] interesting pod/e2e-worker-w4-263f00-openshift-default-b969d55bc-f7blz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Oct 30 16:35:05.059367 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:05.059270 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-f7blz" podUID="1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Oct 30 16:35:06.057692 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:06.057656 5052 patch_prober.go:29] interesting pod/e2e-worker-w4-263f00-openshift-default-b969d55bc-f7blz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Oct 30 16:35:06.058265 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:06.058233 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-f7blz" podUID="1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Oct 30 16:35:07.058156 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:07.058126 5052 patch_prober.go:29] interesting pod/e2e-worker-w4-263f00-openshift-default-b969d55bc-f7blz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Oct 30 16:35:07.058643 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:07.058612 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-f7blz" podUID="1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Oct 30 16:35:08.058082 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:08.058043 5052 patch_prober.go:29] interesting pod/e2e-worker-w4-263f00-openshift-default-b969d55bc-f7blz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Oct 30 16:35:08.058319 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:08.058290 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-f7blz" podUID="1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Oct 30 16:35:09.058196 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:09.058160 5052 patch_prober.go:29] interesting pod/e2e-worker-w4-263f00-openshift-default-b969d55bc-f7blz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Oct 30 16:35:09.058385 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:09.058216 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-f7blz" podUID="1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Oct 30 16:35:10.058438 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:10.058398 5052 patch_prober.go:29] interesting pod/e2e-worker-w4-263f00-openshift-default-b969d55bc-f7blz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Oct 30 16:35:10.058936 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:10.058469 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-f7blz" podUID="1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Oct 30 16:35:11.323695 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:11.323666 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-f7blz" Oct 30 16:35:11.324231 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:11.324177 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-f7blz" Oct 30 16:35:11.325102 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:11.325084 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-f7blz" Oct 30 16:35:14.827205 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:14.827172 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cdwcj"] Oct 30 16:35:18.849799 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:35:18.849755 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 16:35:34.037783 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:34.037744 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cdwcj"] Oct 30 16:35:44.689335 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:44.689303 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4wrf8_4f0f46f4-5cda-4d67-b6f3-47df06eb844c/console-operator/2.log" Oct 30 16:35:44.690187 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:44.690085 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4wrf8_4f0f46f4-5cda-4d67-b6f3-47df06eb844c/console-operator/2.log" Oct 30 16:35:48.132492 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:48.132454 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-dnwt8"] Oct 30 16:35:48.136487 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:48.136469 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-dnwt8" Oct 30 16:35:48.151121 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:48.151071 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-750dcb-openshift-default-dockercfg-fbrrg\"" Oct 30 16:35:48.165020 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:48.164998 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-dnwt8"] Oct 30 16:35:48.261084 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:48.261053 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2h7wj\" (UniqueName: \"kubernetes.io/projected/c8947ffd-7539-43d8-86d5-fc17dd8c7d6e-kube-api-access-2h7wj\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-dnwt8\" (UID: \"c8947ffd-7539-43d8-86d5-fc17dd8c7d6e\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-dnwt8" Oct 30 16:35:48.261222 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:48.261098 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8947ffd-7539-43d8-86d5-fc17dd8c7d6e-workload-socket\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-dnwt8\" (UID: \"c8947ffd-7539-43d8-86d5-fc17dd8c7d6e\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-dnwt8" Oct 30 16:35:48.261222 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:48.261162 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8947ffd-7539-43d8-86d5-fc17dd8c7d6e-istio-envoy\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-dnwt8\" (UID: \"c8947ffd-7539-43d8-86d5-fc17dd8c7d6e\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-dnwt8" Oct 30 16:35:48.261222 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:48.261185 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8947ffd-7539-43d8-86d5-fc17dd8c7d6e-istiod-ca-cert\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-dnwt8\" (UID: \"c8947ffd-7539-43d8-86d5-fc17dd8c7d6e\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-dnwt8" Oct 30 16:35:48.261326 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:48.261301 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8947ffd-7539-43d8-86d5-fc17dd8c7d6e-istio-data\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-dnwt8\" (UID: \"c8947ffd-7539-43d8-86d5-fc17dd8c7d6e\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-dnwt8" Oct 30 16:35:48.261379 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:48.261363 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8947ffd-7539-43d8-86d5-fc17dd8c7d6e-istio-token\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-dnwt8\" (UID: \"c8947ffd-7539-43d8-86d5-fc17dd8c7d6e\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-dnwt8" Oct 30 16:35:48.261414 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:48.261390 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8947ffd-7539-43d8-86d5-fc17dd8c7d6e-workload-certs\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-dnwt8\" (UID: \"c8947ffd-7539-43d8-86d5-fc17dd8c7d6e\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-dnwt8" Oct 30 16:35:48.261451 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:48.261419 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8947ffd-7539-43d8-86d5-fc17dd8c7d6e-credential-socket\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-dnwt8\" (UID: \"c8947ffd-7539-43d8-86d5-fc17dd8c7d6e\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-dnwt8" Oct 30 16:35:48.261558 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:48.261544 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8947ffd-7539-43d8-86d5-fc17dd8c7d6e-istio-podinfo\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-dnwt8\" (UID: \"c8947ffd-7539-43d8-86d5-fc17dd8c7d6e\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-dnwt8" Oct 30 16:35:48.276806 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:48.276785 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-7lh24"] Oct 30 16:35:48.284410 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:48.284389 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-7lh24" Oct 30 16:35:48.302149 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:48.302122 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-3e2f96-openshift-default-dockercfg-v5cl2\"" Oct 30 16:35:48.305684 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:48.305659 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-7lh24"] Oct 30 16:35:48.362791 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:48.362768 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e461432-f429-4153-9e32-954f2ca3fb61-workload-socket\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-7lh24\" (UID: \"0e461432-f429-4153-9e32-954f2ca3fb61\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-7lh24" Oct 30 16:35:48.362926 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:48.362810 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e461432-f429-4153-9e32-954f2ca3fb61-istio-podinfo\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-7lh24\" (UID: \"0e461432-f429-4153-9e32-954f2ca3fb61\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-7lh24" Oct 30 16:35:48.362926 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:48.362893 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2h7wj\" (UniqueName: \"kubernetes.io/projected/c8947ffd-7539-43d8-86d5-fc17dd8c7d6e-kube-api-access-2h7wj\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-dnwt8\" (UID: \"c8947ffd-7539-43d8-86d5-fc17dd8c7d6e\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-dnwt8" Oct 30 16:35:48.363004 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:48.362970 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e461432-f429-4153-9e32-954f2ca3fb61-istio-data\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-7lh24\" (UID: \"0e461432-f429-4153-9e32-954f2ca3fb61\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-7lh24" Oct 30 16:35:48.363004 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:48.362995 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e461432-f429-4153-9e32-954f2ca3fb61-workload-certs\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-7lh24\" (UID: \"0e461432-f429-4153-9e32-954f2ca3fb61\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-7lh24" Oct 30 16:35:48.363071 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:48.363010 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e461432-f429-4153-9e32-954f2ca3fb61-istiod-ca-cert\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-7lh24\" (UID: \"0e461432-f429-4153-9e32-954f2ca3fb61\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-7lh24" Oct 30 16:35:48.363071 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:48.363042 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8947ffd-7539-43d8-86d5-fc17dd8c7d6e-workload-socket\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-dnwt8\" (UID: \"c8947ffd-7539-43d8-86d5-fc17dd8c7d6e\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-dnwt8" Oct 30 16:35:48.363130 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:48.363087 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8947ffd-7539-43d8-86d5-fc17dd8c7d6e-istio-envoy\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-dnwt8\" (UID: \"c8947ffd-7539-43d8-86d5-fc17dd8c7d6e\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-dnwt8" Oct 30 16:35:48.363185 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:48.363169 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8947ffd-7539-43d8-86d5-fc17dd8c7d6e-istiod-ca-cert\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-dnwt8\" (UID: \"c8947ffd-7539-43d8-86d5-fc17dd8c7d6e\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-dnwt8" Oct 30 16:35:48.363318 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:48.363301 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8947ffd-7539-43d8-86d5-fc17dd8c7d6e-istio-data\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-dnwt8\" (UID: \"c8947ffd-7539-43d8-86d5-fc17dd8c7d6e\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-dnwt8" Oct 30 16:35:48.363357 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:48.363330 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e461432-f429-4153-9e32-954f2ca3fb61-istio-envoy\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-7lh24\" (UID: \"0e461432-f429-4153-9e32-954f2ca3fb61\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-7lh24" Oct 30 16:35:48.363404 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:48.363357 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8947ffd-7539-43d8-86d5-fc17dd8c7d6e-istio-token\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-dnwt8\" (UID: \"c8947ffd-7539-43d8-86d5-fc17dd8c7d6e\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-dnwt8" Oct 30 16:35:48.363404 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:48.363382 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kd7ww\" (UniqueName: \"kubernetes.io/projected/0e461432-f429-4153-9e32-954f2ca3fb61-kube-api-access-kd7ww\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-7lh24\" (UID: \"0e461432-f429-4153-9e32-954f2ca3fb61\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-7lh24" Oct 30 16:35:48.363512 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:48.363431 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8947ffd-7539-43d8-86d5-fc17dd8c7d6e-workload-socket\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-dnwt8\" (UID: \"c8947ffd-7539-43d8-86d5-fc17dd8c7d6e\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-dnwt8" Oct 30 16:35:48.363512 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:48.363481 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8947ffd-7539-43d8-86d5-fc17dd8c7d6e-workload-certs\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-dnwt8\" (UID: \"c8947ffd-7539-43d8-86d5-fc17dd8c7d6e\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-dnwt8" Oct 30 16:35:48.363636 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:48.363518 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e461432-f429-4153-9e32-954f2ca3fb61-credential-socket\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-7lh24\" (UID: \"0e461432-f429-4153-9e32-954f2ca3fb61\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-7lh24" Oct 30 16:35:48.363636 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:48.363557 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8947ffd-7539-43d8-86d5-fc17dd8c7d6e-credential-socket\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-dnwt8\" (UID: \"c8947ffd-7539-43d8-86d5-fc17dd8c7d6e\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-dnwt8" Oct 30 16:35:48.363759 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:48.363635 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e461432-f429-4153-9e32-954f2ca3fb61-istio-token\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-7lh24\" (UID: \"0e461432-f429-4153-9e32-954f2ca3fb61\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-7lh24" Oct 30 16:35:48.363759 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:48.363639 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8947ffd-7539-43d8-86d5-fc17dd8c7d6e-istio-data\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-dnwt8\" (UID: \"c8947ffd-7539-43d8-86d5-fc17dd8c7d6e\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-dnwt8" Oct 30 16:35:48.363879 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:48.363780 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8947ffd-7539-43d8-86d5-fc17dd8c7d6e-istio-podinfo\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-dnwt8\" (UID: \"c8947ffd-7539-43d8-86d5-fc17dd8c7d6e\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-dnwt8" Oct 30 16:35:48.363879 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:48.363783 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8947ffd-7539-43d8-86d5-fc17dd8c7d6e-workload-certs\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-dnwt8\" (UID: \"c8947ffd-7539-43d8-86d5-fc17dd8c7d6e\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-dnwt8" Oct 30 16:35:48.363986 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:48.363876 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8947ffd-7539-43d8-86d5-fc17dd8c7d6e-istiod-ca-cert\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-dnwt8\" (UID: \"c8947ffd-7539-43d8-86d5-fc17dd8c7d6e\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-dnwt8" Oct 30 16:35:48.364037 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:48.363983 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8947ffd-7539-43d8-86d5-fc17dd8c7d6e-credential-socket\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-dnwt8\" (UID: \"c8947ffd-7539-43d8-86d5-fc17dd8c7d6e\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-dnwt8" Oct 30 16:35:48.365482 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:48.365459 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8947ffd-7539-43d8-86d5-fc17dd8c7d6e-istio-envoy\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-dnwt8\" (UID: \"c8947ffd-7539-43d8-86d5-fc17dd8c7d6e\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-dnwt8" Oct 30 16:35:48.366086 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:48.366062 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8947ffd-7539-43d8-86d5-fc17dd8c7d6e-istio-podinfo\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-dnwt8\" (UID: \"c8947ffd-7539-43d8-86d5-fc17dd8c7d6e\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-dnwt8" Oct 30 16:35:48.385948 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:48.385886 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8947ffd-7539-43d8-86d5-fc17dd8c7d6e-istio-token\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-dnwt8\" (UID: \"c8947ffd-7539-43d8-86d5-fc17dd8c7d6e\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-dnwt8" Oct 30 16:35:48.385948 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:48.385886 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h7wj\" (UniqueName: \"kubernetes.io/projected/c8947ffd-7539-43d8-86d5-fc17dd8c7d6e-kube-api-access-2h7wj\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-dnwt8\" (UID: \"c8947ffd-7539-43d8-86d5-fc17dd8c7d6e\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-dnwt8" Oct 30 16:35:48.450441 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:48.450402 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-dnwt8" Oct 30 16:35:48.465029 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:48.464996 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e461432-f429-4153-9e32-954f2ca3fb61-workload-socket\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-7lh24\" (UID: \"0e461432-f429-4153-9e32-954f2ca3fb61\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-7lh24" Oct 30 16:35:48.465174 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:48.465038 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e461432-f429-4153-9e32-954f2ca3fb61-istio-podinfo\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-7lh24\" (UID: \"0e461432-f429-4153-9e32-954f2ca3fb61\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-7lh24" Oct 30 16:35:48.465174 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:48.465074 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e461432-f429-4153-9e32-954f2ca3fb61-istio-data\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-7lh24\" (UID: \"0e461432-f429-4153-9e32-954f2ca3fb61\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-7lh24" Oct 30 16:35:48.465174 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:48.465103 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e461432-f429-4153-9e32-954f2ca3fb61-workload-certs\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-7lh24\" (UID: \"0e461432-f429-4153-9e32-954f2ca3fb61\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-7lh24" Oct 30 16:35:48.465174 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:48.465131 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e461432-f429-4153-9e32-954f2ca3fb61-istiod-ca-cert\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-7lh24\" (UID: \"0e461432-f429-4153-9e32-954f2ca3fb61\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-7lh24" Oct 30 16:35:48.465422 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:48.465233 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e461432-f429-4153-9e32-954f2ca3fb61-istio-envoy\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-7lh24\" (UID: \"0e461432-f429-4153-9e32-954f2ca3fb61\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-7lh24" Oct 30 16:35:48.465422 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:48.465270 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kd7ww\" (UniqueName: \"kubernetes.io/projected/0e461432-f429-4153-9e32-954f2ca3fb61-kube-api-access-kd7ww\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-7lh24\" (UID: \"0e461432-f429-4153-9e32-954f2ca3fb61\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-7lh24" Oct 30 16:35:48.465422 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:48.465332 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e461432-f429-4153-9e32-954f2ca3fb61-credential-socket\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-7lh24\" (UID: \"0e461432-f429-4153-9e32-954f2ca3fb61\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-7lh24" Oct 30 16:35:48.465422 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:48.465408 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e461432-f429-4153-9e32-954f2ca3fb61-istio-token\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-7lh24\" (UID: \"0e461432-f429-4153-9e32-954f2ca3fb61\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-7lh24" Oct 30 16:35:48.465648 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:48.465427 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e461432-f429-4153-9e32-954f2ca3fb61-workload-socket\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-7lh24\" (UID: \"0e461432-f429-4153-9e32-954f2ca3fb61\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-7lh24" Oct 30 16:35:48.465648 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:48.465474 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e461432-f429-4153-9e32-954f2ca3fb61-workload-certs\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-7lh24\" (UID: \"0e461432-f429-4153-9e32-954f2ca3fb61\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-7lh24" Oct 30 16:35:48.465648 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:48.465503 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e461432-f429-4153-9e32-954f2ca3fb61-istio-data\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-7lh24\" (UID: \"0e461432-f429-4153-9e32-954f2ca3fb61\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-7lh24" Oct 30 16:35:48.465841 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:48.465818 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e461432-f429-4153-9e32-954f2ca3fb61-credential-socket\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-7lh24\" (UID: \"0e461432-f429-4153-9e32-954f2ca3fb61\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-7lh24" Oct 30 16:35:48.466201 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:48.466177 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e461432-f429-4153-9e32-954f2ca3fb61-istiod-ca-cert\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-7lh24\" (UID: \"0e461432-f429-4153-9e32-954f2ca3fb61\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-7lh24" Oct 30 16:35:48.467676 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:48.467656 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e461432-f429-4153-9e32-954f2ca3fb61-istio-envoy\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-7lh24\" (UID: \"0e461432-f429-4153-9e32-954f2ca3fb61\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-7lh24" Oct 30 16:35:48.468178 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:48.468161 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e461432-f429-4153-9e32-954f2ca3fb61-istio-podinfo\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-7lh24\" (UID: \"0e461432-f429-4153-9e32-954f2ca3fb61\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-7lh24" Oct 30 16:35:48.490730 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:48.490703 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e461432-f429-4153-9e32-954f2ca3fb61-istio-token\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-7lh24\" (UID: \"0e461432-f429-4153-9e32-954f2ca3fb61\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-7lh24" Oct 30 16:35:48.490843 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:48.490765 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kd7ww\" (UniqueName: \"kubernetes.io/projected/0e461432-f429-4153-9e32-954f2ca3fb61-kube-api-access-kd7ww\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-7lh24\" (UID: \"0e461432-f429-4153-9e32-954f2ca3fb61\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-7lh24" Oct 30 16:35:48.606188 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:48.606146 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-7lh24" Oct 30 16:35:48.770442 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:48.770338 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-dnwt8"] Oct 30 16:35:48.772859 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:35:48.772830 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8947ffd_7539_43d8_86d5_fc17dd8c7d6e.slice/crio-2ce1471145b6e14b0aaa1141f3bc321551749b6d30bf4ebc1f823a929806562f WatchSource:0}: Error finding container 2ce1471145b6e14b0aaa1141f3bc321551749b6d30bf4ebc1f823a929806562f: Status 404 returned error can't find the container with id 2ce1471145b6e14b0aaa1141f3bc321551749b6d30bf4ebc1f823a929806562f Oct 30 16:35:48.775376 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:48.775343 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:35:48.775476 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:48.775425 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:35:48.775476 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:48.775454 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:35:48.882201 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:35:48.882174 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 16:35:48.914475 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:48.914396 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-7lh24"] Oct 30 16:35:48.918415 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:35:48.918381 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e461432_f429_4153_9e32_954f2ca3fb61.slice/crio-6180db34cc71ab459ecae8298f7fb469f7ac64f43a603e8ea9fb81f19075e6a9 WatchSource:0}: Error finding container 6180db34cc71ab459ecae8298f7fb469f7ac64f43a603e8ea9fb81f19075e6a9: Status 404 returned error can't find the container with id 6180db34cc71ab459ecae8298f7fb469f7ac64f43a603e8ea9fb81f19075e6a9 Oct 30 16:35:48.922268 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:48.922235 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:35:48.922357 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:48.922314 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:35:48.922397 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:48.922353 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:35:49.154772 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:49.154513 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-7lh24" event={"ID":"0e461432-f429-4153-9e32-954f2ca3fb61","Type":"ContainerStarted","Data":"1676a9ec909fdd11894cdf18ab1500259cbbb7eb1394f4717fe5cd66cb86f5d6"} Oct 30 16:35:49.154772 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:49.154575 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-7lh24" event={"ID":"0e461432-f429-4153-9e32-954f2ca3fb61","Type":"ContainerStarted","Data":"6180db34cc71ab459ecae8298f7fb469f7ac64f43a603e8ea9fb81f19075e6a9"} Oct 30 16:35:49.156354 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:49.156324 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-dnwt8" event={"ID":"c8947ffd-7539-43d8-86d5-fc17dd8c7d6e","Type":"ContainerStarted","Data":"061347d21117ffeb6f479ab871723e42258058119396dfb814a824800ae970ef"} Oct 30 16:35:49.156497 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:49.156376 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-dnwt8" event={"ID":"c8947ffd-7539-43d8-86d5-fc17dd8c7d6e","Type":"ContainerStarted","Data":"2ce1471145b6e14b0aaa1141f3bc321551749b6d30bf4ebc1f823a929806562f"} Oct 30 16:35:49.200732 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:49.200634 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-7lh24" podStartSLOduration=1.200618898 podStartE2EDuration="1.200618898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:35:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:35:49.199237053 +0000 UTC m=+904.726572273" watchObservedRunningTime="2026-10-30 16:35:49.200618898 +0000 UTC m=+904.727954116" Oct 30 16:35:49.244433 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:49.244364 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-dnwt8" podStartSLOduration=1.244347337 podStartE2EDuration="1.244347337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:35:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:35:49.243283498 +0000 UTC m=+904.770618718" watchObservedRunningTime="2026-10-30 16:35:49.244347337 +0000 UTC m=+904.771682555" Oct 30 16:35:49.454658 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:49.453965 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-dnwt8" Oct 30 16:35:49.461661 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:49.459727 5052 patch_prober.go:29] interesting pod/e2e-worker-w2-750dcb-openshift-default-777f567b9f-dnwt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 30 16:35:49.461661 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:49.459792 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-dnwt8" podUID="c8947ffd-7539-43d8-86d5-fc17dd8c7d6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 30 16:35:49.607029 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:49.606936 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-7lh24" Oct 30 16:35:49.609818 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:49.608685 5052 patch_prober.go:29] interesting pod/e2e-worker-w0-3e2f96-openshift-default-84865466dc-7lh24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Oct 30 16:35:49.609818 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:49.608749 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-7lh24" podUID="0e461432-f429-4153-9e32-954f2ca3fb61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Oct 30 16:35:50.450989 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:50.450950 5052 patch_prober.go:29] interesting pod/e2e-worker-w2-750dcb-openshift-default-777f567b9f-dnwt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 30 16:35:50.451623 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:50.451573 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-dnwt8" podUID="c8947ffd-7539-43d8-86d5-fc17dd8c7d6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 30 16:35:50.610439 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:50.610388 5052 patch_prober.go:29] interesting pod/e2e-worker-w0-3e2f96-openshift-default-84865466dc-7lh24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Oct 30 16:35:50.610703 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:50.610457 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-7lh24" podUID="0e461432-f429-4153-9e32-954f2ca3fb61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Oct 30 16:35:51.451605 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:51.450969 5052 patch_prober.go:29] interesting pod/e2e-worker-w2-750dcb-openshift-default-777f567b9f-dnwt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 30 16:35:51.451605 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:51.451039 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-dnwt8" podUID="c8947ffd-7539-43d8-86d5-fc17dd8c7d6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 30 16:35:51.607561 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:51.607525 5052 patch_prober.go:29] interesting pod/e2e-worker-w0-3e2f96-openshift-default-84865466dc-7lh24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Oct 30 16:35:51.607739 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:51.607645 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-7lh24" podUID="0e461432-f429-4153-9e32-954f2ca3fb61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Oct 30 16:35:52.451152 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:52.451103 5052 patch_prober.go:29] interesting pod/e2e-worker-w2-750dcb-openshift-default-777f567b9f-dnwt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 30 16:35:52.451768 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:52.451721 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-dnwt8" podUID="c8947ffd-7539-43d8-86d5-fc17dd8c7d6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 30 16:35:52.607822 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:52.607788 5052 patch_prober.go:29] interesting pod/e2e-worker-w0-3e2f96-openshift-default-84865466dc-7lh24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Oct 30 16:35:52.607981 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:52.607837 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-7lh24" podUID="0e461432-f429-4153-9e32-954f2ca3fb61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Oct 30 16:35:53.455607 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:53.455515 5052 patch_prober.go:29] interesting pod/e2e-worker-w2-750dcb-openshift-default-777f567b9f-dnwt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 30 16:35:53.455607 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:53.455596 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-dnwt8" podUID="c8947ffd-7539-43d8-86d5-fc17dd8c7d6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 30 16:35:53.607564 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:53.607522 5052 patch_prober.go:29] interesting pod/e2e-worker-w0-3e2f96-openshift-default-84865466dc-7lh24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Oct 30 16:35:53.607747 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:53.607606 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-7lh24" podUID="0e461432-f429-4153-9e32-954f2ca3fb61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Oct 30 16:35:54.004677 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:54.004644 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cdwcj"] Oct 30 16:35:54.451606 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:54.451171 5052 patch_prober.go:29] interesting pod/e2e-worker-w2-750dcb-openshift-default-777f567b9f-dnwt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 30 16:35:54.451606 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:54.451251 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-dnwt8" podUID="c8947ffd-7539-43d8-86d5-fc17dd8c7d6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 30 16:35:54.606988 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:54.606896 5052 patch_prober.go:29] interesting pod/e2e-worker-w0-3e2f96-openshift-default-84865466dc-7lh24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Oct 30 16:35:54.606988 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:54.606974 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-7lh24" podUID="0e461432-f429-4153-9e32-954f2ca3fb61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Oct 30 16:35:55.451017 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:55.450980 5052 patch_prober.go:29] interesting pod/e2e-worker-w2-750dcb-openshift-default-777f567b9f-dnwt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 30 16:35:55.451330 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:55.451298 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-dnwt8" podUID="c8947ffd-7539-43d8-86d5-fc17dd8c7d6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 30 16:35:55.611001 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:55.610961 5052 patch_prober.go:29] interesting pod/e2e-worker-w0-3e2f96-openshift-default-84865466dc-7lh24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Oct 30 16:35:55.611624 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:55.611573 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-7lh24" podUID="0e461432-f429-4153-9e32-954f2ca3fb61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Oct 30 16:35:57.175326 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:57.175289 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-7lh24" Oct 30 16:35:57.175935 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:57.175908 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-7lh24" Oct 30 16:35:57.176759 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:57.176731 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-7lh24" Oct 30 16:35:57.352784 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:57.352756 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-dnwt8" Oct 30 16:35:57.354687 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:57.354660 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-dnwt8" Oct 30 16:35:57.357866 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:35:57.357847 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-dnwt8" Oct 30 16:36:03.607130 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:03.607077 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-98gfq"] Oct 30 16:36:03.612728 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:03.612707 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-98gfq" Oct 30 16:36:03.627424 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:03.627404 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-d693d0-openshift-default-dockercfg-qggxm\"" Oct 30 16:36:03.642252 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:03.642225 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-98gfq"] Oct 30 16:36:03.749099 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:03.749063 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c22edc73-463a-4439-92aa-601b4cddd714-istio-token\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-98gfq\" (UID: \"c22edc73-463a-4439-92aa-601b4cddd714\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-98gfq" Oct 30 16:36:03.749265 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:03.749109 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c22edc73-463a-4439-92aa-601b4cddd714-istiod-ca-cert\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-98gfq\" (UID: \"c22edc73-463a-4439-92aa-601b4cddd714\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-98gfq" Oct 30 16:36:03.749265 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:03.749132 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c22edc73-463a-4439-92aa-601b4cddd714-istio-data\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-98gfq\" (UID: \"c22edc73-463a-4439-92aa-601b4cddd714\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-98gfq" Oct 30 16:36:03.749391 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:03.749267 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c22edc73-463a-4439-92aa-601b4cddd714-istio-envoy\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-98gfq\" (UID: \"c22edc73-463a-4439-92aa-601b4cddd714\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-98gfq" Oct 30 16:36:03.749391 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:03.749352 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c22edc73-463a-4439-92aa-601b4cddd714-workload-socket\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-98gfq\" (UID: \"c22edc73-463a-4439-92aa-601b4cddd714\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-98gfq" Oct 30 16:36:03.749484 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:03.749464 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c22edc73-463a-4439-92aa-601b4cddd714-credential-socket\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-98gfq\" (UID: \"c22edc73-463a-4439-92aa-601b4cddd714\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-98gfq" Oct 30 16:36:03.749635 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:03.749611 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c22edc73-463a-4439-92aa-601b4cddd714-workload-certs\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-98gfq\" (UID: \"c22edc73-463a-4439-92aa-601b4cddd714\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-98gfq" Oct 30 16:36:03.749736 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:03.749718 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c22edc73-463a-4439-92aa-601b4cddd714-istio-podinfo\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-98gfq\" (UID: \"c22edc73-463a-4439-92aa-601b4cddd714\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-98gfq" Oct 30 16:36:03.749868 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:03.749851 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrkxn\" (UniqueName: \"kubernetes.io/projected/c22edc73-463a-4439-92aa-601b4cddd714-kube-api-access-nrkxn\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-98gfq\" (UID: \"c22edc73-463a-4439-92aa-601b4cddd714\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-98gfq" Oct 30 16:36:03.850849 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:03.850819 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c22edc73-463a-4439-92aa-601b4cddd714-workload-certs\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-98gfq\" (UID: \"c22edc73-463a-4439-92aa-601b4cddd714\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-98gfq" Oct 30 16:36:03.851001 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:03.850868 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c22edc73-463a-4439-92aa-601b4cddd714-istio-podinfo\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-98gfq\" (UID: \"c22edc73-463a-4439-92aa-601b4cddd714\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-98gfq" Oct 30 16:36:03.851001 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:03.850910 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrkxn\" (UniqueName: \"kubernetes.io/projected/c22edc73-463a-4439-92aa-601b4cddd714-kube-api-access-nrkxn\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-98gfq\" (UID: \"c22edc73-463a-4439-92aa-601b4cddd714\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-98gfq" Oct 30 16:36:03.851001 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:03.850962 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c22edc73-463a-4439-92aa-601b4cddd714-istio-token\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-98gfq\" (UID: \"c22edc73-463a-4439-92aa-601b4cddd714\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-98gfq" Oct 30 16:36:03.851001 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:03.850997 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c22edc73-463a-4439-92aa-601b4cddd714-istiod-ca-cert\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-98gfq\" (UID: \"c22edc73-463a-4439-92aa-601b4cddd714\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-98gfq" Oct 30 16:36:03.851234 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:03.851026 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c22edc73-463a-4439-92aa-601b4cddd714-istio-data\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-98gfq\" (UID: \"c22edc73-463a-4439-92aa-601b4cddd714\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-98gfq" Oct 30 16:36:03.851234 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:03.851060 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c22edc73-463a-4439-92aa-601b4cddd714-istio-envoy\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-98gfq\" (UID: \"c22edc73-463a-4439-92aa-601b4cddd714\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-98gfq" Oct 30 16:36:03.851234 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:03.851118 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c22edc73-463a-4439-92aa-601b4cddd714-workload-socket\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-98gfq\" (UID: \"c22edc73-463a-4439-92aa-601b4cddd714\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-98gfq" Oct 30 16:36:03.851234 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:03.851160 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c22edc73-463a-4439-92aa-601b4cddd714-credential-socket\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-98gfq\" (UID: \"c22edc73-463a-4439-92aa-601b4cddd714\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-98gfq" Oct 30 16:36:03.851442 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:03.851310 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c22edc73-463a-4439-92aa-601b4cddd714-workload-certs\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-98gfq\" (UID: \"c22edc73-463a-4439-92aa-601b4cddd714\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-98gfq" Oct 30 16:36:03.851512 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:03.851493 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c22edc73-463a-4439-92aa-601b4cddd714-credential-socket\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-98gfq\" (UID: \"c22edc73-463a-4439-92aa-601b4cddd714\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-98gfq" Oct 30 16:36:03.851597 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:03.851566 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c22edc73-463a-4439-92aa-601b4cddd714-workload-socket\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-98gfq\" (UID: \"c22edc73-463a-4439-92aa-601b4cddd714\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-98gfq" Oct 30 16:36:03.851910 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:03.851890 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c22edc73-463a-4439-92aa-601b4cddd714-istiod-ca-cert\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-98gfq\" (UID: \"c22edc73-463a-4439-92aa-601b4cddd714\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-98gfq" Oct 30 16:36:03.852060 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:03.852043 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c22edc73-463a-4439-92aa-601b4cddd714-istio-data\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-98gfq\" (UID: \"c22edc73-463a-4439-92aa-601b4cddd714\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-98gfq" Oct 30 16:36:03.853463 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:03.853438 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c22edc73-463a-4439-92aa-601b4cddd714-istio-envoy\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-98gfq\" (UID: \"c22edc73-463a-4439-92aa-601b4cddd714\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-98gfq" Oct 30 16:36:03.853883 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:03.853865 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c22edc73-463a-4439-92aa-601b4cddd714-istio-podinfo\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-98gfq\" (UID: \"c22edc73-463a-4439-92aa-601b4cddd714\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-98gfq" Oct 30 16:36:03.878394 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:03.878340 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrkxn\" (UniqueName: \"kubernetes.io/projected/c22edc73-463a-4439-92aa-601b4cddd714-kube-api-access-nrkxn\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-98gfq\" (UID: \"c22edc73-463a-4439-92aa-601b4cddd714\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-98gfq" Oct 30 16:36:03.879774 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:03.879755 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c22edc73-463a-4439-92aa-601b4cddd714-istio-token\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-98gfq\" (UID: \"c22edc73-463a-4439-92aa-601b4cddd714\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-98gfq" Oct 30 16:36:03.926784 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:03.926760 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-98gfq" Oct 30 16:36:04.245914 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:04.245889 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-98gfq"] Oct 30 16:36:04.248218 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:36:04.248194 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc22edc73_463a_4439_92aa_601b4cddd714.slice/crio-538324dedebe43c2b4c2867c742b92eefb8d3ce338a99020628d9bd5014820b6 WatchSource:0}: Error finding container 538324dedebe43c2b4c2867c742b92eefb8d3ce338a99020628d9bd5014820b6: Status 404 returned error can't find the container with id 538324dedebe43c2b4c2867c742b92eefb8d3ce338a99020628d9bd5014820b6 Oct 30 16:36:04.250907 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:04.250838 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:36:04.251021 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:04.250929 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:36:04.251021 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:04.250974 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:36:04.300974 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:04.300950 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-98gfq" event={"ID":"c22edc73-463a-4439-92aa-601b4cddd714","Type":"ContainerStarted","Data":"538324dedebe43c2b4c2867c742b92eefb8d3ce338a99020628d9bd5014820b6"} Oct 30 16:36:05.307471 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:05.307431 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-98gfq" event={"ID":"c22edc73-463a-4439-92aa-601b4cddd714","Type":"ContainerStarted","Data":"80c70291d36578c73cf3ef5ac29fca0a9b96ca339b43923582b9ecf045d4afdf"} Oct 30 16:36:05.385192 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:05.385138 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-98gfq" podStartSLOduration=2.385124654 podStartE2EDuration="2.385124654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:36:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:36:05.382663785 +0000 UTC m=+920.909999023" watchObservedRunningTime="2026-10-30 16:36:05.385124654 +0000 UTC m=+920.912459873" Oct 30 16:36:05.927481 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:05.927452 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-98gfq" Oct 30 16:36:05.952930 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:05.952893 5052 patch_prober.go:29] interesting pod/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-98gfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Oct 30 16:36:05.953210 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:05.953177 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-98gfq" podUID="c22edc73-463a-4439-92aa-601b4cddd714" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Oct 30 16:36:06.936983 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:06.936942 5052 patch_prober.go:29] interesting pod/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-98gfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Oct 30 16:36:06.937561 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:06.937528 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-98gfq" podUID="c22edc73-463a-4439-92aa-601b4cddd714" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Oct 30 16:36:07.932421 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:07.931977 5052 patch_prober.go:29] interesting pod/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-98gfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Oct 30 16:36:07.932421 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:07.932044 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-98gfq" podUID="c22edc73-463a-4439-92aa-601b4cddd714" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Oct 30 16:36:08.927630 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:08.927575 5052 patch_prober.go:29] interesting pod/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-98gfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Oct 30 16:36:08.928201 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:08.928167 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-98gfq" podUID="c22edc73-463a-4439-92aa-601b4cddd714" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Oct 30 16:36:09.929800 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:09.929761 5052 patch_prober.go:29] interesting pod/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-98gfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Oct 30 16:36:09.930423 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:09.930390 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-98gfq" podUID="c22edc73-463a-4439-92aa-601b4cddd714" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Oct 30 16:36:10.927935 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:10.927900 5052 patch_prober.go:29] interesting pod/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-98gfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Oct 30 16:36:10.928194 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:10.928164 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-98gfq" podUID="c22edc73-463a-4439-92aa-601b4cddd714" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Oct 30 16:36:11.932075 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:11.931992 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-98gfq" Oct 30 16:36:11.932537 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:11.932359 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-98gfq" Oct 30 16:36:11.933142 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:11.933121 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-98gfq" Oct 30 16:36:14.785614 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:14.785561 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cdwcj"] Oct 30 16:36:18.847619 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:36:18.847558 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 16:36:33.573168 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:33.573132 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-xjhj4"] Oct 30 16:36:33.577028 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:33.577005 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-xjhj4" Oct 30 16:36:33.651717 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:33.643996 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-xjhj4"] Oct 30 16:36:33.651960 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:33.650039 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/029d295c-a65a-4191-9754-a421c21464ba-istiod-ca-cert\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-xjhj4\" (UID: \"029d295c-a65a-4191-9754-a421c21464ba\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-xjhj4" Oct 30 16:36:33.669743 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:33.669708 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/029d295c-a65a-4191-9754-a421c21464ba-credential-socket\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-xjhj4\" (UID: \"029d295c-a65a-4191-9754-a421c21464ba\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-xjhj4" Oct 30 16:36:33.669925 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:33.669774 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/029d295c-a65a-4191-9754-a421c21464ba-workload-socket\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-xjhj4\" (UID: \"029d295c-a65a-4191-9754-a421c21464ba\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-xjhj4" Oct 30 16:36:33.669925 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:33.669898 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/029d295c-a65a-4191-9754-a421c21464ba-istio-podinfo\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-xjhj4\" (UID: \"029d295c-a65a-4191-9754-a421c21464ba\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-xjhj4" Oct 30 16:36:33.670010 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:33.669930 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/029d295c-a65a-4191-9754-a421c21464ba-istio-envoy\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-xjhj4\" (UID: \"029d295c-a65a-4191-9754-a421c21464ba\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-xjhj4" Oct 30 16:36:33.670010 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:33.669984 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/029d295c-a65a-4191-9754-a421c21464ba-workload-certs\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-xjhj4\" (UID: \"029d295c-a65a-4191-9754-a421c21464ba\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-xjhj4" Oct 30 16:36:33.670127 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:33.670037 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghmvk\" (UniqueName: \"kubernetes.io/projected/029d295c-a65a-4191-9754-a421c21464ba-kube-api-access-ghmvk\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-xjhj4\" (UID: \"029d295c-a65a-4191-9754-a421c21464ba\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-xjhj4" Oct 30 16:36:33.670127 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:33.670111 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/029d295c-a65a-4191-9754-a421c21464ba-istio-data\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-xjhj4\" (UID: \"029d295c-a65a-4191-9754-a421c21464ba\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-xjhj4" Oct 30 16:36:33.670243 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:33.670182 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/029d295c-a65a-4191-9754-a421c21464ba-istio-token\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-xjhj4\" (UID: \"029d295c-a65a-4191-9754-a421c21464ba\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-xjhj4" Oct 30 16:36:33.776964 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:33.776909 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/029d295c-a65a-4191-9754-a421c21464ba-credential-socket\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-xjhj4\" (UID: \"029d295c-a65a-4191-9754-a421c21464ba\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-xjhj4" Oct 30 16:36:33.777159 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:33.776971 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/029d295c-a65a-4191-9754-a421c21464ba-workload-socket\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-xjhj4\" (UID: \"029d295c-a65a-4191-9754-a421c21464ba\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-xjhj4" Oct 30 16:36:33.777159 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:33.777069 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/029d295c-a65a-4191-9754-a421c21464ba-istio-podinfo\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-xjhj4\" (UID: \"029d295c-a65a-4191-9754-a421c21464ba\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-xjhj4" Oct 30 16:36:33.777159 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:33.777096 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/029d295c-a65a-4191-9754-a421c21464ba-istio-envoy\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-xjhj4\" (UID: \"029d295c-a65a-4191-9754-a421c21464ba\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-xjhj4" Oct 30 16:36:33.777355 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:33.777141 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/029d295c-a65a-4191-9754-a421c21464ba-workload-certs\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-xjhj4\" (UID: \"029d295c-a65a-4191-9754-a421c21464ba\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-xjhj4" Oct 30 16:36:33.777355 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:33.777213 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghmvk\" (UniqueName: \"kubernetes.io/projected/029d295c-a65a-4191-9754-a421c21464ba-kube-api-access-ghmvk\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-xjhj4\" (UID: \"029d295c-a65a-4191-9754-a421c21464ba\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-xjhj4" Oct 30 16:36:33.777355 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:33.777275 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/029d295c-a65a-4191-9754-a421c21464ba-istio-data\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-xjhj4\" (UID: \"029d295c-a65a-4191-9754-a421c21464ba\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-xjhj4" Oct 30 16:36:33.777516 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:33.777359 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/029d295c-a65a-4191-9754-a421c21464ba-istio-token\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-xjhj4\" (UID: \"029d295c-a65a-4191-9754-a421c21464ba\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-xjhj4" Oct 30 16:36:33.777516 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:33.777407 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/029d295c-a65a-4191-9754-a421c21464ba-istiod-ca-cert\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-xjhj4\" (UID: \"029d295c-a65a-4191-9754-a421c21464ba\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-xjhj4" Oct 30 16:36:33.778688 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:33.778634 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/029d295c-a65a-4191-9754-a421c21464ba-workload-socket\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-xjhj4\" (UID: \"029d295c-a65a-4191-9754-a421c21464ba\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-xjhj4" Oct 30 16:36:33.778834 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:33.778683 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/029d295c-a65a-4191-9754-a421c21464ba-credential-socket\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-xjhj4\" (UID: \"029d295c-a65a-4191-9754-a421c21464ba\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-xjhj4" Oct 30 16:36:33.779255 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:33.779231 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/029d295c-a65a-4191-9754-a421c21464ba-istiod-ca-cert\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-xjhj4\" (UID: \"029d295c-a65a-4191-9754-a421c21464ba\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-xjhj4" Oct 30 16:36:33.785846 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:33.785811 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/029d295c-a65a-4191-9754-a421c21464ba-istio-data\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-xjhj4\" (UID: \"029d295c-a65a-4191-9754-a421c21464ba\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-xjhj4" Oct 30 16:36:33.786056 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:33.786019 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/029d295c-a65a-4191-9754-a421c21464ba-workload-certs\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-xjhj4\" (UID: \"029d295c-a65a-4191-9754-a421c21464ba\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-xjhj4" Oct 30 16:36:33.817176 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:33.817135 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/029d295c-a65a-4191-9754-a421c21464ba-istio-podinfo\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-xjhj4\" (UID: \"029d295c-a65a-4191-9754-a421c21464ba\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-xjhj4" Oct 30 16:36:33.821603 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:33.818702 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/029d295c-a65a-4191-9754-a421c21464ba-istio-envoy\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-xjhj4\" (UID: \"029d295c-a65a-4191-9754-a421c21464ba\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-xjhj4" Oct 30 16:36:33.821603 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:33.820133 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/029d295c-a65a-4191-9754-a421c21464ba-istio-token\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-xjhj4\" (UID: \"029d295c-a65a-4191-9754-a421c21464ba\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-xjhj4" Oct 30 16:36:33.824879 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:33.824818 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghmvk\" (UniqueName: \"kubernetes.io/projected/029d295c-a65a-4191-9754-a421c21464ba-kube-api-access-ghmvk\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-xjhj4\" (UID: \"029d295c-a65a-4191-9754-a421c21464ba\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-xjhj4" Oct 30 16:36:33.965861 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:33.965832 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-xjhj4" Oct 30 16:36:34.283345 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:34.283299 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-xjhj4"] Oct 30 16:36:34.286327 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:36:34.286298 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod029d295c_a65a_4191_9754_a421c21464ba.slice/crio-2816532f7db38518153d64bee4ebbd4dcfdd2d08948225428c32a364c50d9443 WatchSource:0}: Error finding container 2816532f7db38518153d64bee4ebbd4dcfdd2d08948225428c32a364c50d9443: Status 404 returned error can't find the container with id 2816532f7db38518153d64bee4ebbd4dcfdd2d08948225428c32a364c50d9443 Oct 30 16:36:34.288574 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:34.288543 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:36:34.288661 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:34.288621 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:36:34.288661 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:34.288651 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:36:34.638077 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:34.637968 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-xjhj4" event={"ID":"029d295c-a65a-4191-9754-a421c21464ba","Type":"ContainerStarted","Data":"8b4392bec4b7bb73bffe133bc736111a10345438eb82677a89f00d7358eb3889"} Oct 30 16:36:34.638077 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:34.638049 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-xjhj4" event={"ID":"029d295c-a65a-4191-9754-a421c21464ba","Type":"ContainerStarted","Data":"2816532f7db38518153d64bee4ebbd4dcfdd2d08948225428c32a364c50d9443"} Oct 30 16:36:34.690406 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:34.690354 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-xjhj4" podStartSLOduration=1.6903402440000002 podStartE2EDuration="1.690340244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:36:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:36:34.68892179 +0000 UTC m=+950.216257009" watchObservedRunningTime="2026-10-30 16:36:34.690340244 +0000 UTC m=+950.217675463" Oct 30 16:36:34.965920 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:34.965893 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-xjhj4" Oct 30 16:36:35.968795 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:35.968761 5052 patch_prober.go:29] interesting pod/e2e-worker-w3-0b2348-openshift-default-85679bc47c-xjhj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Oct 30 16:36:35.969202 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:35.968815 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-xjhj4" podUID="029d295c-a65a-4191-9754-a421c21464ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Oct 30 16:36:36.967115 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:36.967083 5052 patch_prober.go:29] interesting pod/e2e-worker-w3-0b2348-openshift-default-85679bc47c-xjhj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Oct 30 16:36:36.967321 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:36.967179 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-xjhj4" podUID="029d295c-a65a-4191-9754-a421c21464ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Oct 30 16:36:37.966743 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:37.966701 5052 patch_prober.go:29] interesting pod/e2e-worker-w3-0b2348-openshift-default-85679bc47c-xjhj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Oct 30 16:36:37.967128 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:37.966767 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-xjhj4" podUID="029d295c-a65a-4191-9754-a421c21464ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Oct 30 16:36:38.321377 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:38.321297 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cdwcj"] Oct 30 16:36:38.969603 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:38.967964 5052 patch_prober.go:29] interesting pod/e2e-worker-w3-0b2348-openshift-default-85679bc47c-xjhj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Oct 30 16:36:38.969603 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:38.968046 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-xjhj4" podUID="029d295c-a65a-4191-9754-a421c21464ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Oct 30 16:36:39.970078 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:39.970034 5052 patch_prober.go:29] interesting pod/e2e-worker-w3-0b2348-openshift-default-85679bc47c-xjhj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Oct 30 16:36:39.970722 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:39.970687 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-xjhj4" podUID="029d295c-a65a-4191-9754-a421c21464ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Oct 30 16:36:40.967194 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:40.967157 5052 patch_prober.go:29] interesting pod/e2e-worker-w3-0b2348-openshift-default-85679bc47c-xjhj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Oct 30 16:36:40.967482 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:40.967438 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-xjhj4" podUID="029d295c-a65a-4191-9754-a421c21464ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Oct 30 16:36:42.649376 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:42.649344 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-xjhj4" Oct 30 16:36:42.649942 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:42.649624 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-xjhj4" Oct 30 16:36:42.664215 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:42.664192 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-xjhj4" Oct 30 16:36:43.605222 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:43.605188 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-bvwpz"] Oct 30 16:36:43.611043 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:43.611018 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-bvwpz" Oct 30 16:36:43.632153 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:43.632124 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-bvwpz"] Oct 30 16:36:43.639890 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:43.639860 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-a8c22d-openshift-default-dockercfg-qfhzt\"" Oct 30 16:36:43.722253 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:43.722206 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j664n\" (UniqueName: \"kubernetes.io/projected/e8eabbfd-4dce-4087-a350-c55cb7c7baf3-kube-api-access-j664n\") pod \"e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-bvwpz\" (UID: \"e8eabbfd-4dce-4087-a350-c55cb7c7baf3\") " pod="openshift-ingress/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-bvwpz" Oct 30 16:36:43.722949 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:43.722923 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8eabbfd-4dce-4087-a350-c55cb7c7baf3-workload-certs\") pod \"e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-bvwpz\" (UID: \"e8eabbfd-4dce-4087-a350-c55cb7c7baf3\") " pod="openshift-ingress/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-bvwpz" Oct 30 16:36:43.723161 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:43.723147 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8eabbfd-4dce-4087-a350-c55cb7c7baf3-istio-podinfo\") pod \"e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-bvwpz\" (UID: \"e8eabbfd-4dce-4087-a350-c55cb7c7baf3\") " pod="openshift-ingress/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-bvwpz" Oct 30 16:36:43.723301 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:43.723286 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8eabbfd-4dce-4087-a350-c55cb7c7baf3-workload-socket\") pod \"e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-bvwpz\" (UID: \"e8eabbfd-4dce-4087-a350-c55cb7c7baf3\") " pod="openshift-ingress/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-bvwpz" Oct 30 16:36:43.723458 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:43.723445 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8eabbfd-4dce-4087-a350-c55cb7c7baf3-istio-envoy\") pod \"e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-bvwpz\" (UID: \"e8eabbfd-4dce-4087-a350-c55cb7c7baf3\") " pod="openshift-ingress/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-bvwpz" Oct 30 16:36:43.723619 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:43.723605 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8eabbfd-4dce-4087-a350-c55cb7c7baf3-istiod-ca-cert\") pod \"e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-bvwpz\" (UID: \"e8eabbfd-4dce-4087-a350-c55cb7c7baf3\") " pod="openshift-ingress/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-bvwpz" Oct 30 16:36:43.723778 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:43.723762 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8eabbfd-4dce-4087-a350-c55cb7c7baf3-istio-token\") pod \"e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-bvwpz\" (UID: \"e8eabbfd-4dce-4087-a350-c55cb7c7baf3\") " pod="openshift-ingress/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-bvwpz" Oct 30 16:36:43.723908 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:43.723893 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8eabbfd-4dce-4087-a350-c55cb7c7baf3-istio-data\") pod \"e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-bvwpz\" (UID: \"e8eabbfd-4dce-4087-a350-c55cb7c7baf3\") " pod="openshift-ingress/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-bvwpz" Oct 30 16:36:43.724063 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:43.724050 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8eabbfd-4dce-4087-a350-c55cb7c7baf3-credential-socket\") pod \"e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-bvwpz\" (UID: \"e8eabbfd-4dce-4087-a350-c55cb7c7baf3\") " pod="openshift-ingress/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-bvwpz" Oct 30 16:36:43.826260 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:43.826224 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8eabbfd-4dce-4087-a350-c55cb7c7baf3-istiod-ca-cert\") pod \"e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-bvwpz\" (UID: \"e8eabbfd-4dce-4087-a350-c55cb7c7baf3\") " pod="openshift-ingress/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-bvwpz" Oct 30 16:36:43.826529 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:43.826505 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8eabbfd-4dce-4087-a350-c55cb7c7baf3-istio-token\") pod \"e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-bvwpz\" (UID: \"e8eabbfd-4dce-4087-a350-c55cb7c7baf3\") " pod="openshift-ingress/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-bvwpz" Oct 30 16:36:43.826674 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:43.826659 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8eabbfd-4dce-4087-a350-c55cb7c7baf3-istio-data\") pod \"e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-bvwpz\" (UID: \"e8eabbfd-4dce-4087-a350-c55cb7c7baf3\") " pod="openshift-ingress/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-bvwpz" Oct 30 16:36:43.826837 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:43.826822 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8eabbfd-4dce-4087-a350-c55cb7c7baf3-credential-socket\") pod \"e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-bvwpz\" (UID: \"e8eabbfd-4dce-4087-a350-c55cb7c7baf3\") " pod="openshift-ingress/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-bvwpz" Oct 30 16:36:43.827001 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:43.826987 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j664n\" (UniqueName: \"kubernetes.io/projected/e8eabbfd-4dce-4087-a350-c55cb7c7baf3-kube-api-access-j664n\") pod \"e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-bvwpz\" (UID: \"e8eabbfd-4dce-4087-a350-c55cb7c7baf3\") " pod="openshift-ingress/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-bvwpz" Oct 30 16:36:43.827122 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:43.827110 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8eabbfd-4dce-4087-a350-c55cb7c7baf3-workload-certs\") pod \"e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-bvwpz\" (UID: \"e8eabbfd-4dce-4087-a350-c55cb7c7baf3\") " pod="openshift-ingress/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-bvwpz" Oct 30 16:36:43.827268 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:43.827256 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8eabbfd-4dce-4087-a350-c55cb7c7baf3-istio-podinfo\") pod \"e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-bvwpz\" (UID: \"e8eabbfd-4dce-4087-a350-c55cb7c7baf3\") " pod="openshift-ingress/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-bvwpz" Oct 30 16:36:43.827366 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:43.827352 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8eabbfd-4dce-4087-a350-c55cb7c7baf3-workload-socket\") pod \"e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-bvwpz\" (UID: \"e8eabbfd-4dce-4087-a350-c55cb7c7baf3\") " pod="openshift-ingress/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-bvwpz" Oct 30 16:36:43.827483 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:43.827458 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8eabbfd-4dce-4087-a350-c55cb7c7baf3-credential-socket\") pod \"e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-bvwpz\" (UID: \"e8eabbfd-4dce-4087-a350-c55cb7c7baf3\") " pod="openshift-ingress/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-bvwpz" Oct 30 16:36:43.827607 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:43.827570 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8eabbfd-4dce-4087-a350-c55cb7c7baf3-istio-envoy\") pod \"e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-bvwpz\" (UID: \"e8eabbfd-4dce-4087-a350-c55cb7c7baf3\") " pod="openshift-ingress/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-bvwpz" Oct 30 16:36:43.828357 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:43.828330 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8eabbfd-4dce-4087-a350-c55cb7c7baf3-istio-data\") pod \"e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-bvwpz\" (UID: \"e8eabbfd-4dce-4087-a350-c55cb7c7baf3\") " pod="openshift-ingress/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-bvwpz" Oct 30 16:36:43.828965 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:43.828940 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8eabbfd-4dce-4087-a350-c55cb7c7baf3-workload-certs\") pod \"e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-bvwpz\" (UID: \"e8eabbfd-4dce-4087-a350-c55cb7c7baf3\") " pod="openshift-ingress/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-bvwpz" Oct 30 16:36:43.829730 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:43.827173 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8eabbfd-4dce-4087-a350-c55cb7c7baf3-istiod-ca-cert\") pod \"e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-bvwpz\" (UID: \"e8eabbfd-4dce-4087-a350-c55cb7c7baf3\") " pod="openshift-ingress/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-bvwpz" Oct 30 16:36:43.830160 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:43.830141 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8eabbfd-4dce-4087-a350-c55cb7c7baf3-workload-socket\") pod \"e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-bvwpz\" (UID: \"e8eabbfd-4dce-4087-a350-c55cb7c7baf3\") " pod="openshift-ingress/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-bvwpz" Oct 30 16:36:43.840312 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:43.840268 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8eabbfd-4dce-4087-a350-c55cb7c7baf3-istio-envoy\") pod \"e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-bvwpz\" (UID: \"e8eabbfd-4dce-4087-a350-c55cb7c7baf3\") " pod="openshift-ingress/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-bvwpz" Oct 30 16:36:43.853258 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:43.853235 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8eabbfd-4dce-4087-a350-c55cb7c7baf3-istio-podinfo\") pod \"e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-bvwpz\" (UID: \"e8eabbfd-4dce-4087-a350-c55cb7c7baf3\") " pod="openshift-ingress/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-bvwpz" Oct 30 16:36:43.862126 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:43.862065 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8eabbfd-4dce-4087-a350-c55cb7c7baf3-istio-token\") pod \"e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-bvwpz\" (UID: \"e8eabbfd-4dce-4087-a350-c55cb7c7baf3\") " pod="openshift-ingress/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-bvwpz" Oct 30 16:36:43.862931 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:43.862885 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j664n\" (UniqueName: \"kubernetes.io/projected/e8eabbfd-4dce-4087-a350-c55cb7c7baf3-kube-api-access-j664n\") pod \"e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-bvwpz\" (UID: \"e8eabbfd-4dce-4087-a350-c55cb7c7baf3\") " pod="openshift-ingress/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-bvwpz" Oct 30 16:36:43.927887 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:43.927843 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-bvwpz" Oct 30 16:36:44.327684 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:44.327655 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-bvwpz"] Oct 30 16:36:44.330432 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:36:44.330396 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8eabbfd_4dce_4087_a350_c55cb7c7baf3.slice/crio-c1abf298a0705a2443c0b155474860cdb6a41c74c8b8a0955fa9b6d6108266bf WatchSource:0}: Error finding container c1abf298a0705a2443c0b155474860cdb6a41c74c8b8a0955fa9b6d6108266bf: Status 404 returned error can't find the container with id c1abf298a0705a2443c0b155474860cdb6a41c74c8b8a0955fa9b6d6108266bf Oct 30 16:36:44.333471 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:44.333435 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:36:44.333554 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:44.333522 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:36:44.333625 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:44.333562 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:36:44.778477 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:44.777365 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-bvwpz" event={"ID":"e8eabbfd-4dce-4087-a350-c55cb7c7baf3","Type":"ContainerStarted","Data":"e6f611504d91d2a3ed7e8c61e2d6eec3166562879ead1ab42e6e256e155349a9"} Oct 30 16:36:44.778477 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:44.777470 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-bvwpz" event={"ID":"e8eabbfd-4dce-4087-a350-c55cb7c7baf3","Type":"ContainerStarted","Data":"c1abf298a0705a2443c0b155474860cdb6a41c74c8b8a0955fa9b6d6108266bf"} Oct 30 16:36:44.827298 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:44.827237 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-bvwpz" podStartSLOduration=1.827219966 podStartE2EDuration="1.827219966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:36:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:36:44.822964232 +0000 UTC m=+960.350299451" watchObservedRunningTime="2026-10-30 16:36:44.827219966 +0000 UTC m=+960.354555186" Oct 30 16:36:44.945603 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:44.938695 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-bvwpz" Oct 30 16:36:45.956095 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:45.953698 5052 patch_prober.go:29] interesting pod/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-bvwpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 30 16:36:45.956095 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:45.953766 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-bvwpz" podUID="e8eabbfd-4dce-4087-a350-c55cb7c7baf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 30 16:36:46.938801 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:46.938768 5052 patch_prober.go:29] interesting pod/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-bvwpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 30 16:36:46.939083 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:46.939057 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-bvwpz" podUID="e8eabbfd-4dce-4087-a350-c55cb7c7baf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 30 16:36:47.073208 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:47.073167 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-l6dw5"] Oct 30 16:36:47.077183 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:47.077162 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-l6dw5" Oct 30 16:36:47.107549 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:47.107515 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-l6dw5"] Oct 30 16:36:47.180298 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:47.180262 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b3c4341-826b-489d-81ae-14bf71548dd2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-l6dw5\" (UID: \"4b3c4341-826b-489d-81ae-14bf71548dd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-l6dw5" Oct 30 16:36:47.180298 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:47.180306 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b3c4341-826b-489d-81ae-14bf71548dd2-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-l6dw5\" (UID: \"4b3c4341-826b-489d-81ae-14bf71548dd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-l6dw5" Oct 30 16:36:47.180521 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:47.180467 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b3c4341-826b-489d-81ae-14bf71548dd2-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-l6dw5\" (UID: \"4b3c4341-826b-489d-81ae-14bf71548dd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-l6dw5" Oct 30 16:36:47.180521 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:47.180500 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b3c4341-826b-489d-81ae-14bf71548dd2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-l6dw5\" (UID: \"4b3c4341-826b-489d-81ae-14bf71548dd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-l6dw5" Oct 30 16:36:47.180521 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:47.180519 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b3c4341-826b-489d-81ae-14bf71548dd2-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-l6dw5\" (UID: \"4b3c4341-826b-489d-81ae-14bf71548dd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-l6dw5" Oct 30 16:36:47.180720 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:47.180677 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b3c4341-826b-489d-81ae-14bf71548dd2-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-l6dw5\" (UID: \"4b3c4341-826b-489d-81ae-14bf71548dd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-l6dw5" Oct 30 16:36:47.180794 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:47.180771 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b3c4341-826b-489d-81ae-14bf71548dd2-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-l6dw5\" (UID: \"4b3c4341-826b-489d-81ae-14bf71548dd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-l6dw5" Oct 30 16:36:47.180848 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:47.180819 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b3c4341-826b-489d-81ae-14bf71548dd2-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-l6dw5\" (UID: \"4b3c4341-826b-489d-81ae-14bf71548dd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-l6dw5" Oct 30 16:36:47.180903 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:47.180890 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fxpn\" (UniqueName: \"kubernetes.io/projected/4b3c4341-826b-489d-81ae-14bf71548dd2-kube-api-access-4fxpn\") pod \"maas-default-gateway-openshift-default-76f4876667-l6dw5\" (UID: \"4b3c4341-826b-489d-81ae-14bf71548dd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-l6dw5" Oct 30 16:36:47.282359 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:47.282278 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b3c4341-826b-489d-81ae-14bf71548dd2-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-l6dw5\" (UID: \"4b3c4341-826b-489d-81ae-14bf71548dd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-l6dw5" Oct 30 16:36:47.282359 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:47.282323 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b3c4341-826b-489d-81ae-14bf71548dd2-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-l6dw5\" (UID: \"4b3c4341-826b-489d-81ae-14bf71548dd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-l6dw5" Oct 30 16:36:47.282359 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:47.282343 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fxpn\" (UniqueName: \"kubernetes.io/projected/4b3c4341-826b-489d-81ae-14bf71548dd2-kube-api-access-4fxpn\") pod \"maas-default-gateway-openshift-default-76f4876667-l6dw5\" (UID: \"4b3c4341-826b-489d-81ae-14bf71548dd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-l6dw5" Oct 30 16:36:47.282682 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:47.282391 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b3c4341-826b-489d-81ae-14bf71548dd2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-l6dw5\" (UID: \"4b3c4341-826b-489d-81ae-14bf71548dd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-l6dw5" Oct 30 16:36:47.282682 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:47.282412 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b3c4341-826b-489d-81ae-14bf71548dd2-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-l6dw5\" (UID: \"4b3c4341-826b-489d-81ae-14bf71548dd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-l6dw5" Oct 30 16:36:47.282682 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:47.282486 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b3c4341-826b-489d-81ae-14bf71548dd2-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-l6dw5\" (UID: \"4b3c4341-826b-489d-81ae-14bf71548dd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-l6dw5" Oct 30 16:36:47.282682 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:47.282513 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b3c4341-826b-489d-81ae-14bf71548dd2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-l6dw5\" (UID: \"4b3c4341-826b-489d-81ae-14bf71548dd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-l6dw5" Oct 30 16:36:47.282682 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:47.282539 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b3c4341-826b-489d-81ae-14bf71548dd2-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-l6dw5\" (UID: \"4b3c4341-826b-489d-81ae-14bf71548dd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-l6dw5" Oct 30 16:36:47.282682 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:47.282661 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b3c4341-826b-489d-81ae-14bf71548dd2-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-l6dw5\" (UID: \"4b3c4341-826b-489d-81ae-14bf71548dd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-l6dw5" Oct 30 16:36:47.282995 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:47.282730 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b3c4341-826b-489d-81ae-14bf71548dd2-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-l6dw5\" (UID: \"4b3c4341-826b-489d-81ae-14bf71548dd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-l6dw5" Oct 30 16:36:47.282995 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:47.282836 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b3c4341-826b-489d-81ae-14bf71548dd2-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-l6dw5\" (UID: \"4b3c4341-826b-489d-81ae-14bf71548dd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-l6dw5" Oct 30 16:36:47.283104 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:47.283047 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b3c4341-826b-489d-81ae-14bf71548dd2-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-l6dw5\" (UID: \"4b3c4341-826b-489d-81ae-14bf71548dd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-l6dw5" Oct 30 16:36:47.283165 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:47.283112 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b3c4341-826b-489d-81ae-14bf71548dd2-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-l6dw5\" (UID: \"4b3c4341-826b-489d-81ae-14bf71548dd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-l6dw5" Oct 30 16:36:47.283332 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:47.283308 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b3c4341-826b-489d-81ae-14bf71548dd2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-l6dw5\" (UID: \"4b3c4341-826b-489d-81ae-14bf71548dd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-l6dw5" Oct 30 16:36:47.285428 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:47.285400 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b3c4341-826b-489d-81ae-14bf71548dd2-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-l6dw5\" (UID: \"4b3c4341-826b-489d-81ae-14bf71548dd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-l6dw5" Oct 30 16:36:47.285877 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:47.285834 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b3c4341-826b-489d-81ae-14bf71548dd2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-l6dw5\" (UID: \"4b3c4341-826b-489d-81ae-14bf71548dd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-l6dw5" Oct 30 16:36:47.305687 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:47.305662 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b3c4341-826b-489d-81ae-14bf71548dd2-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-l6dw5\" (UID: \"4b3c4341-826b-489d-81ae-14bf71548dd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-l6dw5" Oct 30 16:36:47.305768 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:47.305697 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fxpn\" (UniqueName: \"kubernetes.io/projected/4b3c4341-826b-489d-81ae-14bf71548dd2-kube-api-access-4fxpn\") pod \"maas-default-gateway-openshift-default-76f4876667-l6dw5\" (UID: \"4b3c4341-826b-489d-81ae-14bf71548dd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-l6dw5" Oct 30 16:36:47.392464 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:47.392428 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-l6dw5" Oct 30 16:36:47.764836 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:47.764802 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-l6dw5"] Oct 30 16:36:47.768725 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:36:47.768694 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b3c4341_826b_489d_81ae_14bf71548dd2.slice/crio-6c135430463cbe1328bdae9a63dbcb496fbdb48e1472839813cd052a427ade35 WatchSource:0}: Error finding container 6c135430463cbe1328bdae9a63dbcb496fbdb48e1472839813cd052a427ade35: Status 404 returned error can't find the container with id 6c135430463cbe1328bdae9a63dbcb496fbdb48e1472839813cd052a427ade35 Oct 30 16:36:47.771282 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:47.771254 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:36:47.771353 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:47.771318 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:36:47.771353 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:47.771345 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:36:47.796762 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:47.796729 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-l6dw5" event={"ID":"4b3c4341-826b-489d-81ae-14bf71548dd2","Type":"ContainerStarted","Data":"6c135430463cbe1328bdae9a63dbcb496fbdb48e1472839813cd052a427ade35"} Oct 30 16:36:47.932010 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:47.931971 5052 patch_prober.go:29] interesting pod/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-bvwpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 30 16:36:47.932304 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:47.932273 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-bvwpz" podUID="e8eabbfd-4dce-4087-a350-c55cb7c7baf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 30 16:36:48.789710 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:48.788315 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2slrc"] Oct 30 16:36:48.794675 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:48.794649 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2slrc" Oct 30 16:36:48.809783 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:48.809757 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2slrc"] Oct 30 16:36:48.813409 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:48.813375 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-l6dw5" event={"ID":"4b3c4341-826b-489d-81ae-14bf71548dd2","Type":"ContainerStarted","Data":"61a80927dc0ea83ef7657778c0d0804ae5e9df84cd7cd20f0694497902857cbd"} Oct 30 16:36:48.896064 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:36:48.896033 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 16:36:48.914890 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:48.914853 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69a65b89-4d81-476b-b4b7-06736537fa4a-istio-envoy\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2slrc\" (UID: \"69a65b89-4d81-476b-b4b7-06736537fa4a\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2slrc" Oct 30 16:36:48.915065 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:48.914908 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69a65b89-4d81-476b-b4b7-06736537fa4a-istio-data\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2slrc\" (UID: \"69a65b89-4d81-476b-b4b7-06736537fa4a\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2slrc" Oct 30 16:36:48.915065 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:48.914974 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69a65b89-4d81-476b-b4b7-06736537fa4a-workload-socket\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2slrc\" (UID: \"69a65b89-4d81-476b-b4b7-06736537fa4a\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2slrc" Oct 30 16:36:48.915065 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:48.914999 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69a65b89-4d81-476b-b4b7-06736537fa4a-istiod-ca-cert\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2slrc\" (UID: \"69a65b89-4d81-476b-b4b7-06736537fa4a\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2slrc" Oct 30 16:36:48.915065 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:48.915029 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69a65b89-4d81-476b-b4b7-06736537fa4a-workload-certs\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2slrc\" (UID: \"69a65b89-4d81-476b-b4b7-06736537fa4a\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2slrc" Oct 30 16:36:48.915065 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:48.915060 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9pf8\" (UniqueName: \"kubernetes.io/projected/69a65b89-4d81-476b-b4b7-06736537fa4a-kube-api-access-m9pf8\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2slrc\" (UID: \"69a65b89-4d81-476b-b4b7-06736537fa4a\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2slrc" Oct 30 16:36:48.915326 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:48.915148 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69a65b89-4d81-476b-b4b7-06736537fa4a-istio-podinfo\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2slrc\" (UID: \"69a65b89-4d81-476b-b4b7-06736537fa4a\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2slrc" Oct 30 16:36:48.915326 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:48.915208 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69a65b89-4d81-476b-b4b7-06736537fa4a-credential-socket\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2slrc\" (UID: \"69a65b89-4d81-476b-b4b7-06736537fa4a\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2slrc" Oct 30 16:36:48.915326 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:48.915286 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69a65b89-4d81-476b-b4b7-06736537fa4a-istio-token\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2slrc\" (UID: \"69a65b89-4d81-476b-b4b7-06736537fa4a\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2slrc" Oct 30 16:36:48.937610 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:48.931919 5052 patch_prober.go:29] interesting pod/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-bvwpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 30 16:36:48.982760 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:48.982696 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-bvwpz" podUID="e8eabbfd-4dce-4087-a350-c55cb7c7baf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 30 16:36:48.985013 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:48.984945 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-l6dw5" podStartSLOduration=1.984930911 podStartE2EDuration="1.984930911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:36:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:36:48.938209722 +0000 UTC m=+964.465544944" watchObservedRunningTime="2026-10-30 16:36:48.984930911 +0000 UTC m=+964.512266129" Oct 30 16:36:49.016371 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:49.016340 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69a65b89-4d81-476b-b4b7-06736537fa4a-istio-envoy\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2slrc\" (UID: \"69a65b89-4d81-476b-b4b7-06736537fa4a\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2slrc" Oct 30 16:36:49.016540 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:49.016400 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69a65b89-4d81-476b-b4b7-06736537fa4a-istio-data\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2slrc\" (UID: \"69a65b89-4d81-476b-b4b7-06736537fa4a\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2slrc" Oct 30 16:36:49.016540 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:49.016454 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69a65b89-4d81-476b-b4b7-06736537fa4a-workload-socket\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2slrc\" (UID: \"69a65b89-4d81-476b-b4b7-06736537fa4a\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2slrc" Oct 30 16:36:49.016540 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:49.016477 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69a65b89-4d81-476b-b4b7-06736537fa4a-istiod-ca-cert\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2slrc\" (UID: \"69a65b89-4d81-476b-b4b7-06736537fa4a\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2slrc" Oct 30 16:36:49.016540 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:49.016508 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69a65b89-4d81-476b-b4b7-06736537fa4a-workload-certs\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2slrc\" (UID: \"69a65b89-4d81-476b-b4b7-06736537fa4a\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2slrc" Oct 30 16:36:49.016793 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:49.016542 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9pf8\" (UniqueName: \"kubernetes.io/projected/69a65b89-4d81-476b-b4b7-06736537fa4a-kube-api-access-m9pf8\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2slrc\" (UID: \"69a65b89-4d81-476b-b4b7-06736537fa4a\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2slrc" Oct 30 16:36:49.016793 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:49.016635 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69a65b89-4d81-476b-b4b7-06736537fa4a-istio-podinfo\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2slrc\" (UID: \"69a65b89-4d81-476b-b4b7-06736537fa4a\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2slrc" Oct 30 16:36:49.016793 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:49.016692 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69a65b89-4d81-476b-b4b7-06736537fa4a-credential-socket\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2slrc\" (UID: \"69a65b89-4d81-476b-b4b7-06736537fa4a\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2slrc" Oct 30 16:36:49.016793 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:49.016755 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69a65b89-4d81-476b-b4b7-06736537fa4a-istio-token\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2slrc\" (UID: \"69a65b89-4d81-476b-b4b7-06736537fa4a\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2slrc" Oct 30 16:36:49.018152 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:49.018104 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69a65b89-4d81-476b-b4b7-06736537fa4a-workload-socket\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2slrc\" (UID: \"69a65b89-4d81-476b-b4b7-06736537fa4a\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2slrc" Oct 30 16:36:49.018270 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:49.018203 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69a65b89-4d81-476b-b4b7-06736537fa4a-istio-data\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2slrc\" (UID: \"69a65b89-4d81-476b-b4b7-06736537fa4a\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2slrc" Oct 30 16:36:49.018495 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:49.018475 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69a65b89-4d81-476b-b4b7-06736537fa4a-workload-certs\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2slrc\" (UID: \"69a65b89-4d81-476b-b4b7-06736537fa4a\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2slrc" Oct 30 16:36:49.018624 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:49.018519 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69a65b89-4d81-476b-b4b7-06736537fa4a-credential-socket\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2slrc\" (UID: \"69a65b89-4d81-476b-b4b7-06736537fa4a\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2slrc" Oct 30 16:36:49.018691 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:49.018542 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69a65b89-4d81-476b-b4b7-06736537fa4a-istiod-ca-cert\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2slrc\" (UID: \"69a65b89-4d81-476b-b4b7-06736537fa4a\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2slrc" Oct 30 16:36:49.021458 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:49.021358 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69a65b89-4d81-476b-b4b7-06736537fa4a-istio-podinfo\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2slrc\" (UID: \"69a65b89-4d81-476b-b4b7-06736537fa4a\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2slrc" Oct 30 16:36:49.022042 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:49.022017 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69a65b89-4d81-476b-b4b7-06736537fa4a-istio-envoy\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2slrc\" (UID: \"69a65b89-4d81-476b-b4b7-06736537fa4a\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2slrc" Oct 30 16:36:49.092714 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:49.092631 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-55x7p"] Oct 30 16:36:49.103256 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:49.103220 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69a65b89-4d81-476b-b4b7-06736537fa4a-istio-token\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2slrc\" (UID: \"69a65b89-4d81-476b-b4b7-06736537fa4a\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2slrc" Oct 30 16:36:49.103917 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:49.103888 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9pf8\" (UniqueName: \"kubernetes.io/projected/69a65b89-4d81-476b-b4b7-06736537fa4a-kube-api-access-m9pf8\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2slrc\" (UID: \"69a65b89-4d81-476b-b4b7-06736537fa4a\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2slrc" Oct 30 16:36:49.116986 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:49.116784 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2slrc" Oct 30 16:36:49.142623 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:49.118926 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-55x7p"] Oct 30 16:36:49.142623 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:49.118924 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-55x7p" Oct 30 16:36:49.277605 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:49.276712 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7l9bl\" (UniqueName: \"kubernetes.io/projected/52ba3acd-4de6-4ae2-8a20-49bf7f966ab0-kube-api-access-7l9bl\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-55x7p\" (UID: \"52ba3acd-4de6-4ae2-8a20-49bf7f966ab0\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-55x7p" Oct 30 16:36:49.277605 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:49.276990 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52ba3acd-4de6-4ae2-8a20-49bf7f966ab0-workload-socket\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-55x7p\" (UID: \"52ba3acd-4de6-4ae2-8a20-49bf7f966ab0\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-55x7p" Oct 30 16:36:49.277605 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:49.277026 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52ba3acd-4de6-4ae2-8a20-49bf7f966ab0-credential-socket\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-55x7p\" (UID: \"52ba3acd-4de6-4ae2-8a20-49bf7f966ab0\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-55x7p" Oct 30 16:36:49.277605 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:49.277067 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52ba3acd-4de6-4ae2-8a20-49bf7f966ab0-istio-podinfo\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-55x7p\" (UID: \"52ba3acd-4de6-4ae2-8a20-49bf7f966ab0\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-55x7p" Oct 30 16:36:49.277605 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:49.277136 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52ba3acd-4de6-4ae2-8a20-49bf7f966ab0-istio-data\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-55x7p\" (UID: \"52ba3acd-4de6-4ae2-8a20-49bf7f966ab0\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-55x7p" Oct 30 16:36:49.277605 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:49.277170 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52ba3acd-4de6-4ae2-8a20-49bf7f966ab0-istio-envoy\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-55x7p\" (UID: \"52ba3acd-4de6-4ae2-8a20-49bf7f966ab0\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-55x7p" Oct 30 16:36:49.277605 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:49.277208 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52ba3acd-4de6-4ae2-8a20-49bf7f966ab0-istiod-ca-cert\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-55x7p\" (UID: \"52ba3acd-4de6-4ae2-8a20-49bf7f966ab0\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-55x7p" Oct 30 16:36:49.277605 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:49.277257 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52ba3acd-4de6-4ae2-8a20-49bf7f966ab0-workload-certs\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-55x7p\" (UID: \"52ba3acd-4de6-4ae2-8a20-49bf7f966ab0\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-55x7p" Oct 30 16:36:49.277605 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:49.277376 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52ba3acd-4de6-4ae2-8a20-49bf7f966ab0-istio-token\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-55x7p\" (UID: \"52ba3acd-4de6-4ae2-8a20-49bf7f966ab0\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-55x7p" Oct 30 16:36:49.382665 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:49.382357 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52ba3acd-4de6-4ae2-8a20-49bf7f966ab0-istio-podinfo\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-55x7p\" (UID: \"52ba3acd-4de6-4ae2-8a20-49bf7f966ab0\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-55x7p" Oct 30 16:36:49.382665 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:49.382446 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52ba3acd-4de6-4ae2-8a20-49bf7f966ab0-istio-data\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-55x7p\" (UID: \"52ba3acd-4de6-4ae2-8a20-49bf7f966ab0\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-55x7p" Oct 30 16:36:49.382665 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:49.382485 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52ba3acd-4de6-4ae2-8a20-49bf7f966ab0-istio-envoy\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-55x7p\" (UID: \"52ba3acd-4de6-4ae2-8a20-49bf7f966ab0\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-55x7p" Oct 30 16:36:49.382665 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:49.382523 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52ba3acd-4de6-4ae2-8a20-49bf7f966ab0-istiod-ca-cert\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-55x7p\" (UID: \"52ba3acd-4de6-4ae2-8a20-49bf7f966ab0\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-55x7p" Oct 30 16:36:49.382665 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:49.382570 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52ba3acd-4de6-4ae2-8a20-49bf7f966ab0-workload-certs\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-55x7p\" (UID: \"52ba3acd-4de6-4ae2-8a20-49bf7f966ab0\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-55x7p" Oct 30 16:36:49.383073 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:49.382730 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52ba3acd-4de6-4ae2-8a20-49bf7f966ab0-istio-token\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-55x7p\" (UID: \"52ba3acd-4de6-4ae2-8a20-49bf7f966ab0\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-55x7p" Oct 30 16:36:49.383073 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:49.382770 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l9bl\" (UniqueName: \"kubernetes.io/projected/52ba3acd-4de6-4ae2-8a20-49bf7f966ab0-kube-api-access-7l9bl\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-55x7p\" (UID: \"52ba3acd-4de6-4ae2-8a20-49bf7f966ab0\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-55x7p" Oct 30 16:36:49.383073 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:49.382906 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52ba3acd-4de6-4ae2-8a20-49bf7f966ab0-workload-socket\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-55x7p\" (UID: \"52ba3acd-4de6-4ae2-8a20-49bf7f966ab0\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-55x7p" Oct 30 16:36:49.383073 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:49.382930 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52ba3acd-4de6-4ae2-8a20-49bf7f966ab0-credential-socket\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-55x7p\" (UID: \"52ba3acd-4de6-4ae2-8a20-49bf7f966ab0\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-55x7p" Oct 30 16:36:49.386640 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:49.383885 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52ba3acd-4de6-4ae2-8a20-49bf7f966ab0-credential-socket\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-55x7p\" (UID: \"52ba3acd-4de6-4ae2-8a20-49bf7f966ab0\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-55x7p" Oct 30 16:36:49.386640 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:49.384233 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52ba3acd-4de6-4ae2-8a20-49bf7f966ab0-workload-certs\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-55x7p\" (UID: \"52ba3acd-4de6-4ae2-8a20-49bf7f966ab0\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-55x7p" Oct 30 16:36:49.386640 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:49.384382 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52ba3acd-4de6-4ae2-8a20-49bf7f966ab0-workload-socket\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-55x7p\" (UID: \"52ba3acd-4de6-4ae2-8a20-49bf7f966ab0\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-55x7p" Oct 30 16:36:49.386640 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:49.384991 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52ba3acd-4de6-4ae2-8a20-49bf7f966ab0-istio-data\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-55x7p\" (UID: \"52ba3acd-4de6-4ae2-8a20-49bf7f966ab0\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-55x7p" Oct 30 16:36:49.386640 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:49.385965 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52ba3acd-4de6-4ae2-8a20-49bf7f966ab0-istiod-ca-cert\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-55x7p\" (UID: \"52ba3acd-4de6-4ae2-8a20-49bf7f966ab0\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-55x7p" Oct 30 16:36:49.389278 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:49.389252 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52ba3acd-4de6-4ae2-8a20-49bf7f966ab0-istio-podinfo\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-55x7p\" (UID: \"52ba3acd-4de6-4ae2-8a20-49bf7f966ab0\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-55x7p" Oct 30 16:36:49.389982 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:49.389961 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52ba3acd-4de6-4ae2-8a20-49bf7f966ab0-istio-envoy\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-55x7p\" (UID: \"52ba3acd-4de6-4ae2-8a20-49bf7f966ab0\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-55x7p" Oct 30 16:36:49.393181 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:49.393084 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-l6dw5" Oct 30 16:36:49.394405 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:49.394380 5052 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-l6dw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 30 16:36:49.394506 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:49.394427 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-l6dw5" podUID="4b3c4341-826b-489d-81ae-14bf71548dd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 30 16:36:49.407446 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:49.407416 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52ba3acd-4de6-4ae2-8a20-49bf7f966ab0-istio-token\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-55x7p\" (UID: \"52ba3acd-4de6-4ae2-8a20-49bf7f966ab0\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-55x7p" Oct 30 16:36:49.408012 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:49.407990 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7l9bl\" (UniqueName: \"kubernetes.io/projected/52ba3acd-4de6-4ae2-8a20-49bf7f966ab0-kube-api-access-7l9bl\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-55x7p\" (UID: \"52ba3acd-4de6-4ae2-8a20-49bf7f966ab0\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-55x7p" Oct 30 16:36:49.502611 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:49.501762 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-55x7p" Oct 30 16:36:49.584878 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:49.584823 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2slrc"] Oct 30 16:36:49.590399 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:36:49.589826 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69a65b89_4d81_476b_b4b7_06736537fa4a.slice/crio-08c518ba2b3f5eec81978e237b3b1210114de3d3f2803e177207f817235d04a8 WatchSource:0}: Error finding container 08c518ba2b3f5eec81978e237b3b1210114de3d3f2803e177207f817235d04a8: Status 404 returned error can't find the container with id 08c518ba2b3f5eec81978e237b3b1210114de3d3f2803e177207f817235d04a8 Oct 30 16:36:49.592273 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:49.592242 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:36:49.592330 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:49.592308 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:36:49.592365 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:49.592339 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:36:49.823512 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:49.823476 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2slrc" event={"ID":"69a65b89-4d81-476b-b4b7-06736537fa4a","Type":"ContainerStarted","Data":"653b37f0a36ca625c0e710ae2c5299203f8da7df81f26a3b6513447f5ef2d81f"} Oct 30 16:36:49.823952 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:49.823521 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2slrc" event={"ID":"69a65b89-4d81-476b-b4b7-06736537fa4a","Type":"ContainerStarted","Data":"08c518ba2b3f5eec81978e237b3b1210114de3d3f2803e177207f817235d04a8"} Oct 30 16:36:49.925682 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:49.925388 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-55x7p"] Oct 30 16:36:49.936622 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:49.928436 5052 patch_prober.go:29] interesting pod/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-bvwpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 30 16:36:49.936622 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:49.928484 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-bvwpz" podUID="e8eabbfd-4dce-4087-a350-c55cb7c7baf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 30 16:36:49.950796 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:49.939739 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:36:49.950796 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:49.939838 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:36:49.950796 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:49.939908 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:36:49.953092 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:49.953035 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2slrc" podStartSLOduration=1.953015143 podStartE2EDuration="1.953015143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:36:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:36:49.945624412 +0000 UTC m=+965.472959636" watchObservedRunningTime="2026-10-30 16:36:49.953015143 +0000 UTC m=+965.480350362" Oct 30 16:36:50.118396 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:50.118222 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2slrc" Oct 30 16:36:50.120305 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:50.120265 5052 patch_prober.go:29] interesting pod/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2slrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Oct 30 16:36:50.120401 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:50.120317 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2slrc" podUID="69a65b89-4d81-476b-b4b7-06736537fa4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Oct 30 16:36:50.394666 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:50.393382 5052 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-l6dw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 30 16:36:50.394666 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:50.393462 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-l6dw5" podUID="4b3c4341-826b-489d-81ae-14bf71548dd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 30 16:36:50.887607 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:50.883574 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-55x7p" event={"ID":"52ba3acd-4de6-4ae2-8a20-49bf7f966ab0","Type":"ContainerStarted","Data":"f6e0da38fce207d9afc7ba78924333b92ac36829e21634dccc8f63a00a430af3"} Oct 30 16:36:50.887607 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:50.883664 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-55x7p" event={"ID":"52ba3acd-4de6-4ae2-8a20-49bf7f966ab0","Type":"ContainerStarted","Data":"7a4f34ccca927aeb562bab8452a807c6e248cd8a012a113aac5c8b26e2ec2fdf"} Oct 30 16:36:50.929614 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:50.929007 5052 patch_prober.go:29] interesting pod/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-bvwpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 30 16:36:50.929614 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:50.929081 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-bvwpz" podUID="e8eabbfd-4dce-4087-a350-c55cb7c7baf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 30 16:36:50.946603 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:50.934267 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-55x7p" podStartSLOduration=1.9342478010000002 podStartE2EDuration="1.934247801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:36:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:36:50.927769616 +0000 UTC m=+966.455104836" watchObservedRunningTime="2026-10-30 16:36:50.934247801 +0000 UTC m=+966.461583020" Oct 30 16:36:51.122606 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:51.121980 5052 patch_prober.go:29] interesting pod/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2slrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Oct 30 16:36:51.122606 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:51.122044 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2slrc" podUID="69a65b89-4d81-476b-b4b7-06736537fa4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Oct 30 16:36:51.398607 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:51.398182 5052 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-l6dw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 30 16:36:51.398607 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:51.398276 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-l6dw5" podUID="4b3c4341-826b-489d-81ae-14bf71548dd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 30 16:36:51.512606 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:51.512555 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-55x7p" Oct 30 16:36:51.520655 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:51.516716 5052 patch_prober.go:29] interesting pod/e2e-worker-w4-263f00-openshift-default-b969d55bc-55x7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 16:36:51.520655 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:51.516784 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-55x7p" podUID="52ba3acd-4de6-4ae2-8a20-49bf7f966ab0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 16:36:51.973603 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:51.972941 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-bvwpz" Oct 30 16:36:51.986607 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:51.974445 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-bvwpz" Oct 30 16:36:52.018607 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:52.015936 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-bvwpz" Oct 30 16:36:52.118150 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:52.118104 5052 patch_prober.go:29] interesting pod/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2slrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Oct 30 16:36:52.118353 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:52.118192 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2slrc" podUID="69a65b89-4d81-476b-b4b7-06736537fa4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Oct 30 16:36:52.437611 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:52.425014 5052 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-l6dw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 30 16:36:52.437611 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:52.425076 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-l6dw5" podUID="4b3c4341-826b-489d-81ae-14bf71548dd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 30 16:36:52.502567 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:52.502447 5052 patch_prober.go:29] interesting pod/e2e-worker-w4-263f00-openshift-default-b969d55bc-55x7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 16:36:52.502975 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:52.502943 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-55x7p" podUID="52ba3acd-4de6-4ae2-8a20-49bf7f966ab0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 16:36:53.121641 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:53.119430 5052 patch_prober.go:29] interesting pod/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2slrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Oct 30 16:36:53.121641 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:53.119497 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2slrc" podUID="69a65b89-4d81-476b-b4b7-06736537fa4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Oct 30 16:36:53.411684 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:53.411648 5052 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-l6dw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 30 16:36:53.412009 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:53.411981 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-l6dw5" podUID="4b3c4341-826b-489d-81ae-14bf71548dd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 30 16:36:53.515345 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:53.515303 5052 patch_prober.go:29] interesting pod/e2e-worker-w4-263f00-openshift-default-b969d55bc-55x7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 16:36:53.515537 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:53.515368 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-55x7p" podUID="52ba3acd-4de6-4ae2-8a20-49bf7f966ab0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 16:36:54.118109 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:54.118073 5052 patch_prober.go:29] interesting pod/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2slrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Oct 30 16:36:54.118394 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:54.118362 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2slrc" podUID="69a65b89-4d81-476b-b4b7-06736537fa4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Oct 30 16:36:54.393102 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:54.392997 5052 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-l6dw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 30 16:36:54.393790 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:54.393753 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-l6dw5" podUID="4b3c4341-826b-489d-81ae-14bf71548dd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 30 16:36:54.507606 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:54.507076 5052 patch_prober.go:29] interesting pod/e2e-worker-w4-263f00-openshift-default-b969d55bc-55x7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 16:36:54.507606 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:54.507139 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-55x7p" podUID="52ba3acd-4de6-4ae2-8a20-49bf7f966ab0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 16:36:55.122606 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:55.118351 5052 patch_prober.go:29] interesting pod/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2slrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Oct 30 16:36:55.122606 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:55.118413 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2slrc" podUID="69a65b89-4d81-476b-b4b7-06736537fa4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Oct 30 16:36:55.396664 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:55.395979 5052 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-l6dw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 30 16:36:55.396664 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:55.396045 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-l6dw5" podUID="4b3c4341-826b-489d-81ae-14bf71548dd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 30 16:36:55.531620 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:55.526392 5052 patch_prober.go:29] interesting pod/e2e-worker-w4-263f00-openshift-default-b969d55bc-55x7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 16:36:55.531620 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:55.526456 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-55x7p" podUID="52ba3acd-4de6-4ae2-8a20-49bf7f966ab0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 16:36:56.120615 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:56.119051 5052 patch_prober.go:29] interesting pod/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2slrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Oct 30 16:36:56.120615 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:56.119124 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2slrc" podUID="69a65b89-4d81-476b-b4b7-06736537fa4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Oct 30 16:36:56.396844 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:56.396754 5052 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-l6dw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 30 16:36:56.397397 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:56.397365 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-l6dw5" podUID="4b3c4341-826b-489d-81ae-14bf71548dd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 30 16:36:56.503141 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:56.502497 5052 patch_prober.go:29] interesting pod/e2e-worker-w4-263f00-openshift-default-b969d55bc-55x7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 16:36:56.503141 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:56.502554 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-55x7p" podUID="52ba3acd-4de6-4ae2-8a20-49bf7f966ab0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 16:36:57.117778 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:57.117734 5052 patch_prober.go:29] interesting pod/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2slrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Oct 30 16:36:57.117988 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:57.117801 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2slrc" podUID="69a65b89-4d81-476b-b4b7-06736537fa4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Oct 30 16:36:57.502475 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:57.502437 5052 patch_prober.go:29] interesting pod/e2e-worker-w4-263f00-openshift-default-b969d55bc-55x7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 16:36:57.502992 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:57.502504 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-55x7p" podUID="52ba3acd-4de6-4ae2-8a20-49bf7f966ab0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 16:36:57.921339 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:57.921309 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-l6dw5" Oct 30 16:36:57.922149 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:57.922120 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-l6dw5" Oct 30 16:36:57.924106 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:57.924093 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-l6dw5" Oct 30 16:36:58.122616 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:58.121820 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2slrc" Oct 30 16:36:58.122616 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:58.122271 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2slrc" Oct 30 16:36:58.122912 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:58.122895 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2slrc" Oct 30 16:36:59.068387 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:59.068346 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-55x7p" Oct 30 16:36:59.069794 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:59.069769 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-55x7p" Oct 30 16:36:59.080913 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:36:59.080885 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-55x7p" Oct 30 16:37:04.820405 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:04.820371 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cdwcj"] Oct 30 16:37:18.841146 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:37:18.841097 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 16:37:19.022750 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:19.022715 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-fvfrl"] Oct 30 16:37:19.026618 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:19.026569 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-fvfrl" Oct 30 16:37:19.052100 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:19.052068 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-fvfrl"] Oct 30 16:37:19.184144 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:19.184102 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc4d11ae-edd2-4143-95fd-b57bf4486e76-istiod-ca-cert\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-fvfrl\" (UID: \"bc4d11ae-edd2-4143-95fd-b57bf4486e76\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-fvfrl" Oct 30 16:37:19.184328 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:19.184192 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc4d11ae-edd2-4143-95fd-b57bf4486e76-istio-token\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-fvfrl\" (UID: \"bc4d11ae-edd2-4143-95fd-b57bf4486e76\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-fvfrl" Oct 30 16:37:19.184328 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:19.184270 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc4d11ae-edd2-4143-95fd-b57bf4486e76-workload-certs\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-fvfrl\" (UID: \"bc4d11ae-edd2-4143-95fd-b57bf4486e76\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-fvfrl" Oct 30 16:37:19.184328 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:19.184301 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc4d11ae-edd2-4143-95fd-b57bf4486e76-istio-envoy\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-fvfrl\" (UID: \"bc4d11ae-edd2-4143-95fd-b57bf4486e76\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-fvfrl" Oct 30 16:37:19.184328 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:19.184322 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbhpg\" (UniqueName: \"kubernetes.io/projected/bc4d11ae-edd2-4143-95fd-b57bf4486e76-kube-api-access-jbhpg\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-fvfrl\" (UID: \"bc4d11ae-edd2-4143-95fd-b57bf4486e76\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-fvfrl" Oct 30 16:37:19.184466 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:19.184417 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc4d11ae-edd2-4143-95fd-b57bf4486e76-istio-podinfo\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-fvfrl\" (UID: \"bc4d11ae-edd2-4143-95fd-b57bf4486e76\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-fvfrl" Oct 30 16:37:19.184546 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:19.184522 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc4d11ae-edd2-4143-95fd-b57bf4486e76-credential-socket\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-fvfrl\" (UID: \"bc4d11ae-edd2-4143-95fd-b57bf4486e76\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-fvfrl" Oct 30 16:37:19.184636 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:19.184618 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc4d11ae-edd2-4143-95fd-b57bf4486e76-workload-socket\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-fvfrl\" (UID: \"bc4d11ae-edd2-4143-95fd-b57bf4486e76\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-fvfrl" Oct 30 16:37:19.184678 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:19.184660 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc4d11ae-edd2-4143-95fd-b57bf4486e76-istio-data\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-fvfrl\" (UID: \"bc4d11ae-edd2-4143-95fd-b57bf4486e76\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-fvfrl" Oct 30 16:37:19.285263 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:19.285236 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc4d11ae-edd2-4143-95fd-b57bf4486e76-workload-socket\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-fvfrl\" (UID: \"bc4d11ae-edd2-4143-95fd-b57bf4486e76\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-fvfrl" Oct 30 16:37:19.285263 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:19.285266 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc4d11ae-edd2-4143-95fd-b57bf4486e76-istio-data\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-fvfrl\" (UID: \"bc4d11ae-edd2-4143-95fd-b57bf4486e76\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-fvfrl" Oct 30 16:37:19.285444 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:19.285301 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc4d11ae-edd2-4143-95fd-b57bf4486e76-istiod-ca-cert\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-fvfrl\" (UID: \"bc4d11ae-edd2-4143-95fd-b57bf4486e76\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-fvfrl" Oct 30 16:37:19.285444 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:19.285359 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc4d11ae-edd2-4143-95fd-b57bf4486e76-istio-token\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-fvfrl\" (UID: \"bc4d11ae-edd2-4143-95fd-b57bf4486e76\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-fvfrl" Oct 30 16:37:19.285524 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:19.285498 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc4d11ae-edd2-4143-95fd-b57bf4486e76-workload-certs\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-fvfrl\" (UID: \"bc4d11ae-edd2-4143-95fd-b57bf4486e76\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-fvfrl" Oct 30 16:37:19.285605 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:19.285566 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc4d11ae-edd2-4143-95fd-b57bf4486e76-istio-envoy\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-fvfrl\" (UID: \"bc4d11ae-edd2-4143-95fd-b57bf4486e76\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-fvfrl" Oct 30 16:37:19.285673 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:19.285623 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbhpg\" (UniqueName: \"kubernetes.io/projected/bc4d11ae-edd2-4143-95fd-b57bf4486e76-kube-api-access-jbhpg\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-fvfrl\" (UID: \"bc4d11ae-edd2-4143-95fd-b57bf4486e76\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-fvfrl" Oct 30 16:37:19.285673 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:19.285659 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc4d11ae-edd2-4143-95fd-b57bf4486e76-istio-podinfo\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-fvfrl\" (UID: \"bc4d11ae-edd2-4143-95fd-b57bf4486e76\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-fvfrl" Oct 30 16:37:19.285947 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:19.285672 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc4d11ae-edd2-4143-95fd-b57bf4486e76-istio-data\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-fvfrl\" (UID: \"bc4d11ae-edd2-4143-95fd-b57bf4486e76\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-fvfrl" Oct 30 16:37:19.285947 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:19.285704 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc4d11ae-edd2-4143-95fd-b57bf4486e76-workload-socket\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-fvfrl\" (UID: \"bc4d11ae-edd2-4143-95fd-b57bf4486e76\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-fvfrl" Oct 30 16:37:19.285947 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:19.285760 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc4d11ae-edd2-4143-95fd-b57bf4486e76-credential-socket\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-fvfrl\" (UID: \"bc4d11ae-edd2-4143-95fd-b57bf4486e76\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-fvfrl" Oct 30 16:37:19.285947 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:19.285925 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc4d11ae-edd2-4143-95fd-b57bf4486e76-workload-certs\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-fvfrl\" (UID: \"bc4d11ae-edd2-4143-95fd-b57bf4486e76\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-fvfrl" Oct 30 16:37:19.286140 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:19.286115 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc4d11ae-edd2-4143-95fd-b57bf4486e76-istiod-ca-cert\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-fvfrl\" (UID: \"bc4d11ae-edd2-4143-95fd-b57bf4486e76\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-fvfrl" Oct 30 16:37:19.286250 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:19.286230 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc4d11ae-edd2-4143-95fd-b57bf4486e76-credential-socket\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-fvfrl\" (UID: \"bc4d11ae-edd2-4143-95fd-b57bf4486e76\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-fvfrl" Oct 30 16:37:19.287844 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:19.287825 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc4d11ae-edd2-4143-95fd-b57bf4486e76-istio-envoy\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-fvfrl\" (UID: \"bc4d11ae-edd2-4143-95fd-b57bf4486e76\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-fvfrl" Oct 30 16:37:19.288059 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:19.288040 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc4d11ae-edd2-4143-95fd-b57bf4486e76-istio-podinfo\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-fvfrl\" (UID: \"bc4d11ae-edd2-4143-95fd-b57bf4486e76\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-fvfrl" Oct 30 16:37:19.308436 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:19.308416 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc4d11ae-edd2-4143-95fd-b57bf4486e76-istio-token\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-fvfrl\" (UID: \"bc4d11ae-edd2-4143-95fd-b57bf4486e76\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-fvfrl" Oct 30 16:37:19.308652 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:19.308633 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbhpg\" (UniqueName: \"kubernetes.io/projected/bc4d11ae-edd2-4143-95fd-b57bf4486e76-kube-api-access-jbhpg\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-fvfrl\" (UID: \"bc4d11ae-edd2-4143-95fd-b57bf4486e76\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-fvfrl" Oct 30 16:37:19.341457 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:19.341431 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-fvfrl" Oct 30 16:37:19.660331 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:19.660293 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-fvfrl"] Oct 30 16:37:19.663639 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:37:19.663615 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc4d11ae_edd2_4143_95fd_b57bf4486e76.slice/crio-62fcf89e01fbbf9036289232ab19a8ccd9c13583951d8e27583c0bd75c1ba97d WatchSource:0}: Error finding container 62fcf89e01fbbf9036289232ab19a8ccd9c13583951d8e27583c0bd75c1ba97d: Status 404 returned error can't find the container with id 62fcf89e01fbbf9036289232ab19a8ccd9c13583951d8e27583c0bd75c1ba97d Oct 30 16:37:19.666037 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:19.666007 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:37:19.666108 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:19.666082 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:37:19.666145 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:19.666110 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:37:20.398236 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:20.398193 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-fvfrl" event={"ID":"bc4d11ae-edd2-4143-95fd-b57bf4486e76","Type":"ContainerStarted","Data":"64186719511d27956a6d28fbb00d1f613a2086dc379cddd7f08590bdee8afc9c"} Oct 30 16:37:20.398728 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:20.398260 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-fvfrl" event={"ID":"bc4d11ae-edd2-4143-95fd-b57bf4486e76","Type":"ContainerStarted","Data":"62fcf89e01fbbf9036289232ab19a8ccd9c13583951d8e27583c0bd75c1ba97d"} Oct 30 16:37:21.344761 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:21.343638 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-fvfrl" Oct 30 16:37:21.360604 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:21.347708 5052 patch_prober.go:29] interesting pod/e2e-worker-w5-967fac-openshift-default-dffc67487-fvfrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 16:37:21.360604 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:21.347768 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-fvfrl" podUID="bc4d11ae-edd2-4143-95fd-b57bf4486e76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 16:37:21.500412 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:21.500354 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-fvfrl" podStartSLOduration=3.500335233 podStartE2EDuration="3.500335233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:37:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:37:20.459924104 +0000 UTC m=+995.987259323" watchObservedRunningTime="2026-10-30 16:37:21.500335233 +0000 UTC m=+997.027670453" Oct 30 16:37:21.503257 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:21.503228 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-967fac/distinct-w5-967fac-kserve-69cbcccd8b-b85bn"] Oct 30 16:37:21.508624 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:21.508603 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-967fac/distinct-w5-967fac-kserve-69cbcccd8b-b85bn" Oct 30 16:37:21.542515 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:21.539708 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-967fac/distinct-w5-967fac-kserve-69cbcccd8b-b85bn"] Oct 30 16:37:21.542515 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:21.540572 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-967fac\"/\"distinct-w5-967fac-kserve-self-signed-certs\"" Oct 30 16:37:21.542515 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:21.540728 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-967fac\"/\"default-dockercfg-crb6m\"" Oct 30 16:37:21.542515 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:21.541671 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-967fac\"/\"openshift-service-ca.crt\"" Oct 30 16:37:21.542515 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:21.541746 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-967fac\"/\"kube-root-ca.crt\"" Oct 30 16:37:21.630460 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:21.630362 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f6b9354-378f-4451-9ff7-be7f7fec12f2-tmp-dir\") pod \"distinct-w5-967fac-kserve-69cbcccd8b-b85bn\" (UID: \"8f6b9354-378f-4451-9ff7-be7f7fec12f2\") " pod="e2e-models-e2e-worker-w5-967fac/distinct-w5-967fac-kserve-69cbcccd8b-b85bn" Oct 30 16:37:21.640031 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:21.640005 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f6b9354-378f-4451-9ff7-be7f7fec12f2-home\") pod \"distinct-w5-967fac-kserve-69cbcccd8b-b85bn\" (UID: \"8f6b9354-378f-4451-9ff7-be7f7fec12f2\") " pod="e2e-models-e2e-worker-w5-967fac/distinct-w5-967fac-kserve-69cbcccd8b-b85bn" Oct 30 16:37:21.640325 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:21.640305 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f6b9354-378f-4451-9ff7-be7f7fec12f2-tls-certs\") pod \"distinct-w5-967fac-kserve-69cbcccd8b-b85bn\" (UID: \"8f6b9354-378f-4451-9ff7-be7f7fec12f2\") " pod="e2e-models-e2e-worker-w5-967fac/distinct-w5-967fac-kserve-69cbcccd8b-b85bn" Oct 30 16:37:21.647665 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:21.647640 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f6b9354-378f-4451-9ff7-be7f7fec12f2-model-cache\") pod \"distinct-w5-967fac-kserve-69cbcccd8b-b85bn\" (UID: \"8f6b9354-378f-4451-9ff7-be7f7fec12f2\") " pod="e2e-models-e2e-worker-w5-967fac/distinct-w5-967fac-kserve-69cbcccd8b-b85bn" Oct 30 16:37:21.647927 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:21.647907 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f6b9354-378f-4451-9ff7-be7f7fec12f2-dshm\") pod \"distinct-w5-967fac-kserve-69cbcccd8b-b85bn\" (UID: \"8f6b9354-378f-4451-9ff7-be7f7fec12f2\") " pod="e2e-models-e2e-worker-w5-967fac/distinct-w5-967fac-kserve-69cbcccd8b-b85bn" Oct 30 16:37:21.648115 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:21.648096 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5g7b2\" (UniqueName: \"kubernetes.io/projected/8f6b9354-378f-4451-9ff7-be7f7fec12f2-kube-api-access-5g7b2\") pod \"distinct-w5-967fac-kserve-69cbcccd8b-b85bn\" (UID: \"8f6b9354-378f-4451-9ff7-be7f7fec12f2\") " pod="e2e-models-e2e-worker-w5-967fac/distinct-w5-967fac-kserve-69cbcccd8b-b85bn" Oct 30 16:37:21.752751 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:21.752487 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f6b9354-378f-4451-9ff7-be7f7fec12f2-model-cache\") pod \"distinct-w5-967fac-kserve-69cbcccd8b-b85bn\" (UID: \"8f6b9354-378f-4451-9ff7-be7f7fec12f2\") " pod="e2e-models-e2e-worker-w5-967fac/distinct-w5-967fac-kserve-69cbcccd8b-b85bn" Oct 30 16:37:21.752751 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:21.752565 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f6b9354-378f-4451-9ff7-be7f7fec12f2-dshm\") pod \"distinct-w5-967fac-kserve-69cbcccd8b-b85bn\" (UID: \"8f6b9354-378f-4451-9ff7-be7f7fec12f2\") " pod="e2e-models-e2e-worker-w5-967fac/distinct-w5-967fac-kserve-69cbcccd8b-b85bn" Oct 30 16:37:21.752751 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:21.752732 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5g7b2\" (UniqueName: \"kubernetes.io/projected/8f6b9354-378f-4451-9ff7-be7f7fec12f2-kube-api-access-5g7b2\") pod \"distinct-w5-967fac-kserve-69cbcccd8b-b85bn\" (UID: \"8f6b9354-378f-4451-9ff7-be7f7fec12f2\") " pod="e2e-models-e2e-worker-w5-967fac/distinct-w5-967fac-kserve-69cbcccd8b-b85bn" Oct 30 16:37:21.753044 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:21.752868 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f6b9354-378f-4451-9ff7-be7f7fec12f2-tmp-dir\") pod \"distinct-w5-967fac-kserve-69cbcccd8b-b85bn\" (UID: \"8f6b9354-378f-4451-9ff7-be7f7fec12f2\") " pod="e2e-models-e2e-worker-w5-967fac/distinct-w5-967fac-kserve-69cbcccd8b-b85bn" Oct 30 16:37:21.753044 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:21.752969 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f6b9354-378f-4451-9ff7-be7f7fec12f2-home\") pod \"distinct-w5-967fac-kserve-69cbcccd8b-b85bn\" (UID: \"8f6b9354-378f-4451-9ff7-be7f7fec12f2\") " pod="e2e-models-e2e-worker-w5-967fac/distinct-w5-967fac-kserve-69cbcccd8b-b85bn" Oct 30 16:37:21.753166 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:21.753098 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f6b9354-378f-4451-9ff7-be7f7fec12f2-tls-certs\") pod \"distinct-w5-967fac-kserve-69cbcccd8b-b85bn\" (UID: \"8f6b9354-378f-4451-9ff7-be7f7fec12f2\") " pod="e2e-models-e2e-worker-w5-967fac/distinct-w5-967fac-kserve-69cbcccd8b-b85bn" Oct 30 16:37:21.760964 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:21.760935 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f6b9354-378f-4451-9ff7-be7f7fec12f2-model-cache\") pod \"distinct-w5-967fac-kserve-69cbcccd8b-b85bn\" (UID: \"8f6b9354-378f-4451-9ff7-be7f7fec12f2\") " pod="e2e-models-e2e-worker-w5-967fac/distinct-w5-967fac-kserve-69cbcccd8b-b85bn" Oct 30 16:37:21.766610 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:21.764884 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f6b9354-378f-4451-9ff7-be7f7fec12f2-home\") pod \"distinct-w5-967fac-kserve-69cbcccd8b-b85bn\" (UID: \"8f6b9354-378f-4451-9ff7-be7f7fec12f2\") " pod="e2e-models-e2e-worker-w5-967fac/distinct-w5-967fac-kserve-69cbcccd8b-b85bn" Oct 30 16:37:21.767082 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:21.767034 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f6b9354-378f-4451-9ff7-be7f7fec12f2-dshm\") pod \"distinct-w5-967fac-kserve-69cbcccd8b-b85bn\" (UID: \"8f6b9354-378f-4451-9ff7-be7f7fec12f2\") " pod="e2e-models-e2e-worker-w5-967fac/distinct-w5-967fac-kserve-69cbcccd8b-b85bn" Oct 30 16:37:21.779606 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:21.779298 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f6b9354-378f-4451-9ff7-be7f7fec12f2-tls-certs\") pod \"distinct-w5-967fac-kserve-69cbcccd8b-b85bn\" (UID: \"8f6b9354-378f-4451-9ff7-be7f7fec12f2\") " pod="e2e-models-e2e-worker-w5-967fac/distinct-w5-967fac-kserve-69cbcccd8b-b85bn" Oct 30 16:37:21.791607 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:21.783760 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f6b9354-378f-4451-9ff7-be7f7fec12f2-tmp-dir\") pod \"distinct-w5-967fac-kserve-69cbcccd8b-b85bn\" (UID: \"8f6b9354-378f-4451-9ff7-be7f7fec12f2\") " pod="e2e-models-e2e-worker-w5-967fac/distinct-w5-967fac-kserve-69cbcccd8b-b85bn" Oct 30 16:37:21.791607 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:21.790232 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5g7b2\" (UniqueName: \"kubernetes.io/projected/8f6b9354-378f-4451-9ff7-be7f7fec12f2-kube-api-access-5g7b2\") pod \"distinct-w5-967fac-kserve-69cbcccd8b-b85bn\" (UID: \"8f6b9354-378f-4451-9ff7-be7f7fec12f2\") " pod="e2e-models-e2e-worker-w5-967fac/distinct-w5-967fac-kserve-69cbcccd8b-b85bn" Oct 30 16:37:21.858607 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:21.852919 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-967fac/distinct-w5-967fac-kserve-69cbcccd8b-b85bn" Oct 30 16:37:22.381606 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:22.345073 5052 patch_prober.go:29] interesting pod/e2e-worker-w5-967fac-openshift-default-dffc67487-fvfrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 16:37:22.381606 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:22.345132 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-fvfrl" podUID="bc4d11ae-edd2-4143-95fd-b57bf4486e76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 16:37:22.419733 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:22.419704 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-967fac/distinct-w5-967fac-kserve-69cbcccd8b-b85bn"] Oct 30 16:37:23.349035 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:23.348992 5052 patch_prober.go:29] interesting pod/e2e-worker-w5-967fac-openshift-default-dffc67487-fvfrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 16:37:23.349676 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:23.349641 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-fvfrl" podUID="bc4d11ae-edd2-4143-95fd-b57bf4486e76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 16:37:23.429338 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:23.429291 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-967fac/distinct-w5-967fac-kserve-69cbcccd8b-b85bn" event={"ID":"8f6b9354-378f-4451-9ff7-be7f7fec12f2","Type":"ContainerStarted","Data":"9494a34003b003d9843e8c7adfb410a88e495c6f69d79d636b12d3a550be0fba"} Oct 30 16:37:23.433654 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:23.433611 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-967fac/distinct-w5-967fac-kserve-69cbcccd8b-b85bn" event={"ID":"8f6b9354-378f-4451-9ff7-be7f7fec12f2","Type":"ContainerStarted","Data":"e9d2fd09af0ecc8dbede5c3ad921446c488ec3268a14e9a13fe8dc0c6d5c8b78"} Oct 30 16:37:23.476620 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:23.476528 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-967fac/distinct-w5-967fac-kserve-69cbcccd8b-b85bn" podStartSLOduration=2.476509461 podStartE2EDuration="2.476509461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:37:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:37:23.475684806 +0000 UTC m=+999.003020026" watchObservedRunningTime="2026-10-30 16:37:23.476509461 +0000 UTC m=+999.003844681" Oct 30 16:37:23.857862 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:23.857822 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-967fac/distinct-w5-967fac-kserve-69cbcccd8b-b85bn" Oct 30 16:37:24.357720 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:24.357684 5052 patch_prober.go:29] interesting pod/e2e-worker-w5-967fac-openshift-default-dffc67487-fvfrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 16:37:24.358708 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:24.358663 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-fvfrl" podUID="bc4d11ae-edd2-4143-95fd-b57bf4486e76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 16:37:25.352633 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:25.350680 5052 patch_prober.go:29] interesting pod/e2e-worker-w5-967fac-openshift-default-dffc67487-fvfrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 16:37:25.352633 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:25.350737 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-fvfrl" podUID="bc4d11ae-edd2-4143-95fd-b57bf4486e76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 16:37:27.365424 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:27.365368 5052 patch_prober.go:29] interesting pod/e2e-worker-w5-967fac-openshift-default-dffc67487-fvfrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:37:27.365910 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:27.365451 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-fvfrl" podUID="bc4d11ae-edd2-4143-95fd-b57bf4486e76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:27.387926 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:27.387896 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-fvfrl" Oct 30 16:37:27.388398 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:27.388375 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-fvfrl" Oct 30 16:37:27.389340 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:27.389321 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-fvfrl" Oct 30 16:37:27.860272 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:27.860242 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-263f00/distinct-2-w4-263f00-kserve-8464897cf5-q8ck8"] Oct 30 16:37:27.869148 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:27.869123 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-263f00/distinct-2-w4-263f00-kserve-8464897cf5-q8ck8" Oct 30 16:37:27.884133 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:27.884109 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-263f00\"/\"openshift-service-ca.crt\"" Oct 30 16:37:27.884330 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:27.884314 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-263f00\"/\"kube-root-ca.crt\"" Oct 30 16:37:27.884979 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:27.884957 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-263f00/distinct-2-w4-263f00-kserve-8464897cf5-q8ck8"] Oct 30 16:37:27.892311 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:27.892293 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-263f00\"/\"default-dockercfg-b54g7\"" Oct 30 16:37:27.892426 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:27.892364 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-263f00\"/\"distinct-2-w4-263f00-kserve-self-signed-certs\"" Oct 30 16:37:27.939004 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:27.938972 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15ed43ef-5cdd-489a-a70e-063239822ff3-dshm\") pod \"distinct-2-w4-263f00-kserve-8464897cf5-q8ck8\" (UID: \"15ed43ef-5cdd-489a-a70e-063239822ff3\") " pod="e2e-models-e2e-worker-w4-263f00/distinct-2-w4-263f00-kserve-8464897cf5-q8ck8" Oct 30 16:37:27.939130 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:27.939110 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15ed43ef-5cdd-489a-a70e-063239822ff3-tmp-dir\") pod \"distinct-2-w4-263f00-kserve-8464897cf5-q8ck8\" (UID: \"15ed43ef-5cdd-489a-a70e-063239822ff3\") " pod="e2e-models-e2e-worker-w4-263f00/distinct-2-w4-263f00-kserve-8464897cf5-q8ck8" Oct 30 16:37:27.939177 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:27.939161 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15ed43ef-5cdd-489a-a70e-063239822ff3-tls-certs\") pod \"distinct-2-w4-263f00-kserve-8464897cf5-q8ck8\" (UID: \"15ed43ef-5cdd-489a-a70e-063239822ff3\") " pod="e2e-models-e2e-worker-w4-263f00/distinct-2-w4-263f00-kserve-8464897cf5-q8ck8" Oct 30 16:37:27.939244 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:27.939231 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15ed43ef-5cdd-489a-a70e-063239822ff3-home\") pod \"distinct-2-w4-263f00-kserve-8464897cf5-q8ck8\" (UID: \"15ed43ef-5cdd-489a-a70e-063239822ff3\") " pod="e2e-models-e2e-worker-w4-263f00/distinct-2-w4-263f00-kserve-8464897cf5-q8ck8" Oct 30 16:37:27.939288 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:27.939277 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fx2b7\" (UniqueName: \"kubernetes.io/projected/15ed43ef-5cdd-489a-a70e-063239822ff3-kube-api-access-fx2b7\") pod \"distinct-2-w4-263f00-kserve-8464897cf5-q8ck8\" (UID: \"15ed43ef-5cdd-489a-a70e-063239822ff3\") " pod="e2e-models-e2e-worker-w4-263f00/distinct-2-w4-263f00-kserve-8464897cf5-q8ck8" Oct 30 16:37:27.939370 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:27.939357 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15ed43ef-5cdd-489a-a70e-063239822ff3-model-cache\") pod \"distinct-2-w4-263f00-kserve-8464897cf5-q8ck8\" (UID: \"15ed43ef-5cdd-489a-a70e-063239822ff3\") " pod="e2e-models-e2e-worker-w4-263f00/distinct-2-w4-263f00-kserve-8464897cf5-q8ck8" Oct 30 16:37:28.052604 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:28.044277 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fx2b7\" (UniqueName: \"kubernetes.io/projected/15ed43ef-5cdd-489a-a70e-063239822ff3-kube-api-access-fx2b7\") pod \"distinct-2-w4-263f00-kserve-8464897cf5-q8ck8\" (UID: \"15ed43ef-5cdd-489a-a70e-063239822ff3\") " pod="e2e-models-e2e-worker-w4-263f00/distinct-2-w4-263f00-kserve-8464897cf5-q8ck8" Oct 30 16:37:28.061314 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:28.061278 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15ed43ef-5cdd-489a-a70e-063239822ff3-model-cache\") pod \"distinct-2-w4-263f00-kserve-8464897cf5-q8ck8\" (UID: \"15ed43ef-5cdd-489a-a70e-063239822ff3\") " pod="e2e-models-e2e-worker-w4-263f00/distinct-2-w4-263f00-kserve-8464897cf5-q8ck8" Oct 30 16:37:28.061732 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:28.061708 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15ed43ef-5cdd-489a-a70e-063239822ff3-dshm\") pod \"distinct-2-w4-263f00-kserve-8464897cf5-q8ck8\" (UID: \"15ed43ef-5cdd-489a-a70e-063239822ff3\") " pod="e2e-models-e2e-worker-w4-263f00/distinct-2-w4-263f00-kserve-8464897cf5-q8ck8" Oct 30 16:37:28.062005 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:28.061978 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15ed43ef-5cdd-489a-a70e-063239822ff3-tmp-dir\") pod \"distinct-2-w4-263f00-kserve-8464897cf5-q8ck8\" (UID: \"15ed43ef-5cdd-489a-a70e-063239822ff3\") " pod="e2e-models-e2e-worker-w4-263f00/distinct-2-w4-263f00-kserve-8464897cf5-q8ck8" Oct 30 16:37:28.062152 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:28.062026 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15ed43ef-5cdd-489a-a70e-063239822ff3-tls-certs\") pod \"distinct-2-w4-263f00-kserve-8464897cf5-q8ck8\" (UID: \"15ed43ef-5cdd-489a-a70e-063239822ff3\") " pod="e2e-models-e2e-worker-w4-263f00/distinct-2-w4-263f00-kserve-8464897cf5-q8ck8" Oct 30 16:37:28.062152 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:28.062107 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15ed43ef-5cdd-489a-a70e-063239822ff3-home\") pod \"distinct-2-w4-263f00-kserve-8464897cf5-q8ck8\" (UID: \"15ed43ef-5cdd-489a-a70e-063239822ff3\") " pod="e2e-models-e2e-worker-w4-263f00/distinct-2-w4-263f00-kserve-8464897cf5-q8ck8" Oct 30 16:37:28.064604 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:28.062639 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15ed43ef-5cdd-489a-a70e-063239822ff3-home\") pod \"distinct-2-w4-263f00-kserve-8464897cf5-q8ck8\" (UID: \"15ed43ef-5cdd-489a-a70e-063239822ff3\") " pod="e2e-models-e2e-worker-w4-263f00/distinct-2-w4-263f00-kserve-8464897cf5-q8ck8" Oct 30 16:37:28.064604 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:28.062900 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15ed43ef-5cdd-489a-a70e-063239822ff3-model-cache\") pod \"distinct-2-w4-263f00-kserve-8464897cf5-q8ck8\" (UID: \"15ed43ef-5cdd-489a-a70e-063239822ff3\") " pod="e2e-models-e2e-worker-w4-263f00/distinct-2-w4-263f00-kserve-8464897cf5-q8ck8" Oct 30 16:37:28.064604 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:28.063472 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15ed43ef-5cdd-489a-a70e-063239822ff3-tmp-dir\") pod \"distinct-2-w4-263f00-kserve-8464897cf5-q8ck8\" (UID: \"15ed43ef-5cdd-489a-a70e-063239822ff3\") " pod="e2e-models-e2e-worker-w4-263f00/distinct-2-w4-263f00-kserve-8464897cf5-q8ck8" Oct 30 16:37:28.118651 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:28.117023 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15ed43ef-5cdd-489a-a70e-063239822ff3-dshm\") pod \"distinct-2-w4-263f00-kserve-8464897cf5-q8ck8\" (UID: \"15ed43ef-5cdd-489a-a70e-063239822ff3\") " pod="e2e-models-e2e-worker-w4-263f00/distinct-2-w4-263f00-kserve-8464897cf5-q8ck8" Oct 30 16:37:28.122470 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:28.120840 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15ed43ef-5cdd-489a-a70e-063239822ff3-tls-certs\") pod \"distinct-2-w4-263f00-kserve-8464897cf5-q8ck8\" (UID: \"15ed43ef-5cdd-489a-a70e-063239822ff3\") " pod="e2e-models-e2e-worker-w4-263f00/distinct-2-w4-263f00-kserve-8464897cf5-q8ck8" Oct 30 16:37:28.135113 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:28.132097 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fx2b7\" (UniqueName: \"kubernetes.io/projected/15ed43ef-5cdd-489a-a70e-063239822ff3-kube-api-access-fx2b7\") pod \"distinct-2-w4-263f00-kserve-8464897cf5-q8ck8\" (UID: \"15ed43ef-5cdd-489a-a70e-063239822ff3\") " pod="e2e-models-e2e-worker-w4-263f00/distinct-2-w4-263f00-kserve-8464897cf5-q8ck8" Oct 30 16:37:28.186056 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:28.186023 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-263f00/distinct-2-w4-263f00-kserve-8464897cf5-q8ck8" Oct 30 16:37:28.496073 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:28.495985 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-263f00/distinct-2-w4-263f00-kserve-8464897cf5-q8ck8"] Oct 30 16:37:28.499383 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:37:28.499357 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod15ed43ef_5cdd_489a_a70e_063239822ff3.slice/crio-638469e57a3882f5a49934f8851d1fb748ac6f241ee08e4b2c7878637a053529 WatchSource:0}: Error finding container 638469e57a3882f5a49934f8851d1fb748ac6f241ee08e4b2c7878637a053529: Status 404 returned error can't find the container with id 638469e57a3882f5a49934f8851d1fb748ac6f241ee08e4b2c7878637a053529 Oct 30 16:37:29.488869 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:29.488830 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-263f00/distinct-2-w4-263f00-kserve-8464897cf5-q8ck8" event={"ID":"15ed43ef-5cdd-489a-a70e-063239822ff3","Type":"ContainerStarted","Data":"b44b08923a9939b80b5c8728022b13331d54380e8c8f77e8a0de2063f821bc3b"} Oct 30 16:37:29.489044 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:29.488878 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-263f00/distinct-2-w4-263f00-kserve-8464897cf5-q8ck8" event={"ID":"15ed43ef-5cdd-489a-a70e-063239822ff3","Type":"ContainerStarted","Data":"638469e57a3882f5a49934f8851d1fb748ac6f241ee08e4b2c7878637a053529"} Oct 30 16:37:29.531095 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:29.531048 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-263f00/distinct-2-w4-263f00-kserve-8464897cf5-q8ck8" podStartSLOduration=2.531034385 podStartE2EDuration="2.531034385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:37:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:37:29.528611288 +0000 UTC m=+1005.055946502" watchObservedRunningTime="2026-10-30 16:37:29.531034385 +0000 UTC m=+1005.058369604" Oct 30 16:37:30.186734 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:30.186686 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-263f00/distinct-2-w4-263f00-kserve-8464897cf5-q8ck8" Oct 30 16:37:31.853826 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:31.853793 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-967fac/distinct-w5-967fac-kserve-69cbcccd8b-b85bn" Oct 30 16:37:31.866743 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:31.866718 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-967fac/distinct-w5-967fac-kserve-69cbcccd8b-b85bn" Oct 30 16:37:32.519802 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:32.519771 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-967fac/distinct-w5-967fac-kserve-69cbcccd8b-b85bn" Oct 30 16:37:35.258282 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:35.258245 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cdwcj"] Oct 30 16:37:38.189602 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:38.189108 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-263f00/distinct-2-w4-263f00-kserve-8464897cf5-q8ck8" Oct 30 16:37:38.221601 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:38.220629 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-263f00/distinct-2-w4-263f00-kserve-8464897cf5-q8ck8" Oct 30 16:37:38.600573 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:38.600496 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-263f00/distinct-2-w4-263f00-kserve-8464897cf5-q8ck8" Oct 30 16:37:43.125909 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:43.125834 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-967fac/distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx"] Oct 30 16:37:43.131371 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:43.131347 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-967fac/distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx" Oct 30 16:37:43.153115 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:43.153084 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-967fac/distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx"] Oct 30 16:37:43.160101 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:43.160076 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-967fac\"/\"distinct-2-w5-967fac-kserve-self-signed-certs\"" Oct 30 16:37:43.230976 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:43.230946 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd662f3a-659f-477c-b8d5-94c64288c0c1-home\") pod \"distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx\" (UID: \"bd662f3a-659f-477c-b8d5-94c64288c0c1\") " pod="e2e-models-e2e-worker-w5-967fac/distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx" Oct 30 16:37:43.230976 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:43.230982 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd662f3a-659f-477c-b8d5-94c64288c0c1-tmp-dir\") pod \"distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx\" (UID: \"bd662f3a-659f-477c-b8d5-94c64288c0c1\") " pod="e2e-models-e2e-worker-w5-967fac/distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx" Oct 30 16:37:43.231182 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:43.231071 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd662f3a-659f-477c-b8d5-94c64288c0c1-tls-certs\") pod \"distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx\" (UID: \"bd662f3a-659f-477c-b8d5-94c64288c0c1\") " pod="e2e-models-e2e-worker-w5-967fac/distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx" Oct 30 16:37:43.231182 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:43.231129 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd662f3a-659f-477c-b8d5-94c64288c0c1-model-cache\") pod \"distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx\" (UID: \"bd662f3a-659f-477c-b8d5-94c64288c0c1\") " pod="e2e-models-e2e-worker-w5-967fac/distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx" Oct 30 16:37:43.231248 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:43.231205 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkjvr\" (UniqueName: \"kubernetes.io/projected/bd662f3a-659f-477c-b8d5-94c64288c0c1-kube-api-access-vkjvr\") pod \"distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx\" (UID: \"bd662f3a-659f-477c-b8d5-94c64288c0c1\") " pod="e2e-models-e2e-worker-w5-967fac/distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx" Oct 30 16:37:43.231282 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:43.231256 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd662f3a-659f-477c-b8d5-94c64288c0c1-dshm\") pod \"distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx\" (UID: \"bd662f3a-659f-477c-b8d5-94c64288c0c1\") " pod="e2e-models-e2e-worker-w5-967fac/distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx" Oct 30 16:37:43.370610 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:43.355685 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd662f3a-659f-477c-b8d5-94c64288c0c1-home\") pod \"distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx\" (UID: \"bd662f3a-659f-477c-b8d5-94c64288c0c1\") " pod="e2e-models-e2e-worker-w5-967fac/distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx" Oct 30 16:37:43.371965 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:43.363185 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd662f3a-659f-477c-b8d5-94c64288c0c1-home\") pod \"distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx\" (UID: \"bd662f3a-659f-477c-b8d5-94c64288c0c1\") " pod="e2e-models-e2e-worker-w5-967fac/distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx" Oct 30 16:37:43.371965 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:43.370886 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd662f3a-659f-477c-b8d5-94c64288c0c1-tmp-dir\") pod \"distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx\" (UID: \"bd662f3a-659f-477c-b8d5-94c64288c0c1\") " pod="e2e-models-e2e-worker-w5-967fac/distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx" Oct 30 16:37:43.371965 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:43.371041 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd662f3a-659f-477c-b8d5-94c64288c0c1-tls-certs\") pod \"distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx\" (UID: \"bd662f3a-659f-477c-b8d5-94c64288c0c1\") " pod="e2e-models-e2e-worker-w5-967fac/distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx" Oct 30 16:37:43.371965 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:43.371122 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd662f3a-659f-477c-b8d5-94c64288c0c1-model-cache\") pod \"distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx\" (UID: \"bd662f3a-659f-477c-b8d5-94c64288c0c1\") " pod="e2e-models-e2e-worker-w5-967fac/distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx" Oct 30 16:37:43.371965 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:43.371171 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkjvr\" (UniqueName: \"kubernetes.io/projected/bd662f3a-659f-477c-b8d5-94c64288c0c1-kube-api-access-vkjvr\") pod \"distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx\" (UID: \"bd662f3a-659f-477c-b8d5-94c64288c0c1\") " pod="e2e-models-e2e-worker-w5-967fac/distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx" Oct 30 16:37:43.371965 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:43.371248 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd662f3a-659f-477c-b8d5-94c64288c0c1-dshm\") pod \"distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx\" (UID: \"bd662f3a-659f-477c-b8d5-94c64288c0c1\") " pod="e2e-models-e2e-worker-w5-967fac/distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx" Oct 30 16:37:43.371965 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:43.371284 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd662f3a-659f-477c-b8d5-94c64288c0c1-tmp-dir\") pod \"distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx\" (UID: \"bd662f3a-659f-477c-b8d5-94c64288c0c1\") " pod="e2e-models-e2e-worker-w5-967fac/distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx" Oct 30 16:37:43.371965 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:43.371760 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd662f3a-659f-477c-b8d5-94c64288c0c1-model-cache\") pod \"distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx\" (UID: \"bd662f3a-659f-477c-b8d5-94c64288c0c1\") " pod="e2e-models-e2e-worker-w5-967fac/distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx" Oct 30 16:37:43.436653 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:43.411198 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd662f3a-659f-477c-b8d5-94c64288c0c1-tls-certs\") pod \"distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx\" (UID: \"bd662f3a-659f-477c-b8d5-94c64288c0c1\") " pod="e2e-models-e2e-worker-w5-967fac/distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx" Oct 30 16:37:43.436653 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:43.411407 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd662f3a-659f-477c-b8d5-94c64288c0c1-dshm\") pod \"distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx\" (UID: \"bd662f3a-659f-477c-b8d5-94c64288c0c1\") " pod="e2e-models-e2e-worker-w5-967fac/distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx" Oct 30 16:37:43.436653 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:43.426768 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkjvr\" (UniqueName: \"kubernetes.io/projected/bd662f3a-659f-477c-b8d5-94c64288c0c1-kube-api-access-vkjvr\") pod \"distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx\" (UID: \"bd662f3a-659f-477c-b8d5-94c64288c0c1\") " pod="e2e-models-e2e-worker-w5-967fac/distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx" Oct 30 16:37:43.451261 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:43.446813 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-967fac/distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx" Oct 30 16:37:43.804259 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:43.804126 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-967fac/distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx"] Oct 30 16:37:43.807151 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:37:43.807111 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbd662f3a_659f_477c_b8d5_94c64288c0c1.slice/crio-654c0ad8a3784f9c6ba80cbd68181439d0451cd32849d3279ae8ec2e452a92ad WatchSource:0}: Error finding container 654c0ad8a3784f9c6ba80cbd68181439d0451cd32849d3279ae8ec2e452a92ad: Status 404 returned error can't find the container with id 654c0ad8a3784f9c6ba80cbd68181439d0451cd32849d3279ae8ec2e452a92ad Oct 30 16:37:44.624777 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:44.624745 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-967fac/distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx" event={"ID":"bd662f3a-659f-477c-b8d5-94c64288c0c1","Type":"ContainerStarted","Data":"ad5251608a3f1bb102f143be62b37a74a7d717543190a34039207ec335cf92a9"} Oct 30 16:37:44.624777 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:44.624784 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-967fac/distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx" event={"ID":"bd662f3a-659f-477c-b8d5-94c64288c0c1","Type":"ContainerStarted","Data":"654c0ad8a3784f9c6ba80cbd68181439d0451cd32849d3279ae8ec2e452a92ad"} Oct 30 16:37:44.669640 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:44.669332 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-967fac/distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx" podStartSLOduration=1.669317369 podStartE2EDuration="1.669317369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:37:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:37:44.666724715 +0000 UTC m=+1020.194059934" watchObservedRunningTime="2026-10-30 16:37:44.669317369 +0000 UTC m=+1020.196652587" Oct 30 16:37:45.450834 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:45.450802 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-967fac/distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx" Oct 30 16:37:48.815379 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:48.815334 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-vbvvh"] Oct 30 16:37:48.820334 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:48.820314 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-vbvvh" Oct 30 16:37:48.846772 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:48.846741 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-vbvvh"] Oct 30 16:37:48.855624 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:37:48.854713 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 16:37:48.943782 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:48.943742 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a5f0a2a-f2f2-4bea-98e8-16e6794028de-istio-podinfo\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-vbvvh\" (UID: \"0a5f0a2a-f2f2-4bea-98e8-16e6794028de\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-vbvvh" Oct 30 16:37:48.943949 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:48.943794 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a5f0a2a-f2f2-4bea-98e8-16e6794028de-istiod-ca-cert\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-vbvvh\" (UID: \"0a5f0a2a-f2f2-4bea-98e8-16e6794028de\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-vbvvh" Oct 30 16:37:48.943949 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:48.943820 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a5f0a2a-f2f2-4bea-98e8-16e6794028de-istio-envoy\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-vbvvh\" (UID: \"0a5f0a2a-f2f2-4bea-98e8-16e6794028de\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-vbvvh" Oct 30 16:37:48.943949 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:48.943928 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nt8qt\" (UniqueName: \"kubernetes.io/projected/0a5f0a2a-f2f2-4bea-98e8-16e6794028de-kube-api-access-nt8qt\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-vbvvh\" (UID: \"0a5f0a2a-f2f2-4bea-98e8-16e6794028de\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-vbvvh" Oct 30 16:37:48.944050 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:48.943996 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a5f0a2a-f2f2-4bea-98e8-16e6794028de-workload-certs\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-vbvvh\" (UID: \"0a5f0a2a-f2f2-4bea-98e8-16e6794028de\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-vbvvh" Oct 30 16:37:48.944082 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:48.944072 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a5f0a2a-f2f2-4bea-98e8-16e6794028de-workload-socket\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-vbvvh\" (UID: \"0a5f0a2a-f2f2-4bea-98e8-16e6794028de\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-vbvvh" Oct 30 16:37:48.944148 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:48.944133 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a5f0a2a-f2f2-4bea-98e8-16e6794028de-credential-socket\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-vbvvh\" (UID: \"0a5f0a2a-f2f2-4bea-98e8-16e6794028de\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-vbvvh" Oct 30 16:37:48.944192 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:48.944179 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a5f0a2a-f2f2-4bea-98e8-16e6794028de-istio-data\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-vbvvh\" (UID: \"0a5f0a2a-f2f2-4bea-98e8-16e6794028de\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-vbvvh" Oct 30 16:37:48.944234 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:48.944207 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a5f0a2a-f2f2-4bea-98e8-16e6794028de-istio-token\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-vbvvh\" (UID: \"0a5f0a2a-f2f2-4bea-98e8-16e6794028de\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-vbvvh" Oct 30 16:37:49.051600 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:49.048712 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt8qt\" (UniqueName: \"kubernetes.io/projected/0a5f0a2a-f2f2-4bea-98e8-16e6794028de-kube-api-access-nt8qt\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-vbvvh\" (UID: \"0a5f0a2a-f2f2-4bea-98e8-16e6794028de\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-vbvvh" Oct 30 16:37:49.051600 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:49.049083 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a5f0a2a-f2f2-4bea-98e8-16e6794028de-workload-certs\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-vbvvh\" (UID: \"0a5f0a2a-f2f2-4bea-98e8-16e6794028de\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-vbvvh" Oct 30 16:37:49.051600 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:49.049245 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a5f0a2a-f2f2-4bea-98e8-16e6794028de-workload-socket\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-vbvvh\" (UID: \"0a5f0a2a-f2f2-4bea-98e8-16e6794028de\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-vbvvh" Oct 30 16:37:49.051600 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:49.049382 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a5f0a2a-f2f2-4bea-98e8-16e6794028de-credential-socket\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-vbvvh\" (UID: \"0a5f0a2a-f2f2-4bea-98e8-16e6794028de\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-vbvvh" Oct 30 16:37:49.051600 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:49.049408 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a5f0a2a-f2f2-4bea-98e8-16e6794028de-istio-data\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-vbvvh\" (UID: \"0a5f0a2a-f2f2-4bea-98e8-16e6794028de\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-vbvvh" Oct 30 16:37:49.051600 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:49.049437 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a5f0a2a-f2f2-4bea-98e8-16e6794028de-istio-token\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-vbvvh\" (UID: \"0a5f0a2a-f2f2-4bea-98e8-16e6794028de\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-vbvvh" Oct 30 16:37:49.051600 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:49.049565 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a5f0a2a-f2f2-4bea-98e8-16e6794028de-istio-podinfo\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-vbvvh\" (UID: \"0a5f0a2a-f2f2-4bea-98e8-16e6794028de\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-vbvvh" Oct 30 16:37:49.051600 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:49.049636 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a5f0a2a-f2f2-4bea-98e8-16e6794028de-istiod-ca-cert\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-vbvvh\" (UID: \"0a5f0a2a-f2f2-4bea-98e8-16e6794028de\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-vbvvh" Oct 30 16:37:49.051600 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:49.049663 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a5f0a2a-f2f2-4bea-98e8-16e6794028de-istio-envoy\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-vbvvh\" (UID: \"0a5f0a2a-f2f2-4bea-98e8-16e6794028de\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-vbvvh" Oct 30 16:37:49.051600 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:49.050692 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a5f0a2a-f2f2-4bea-98e8-16e6794028de-credential-socket\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-vbvvh\" (UID: \"0a5f0a2a-f2f2-4bea-98e8-16e6794028de\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-vbvvh" Oct 30 16:37:49.051600 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:49.051485 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a5f0a2a-f2f2-4bea-98e8-16e6794028de-workload-certs\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-vbvvh\" (UID: \"0a5f0a2a-f2f2-4bea-98e8-16e6794028de\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-vbvvh" Oct 30 16:37:49.064654 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:49.064483 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a5f0a2a-f2f2-4bea-98e8-16e6794028de-istio-data\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-vbvvh\" (UID: \"0a5f0a2a-f2f2-4bea-98e8-16e6794028de\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-vbvvh" Oct 30 16:37:49.067599 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:49.065175 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a5f0a2a-f2f2-4bea-98e8-16e6794028de-istiod-ca-cert\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-vbvvh\" (UID: \"0a5f0a2a-f2f2-4bea-98e8-16e6794028de\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-vbvvh" Oct 30 16:37:49.067599 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:49.065451 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a5f0a2a-f2f2-4bea-98e8-16e6794028de-workload-socket\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-vbvvh\" (UID: \"0a5f0a2a-f2f2-4bea-98e8-16e6794028de\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-vbvvh" Oct 30 16:37:49.068256 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:49.068233 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a5f0a2a-f2f2-4bea-98e8-16e6794028de-istio-envoy\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-vbvvh\" (UID: \"0a5f0a2a-f2f2-4bea-98e8-16e6794028de\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-vbvvh" Oct 30 16:37:49.069969 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:49.069934 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a5f0a2a-f2f2-4bea-98e8-16e6794028de-istio-podinfo\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-vbvvh\" (UID: \"0a5f0a2a-f2f2-4bea-98e8-16e6794028de\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-vbvvh" Oct 30 16:37:49.087285 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:49.087226 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt8qt\" (UniqueName: \"kubernetes.io/projected/0a5f0a2a-f2f2-4bea-98e8-16e6794028de-kube-api-access-nt8qt\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-vbvvh\" (UID: \"0a5f0a2a-f2f2-4bea-98e8-16e6794028de\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-vbvvh" Oct 30 16:37:49.088620 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:49.088575 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a5f0a2a-f2f2-4bea-98e8-16e6794028de-istio-token\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-vbvvh\" (UID: \"0a5f0a2a-f2f2-4bea-98e8-16e6794028de\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-vbvvh" Oct 30 16:37:49.135807 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:49.135779 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-vbvvh" Oct 30 16:37:49.459972 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:49.459942 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-vbvvh"] Oct 30 16:37:49.461440 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:37:49.461409 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a5f0a2a_f2f2_4bea_98e8_16e6794028de.slice/crio-9d0ccf93dffcf6bc05ec61a46436061eb042d70da54bc8ee611c6b1449b20885 WatchSource:0}: Error finding container 9d0ccf93dffcf6bc05ec61a46436061eb042d70da54bc8ee611c6b1449b20885: Status 404 returned error can't find the container with id 9d0ccf93dffcf6bc05ec61a46436061eb042d70da54bc8ee611c6b1449b20885 Oct 30 16:37:49.464061 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:49.464026 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:37:49.464152 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:49.464091 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:37:49.464152 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:49.464119 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:37:49.662738 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:49.662703 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-vbvvh" event={"ID":"0a5f0a2a-f2f2-4bea-98e8-16e6794028de","Type":"ContainerStarted","Data":"48c4105bf5427439a31b5cd61c955f82d1e0fa0e1b84b87baeed6c3b947b0793"} Oct 30 16:37:49.662898 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:49.662766 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-vbvvh" event={"ID":"0a5f0a2a-f2f2-4bea-98e8-16e6794028de","Type":"ContainerStarted","Data":"9d0ccf93dffcf6bc05ec61a46436061eb042d70da54bc8ee611c6b1449b20885"} Oct 30 16:37:49.707801 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:49.707746 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-vbvvh" podStartSLOduration=1.707733128 podStartE2EDuration="1.707733128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:37:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:37:49.705902837 +0000 UTC m=+1025.233238080" watchObservedRunningTime="2026-10-30 16:37:49.707733128 +0000 UTC m=+1025.235068349" Oct 30 16:37:50.136919 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:50.136884 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-vbvvh" Oct 30 16:37:50.138624 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:50.138571 5052 patch_prober.go:29] interesting pod/e2e-worker-w3-0b2348-openshift-default-85679bc47c-vbvvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 16:37:50.138758 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:50.138652 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-vbvvh" podUID="0a5f0a2a-f2f2-4bea-98e8-16e6794028de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 16:37:51.144604 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:51.143977 5052 patch_prober.go:29] interesting pod/e2e-worker-w3-0b2348-openshift-default-85679bc47c-vbvvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 16:37:51.144604 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:51.144042 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-vbvvh" podUID="0a5f0a2a-f2f2-4bea-98e8-16e6794028de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 16:37:52.143604 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:52.142987 5052 patch_prober.go:29] interesting pod/e2e-worker-w3-0b2348-openshift-default-85679bc47c-vbvvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 16:37:52.143604 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:52.143068 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-vbvvh" podUID="0a5f0a2a-f2f2-4bea-98e8-16e6794028de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 16:37:53.148605 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:53.139990 5052 patch_prober.go:29] interesting pod/e2e-worker-w3-0b2348-openshift-default-85679bc47c-vbvvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 16:37:53.148605 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:53.140082 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-vbvvh" podUID="0a5f0a2a-f2f2-4bea-98e8-16e6794028de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 16:37:53.461659 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:53.461457 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-967fac/distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx" Oct 30 16:37:53.517100 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:53.517071 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-967fac/distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx" Oct 30 16:37:53.715512 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:53.715431 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-967fac/distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx" Oct 30 16:37:54.153601 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:54.150206 5052 patch_prober.go:29] interesting pod/e2e-worker-w3-0b2348-openshift-default-85679bc47c-vbvvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 16:37:54.153601 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:54.150262 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-vbvvh" podUID="0a5f0a2a-f2f2-4bea-98e8-16e6794028de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 16:37:55.148602 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:55.137949 5052 patch_prober.go:29] interesting pod/e2e-worker-w3-0b2348-openshift-default-85679bc47c-vbvvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 16:37:55.148602 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:55.138027 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-vbvvh" podUID="0a5f0a2a-f2f2-4bea-98e8-16e6794028de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 16:37:56.136979 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:56.136948 5052 patch_prober.go:29] interesting pod/e2e-worker-w3-0b2348-openshift-default-85679bc47c-vbvvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 16:37:56.137355 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:56.137008 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-vbvvh" podUID="0a5f0a2a-f2f2-4bea-98e8-16e6794028de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 16:37:57.441301 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:57.441270 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-vbvvh" Oct 30 16:37:57.441971 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:57.441944 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-vbvvh" Oct 30 16:37:57.442939 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:37:57.442917 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-vbvvh" Oct 30 16:38:04.239560 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:04.239526 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-v2jw7"] Oct 30 16:38:04.245176 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:04.245153 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-v2jw7" Oct 30 16:38:04.328746 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:04.328634 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-v2jw7"] Oct 30 16:38:04.344777 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:04.339204 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-4frq9"] Oct 30 16:38:04.362198 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:04.360401 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-4frq9" Oct 30 16:38:04.365418 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:04.365392 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-4frq9"] Oct 30 16:38:04.404143 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:04.404106 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a40b021f-16e2-4821-ad8b-9a7fb6a4a58f-credential-socket\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-v2jw7\" (UID: \"a40b021f-16e2-4821-ad8b-9a7fb6a4a58f\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-v2jw7" Oct 30 16:38:04.404476 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:04.404443 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a40b021f-16e2-4821-ad8b-9a7fb6a4a58f-istio-data\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-v2jw7\" (UID: \"a40b021f-16e2-4821-ad8b-9a7fb6a4a58f\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-v2jw7" Oct 30 16:38:04.404626 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:04.404491 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a40b021f-16e2-4821-ad8b-9a7fb6a4a58f-istio-envoy\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-v2jw7\" (UID: \"a40b021f-16e2-4821-ad8b-9a7fb6a4a58f\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-v2jw7" Oct 30 16:38:04.404626 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:04.404524 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a40b021f-16e2-4821-ad8b-9a7fb6a4a58f-workload-socket\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-v2jw7\" (UID: \"a40b021f-16e2-4821-ad8b-9a7fb6a4a58f\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-v2jw7" Oct 30 16:38:04.404626 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:04.404622 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a40b021f-16e2-4821-ad8b-9a7fb6a4a58f-workload-certs\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-v2jw7\" (UID: \"a40b021f-16e2-4821-ad8b-9a7fb6a4a58f\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-v2jw7" Oct 30 16:38:04.404799 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:04.404681 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a40b021f-16e2-4821-ad8b-9a7fb6a4a58f-istio-token\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-v2jw7\" (UID: \"a40b021f-16e2-4821-ad8b-9a7fb6a4a58f\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-v2jw7" Oct 30 16:38:04.405213 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:04.405189 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvwvd\" (UniqueName: \"kubernetes.io/projected/a40b021f-16e2-4821-ad8b-9a7fb6a4a58f-kube-api-access-gvwvd\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-v2jw7\" (UID: \"a40b021f-16e2-4821-ad8b-9a7fb6a4a58f\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-v2jw7" Oct 30 16:38:04.405314 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:04.405244 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a40b021f-16e2-4821-ad8b-9a7fb6a4a58f-istio-podinfo\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-v2jw7\" (UID: \"a40b021f-16e2-4821-ad8b-9a7fb6a4a58f\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-v2jw7" Oct 30 16:38:04.405395 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:04.405370 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a40b021f-16e2-4821-ad8b-9a7fb6a4a58f-istiod-ca-cert\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-v2jw7\" (UID: \"a40b021f-16e2-4821-ad8b-9a7fb6a4a58f\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-v2jw7" Oct 30 16:38:04.506715 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:04.506632 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a40b021f-16e2-4821-ad8b-9a7fb6a4a58f-istiod-ca-cert\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-v2jw7\" (UID: \"a40b021f-16e2-4821-ad8b-9a7fb6a4a58f\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-v2jw7" Oct 30 16:38:04.506852 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:04.506723 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a40b021f-16e2-4821-ad8b-9a7fb6a4a58f-credential-socket\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-v2jw7\" (UID: \"a40b021f-16e2-4821-ad8b-9a7fb6a4a58f\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-v2jw7" Oct 30 16:38:04.506852 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:04.506751 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8451a193-e5be-47eb-9e5b-75c5214d3abc-istio-token\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-4frq9\" (UID: \"8451a193-e5be-47eb-9e5b-75c5214d3abc\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-4frq9" Oct 30 16:38:04.506852 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:04.506777 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8451a193-e5be-47eb-9e5b-75c5214d3abc-istio-podinfo\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-4frq9\" (UID: \"8451a193-e5be-47eb-9e5b-75c5214d3abc\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-4frq9" Oct 30 16:38:04.506963 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:04.506848 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8451a193-e5be-47eb-9e5b-75c5214d3abc-workload-certs\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-4frq9\" (UID: \"8451a193-e5be-47eb-9e5b-75c5214d3abc\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-4frq9" Oct 30 16:38:04.506998 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:04.506961 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a40b021f-16e2-4821-ad8b-9a7fb6a4a58f-istio-data\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-v2jw7\" (UID: \"a40b021f-16e2-4821-ad8b-9a7fb6a4a58f\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-v2jw7" Oct 30 16:38:04.507039 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:04.506993 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a40b021f-16e2-4821-ad8b-9a7fb6a4a58f-istio-envoy\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-v2jw7\" (UID: \"a40b021f-16e2-4821-ad8b-9a7fb6a4a58f\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-v2jw7" Oct 30 16:38:04.507039 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:04.507020 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8451a193-e5be-47eb-9e5b-75c5214d3abc-workload-socket\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-4frq9\" (UID: \"8451a193-e5be-47eb-9e5b-75c5214d3abc\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-4frq9" Oct 30 16:38:04.507139 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:04.507050 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a40b021f-16e2-4821-ad8b-9a7fb6a4a58f-workload-socket\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-v2jw7\" (UID: \"a40b021f-16e2-4821-ad8b-9a7fb6a4a58f\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-v2jw7" Oct 30 16:38:04.507139 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:04.507104 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlrrt\" (UniqueName: \"kubernetes.io/projected/8451a193-e5be-47eb-9e5b-75c5214d3abc-kube-api-access-vlrrt\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-4frq9\" (UID: \"8451a193-e5be-47eb-9e5b-75c5214d3abc\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-4frq9" Oct 30 16:38:04.507245 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:04.507191 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8451a193-e5be-47eb-9e5b-75c5214d3abc-istio-data\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-4frq9\" (UID: \"8451a193-e5be-47eb-9e5b-75c5214d3abc\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-4frq9" Oct 30 16:38:04.507245 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:04.507198 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a40b021f-16e2-4821-ad8b-9a7fb6a4a58f-credential-socket\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-v2jw7\" (UID: \"a40b021f-16e2-4821-ad8b-9a7fb6a4a58f\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-v2jw7" Oct 30 16:38:04.507357 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:04.507280 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8451a193-e5be-47eb-9e5b-75c5214d3abc-istiod-ca-cert\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-4frq9\" (UID: \"8451a193-e5be-47eb-9e5b-75c5214d3abc\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-4frq9" Oct 30 16:38:04.507357 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:04.507283 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a40b021f-16e2-4821-ad8b-9a7fb6a4a58f-istio-data\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-v2jw7\" (UID: \"a40b021f-16e2-4821-ad8b-9a7fb6a4a58f\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-v2jw7" Oct 30 16:38:04.507357 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:04.507351 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a40b021f-16e2-4821-ad8b-9a7fb6a4a58f-workload-certs\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-v2jw7\" (UID: \"a40b021f-16e2-4821-ad8b-9a7fb6a4a58f\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-v2jw7" Oct 30 16:38:04.507514 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:04.507372 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a40b021f-16e2-4821-ad8b-9a7fb6a4a58f-istio-token\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-v2jw7\" (UID: \"a40b021f-16e2-4821-ad8b-9a7fb6a4a58f\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-v2jw7" Oct 30 16:38:04.507514 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:04.507410 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a40b021f-16e2-4821-ad8b-9a7fb6a4a58f-workload-socket\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-v2jw7\" (UID: \"a40b021f-16e2-4821-ad8b-9a7fb6a4a58f\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-v2jw7" Oct 30 16:38:04.507514 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:04.507427 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvwvd\" (UniqueName: \"kubernetes.io/projected/a40b021f-16e2-4821-ad8b-9a7fb6a4a58f-kube-api-access-gvwvd\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-v2jw7\" (UID: \"a40b021f-16e2-4821-ad8b-9a7fb6a4a58f\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-v2jw7" Oct 30 16:38:04.507514 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:04.507429 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a40b021f-16e2-4821-ad8b-9a7fb6a4a58f-istiod-ca-cert\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-v2jw7\" (UID: \"a40b021f-16e2-4821-ad8b-9a7fb6a4a58f\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-v2jw7" Oct 30 16:38:04.507514 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:04.507453 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8451a193-e5be-47eb-9e5b-75c5214d3abc-istio-envoy\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-4frq9\" (UID: \"8451a193-e5be-47eb-9e5b-75c5214d3abc\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-4frq9" Oct 30 16:38:04.507790 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:04.507527 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8451a193-e5be-47eb-9e5b-75c5214d3abc-credential-socket\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-4frq9\" (UID: \"8451a193-e5be-47eb-9e5b-75c5214d3abc\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-4frq9" Oct 30 16:38:04.507790 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:04.507561 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a40b021f-16e2-4821-ad8b-9a7fb6a4a58f-istio-podinfo\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-v2jw7\" (UID: \"a40b021f-16e2-4821-ad8b-9a7fb6a4a58f\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-v2jw7" Oct 30 16:38:04.507790 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:04.507640 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a40b021f-16e2-4821-ad8b-9a7fb6a4a58f-workload-certs\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-v2jw7\" (UID: \"a40b021f-16e2-4821-ad8b-9a7fb6a4a58f\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-v2jw7" Oct 30 16:38:04.509366 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:04.509340 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a40b021f-16e2-4821-ad8b-9a7fb6a4a58f-istio-envoy\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-v2jw7\" (UID: \"a40b021f-16e2-4821-ad8b-9a7fb6a4a58f\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-v2jw7" Oct 30 16:38:04.509897 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:04.509878 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a40b021f-16e2-4821-ad8b-9a7fb6a4a58f-istio-podinfo\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-v2jw7\" (UID: \"a40b021f-16e2-4821-ad8b-9a7fb6a4a58f\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-v2jw7" Oct 30 16:38:04.539160 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:04.539137 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a40b021f-16e2-4821-ad8b-9a7fb6a4a58f-istio-token\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-v2jw7\" (UID: \"a40b021f-16e2-4821-ad8b-9a7fb6a4a58f\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-v2jw7" Oct 30 16:38:04.540869 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:04.540845 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvwvd\" (UniqueName: \"kubernetes.io/projected/a40b021f-16e2-4821-ad8b-9a7fb6a4a58f-kube-api-access-gvwvd\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-v2jw7\" (UID: \"a40b021f-16e2-4821-ad8b-9a7fb6a4a58f\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-v2jw7" Oct 30 16:38:04.609473 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:04.609439 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8451a193-e5be-47eb-9e5b-75c5214d3abc-istio-token\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-4frq9\" (UID: \"8451a193-e5be-47eb-9e5b-75c5214d3abc\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-4frq9" Oct 30 16:38:04.609672 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:04.609487 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8451a193-e5be-47eb-9e5b-75c5214d3abc-istio-podinfo\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-4frq9\" (UID: \"8451a193-e5be-47eb-9e5b-75c5214d3abc\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-4frq9" Oct 30 16:38:04.609672 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:04.609521 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8451a193-e5be-47eb-9e5b-75c5214d3abc-workload-certs\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-4frq9\" (UID: \"8451a193-e5be-47eb-9e5b-75c5214d3abc\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-4frq9" Oct 30 16:38:04.609672 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:04.609613 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8451a193-e5be-47eb-9e5b-75c5214d3abc-workload-socket\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-4frq9\" (UID: \"8451a193-e5be-47eb-9e5b-75c5214d3abc\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-4frq9" Oct 30 16:38:04.609672 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:04.609652 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlrrt\" (UniqueName: \"kubernetes.io/projected/8451a193-e5be-47eb-9e5b-75c5214d3abc-kube-api-access-vlrrt\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-4frq9\" (UID: \"8451a193-e5be-47eb-9e5b-75c5214d3abc\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-4frq9" Oct 30 16:38:04.609905 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:04.609695 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8451a193-e5be-47eb-9e5b-75c5214d3abc-istio-data\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-4frq9\" (UID: \"8451a193-e5be-47eb-9e5b-75c5214d3abc\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-4frq9" Oct 30 16:38:04.609905 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:04.609737 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8451a193-e5be-47eb-9e5b-75c5214d3abc-istiod-ca-cert\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-4frq9\" (UID: \"8451a193-e5be-47eb-9e5b-75c5214d3abc\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-4frq9" Oct 30 16:38:04.609905 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:04.609861 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8451a193-e5be-47eb-9e5b-75c5214d3abc-istio-envoy\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-4frq9\" (UID: \"8451a193-e5be-47eb-9e5b-75c5214d3abc\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-4frq9" Oct 30 16:38:04.609905 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:04.609898 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8451a193-e5be-47eb-9e5b-75c5214d3abc-credential-socket\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-4frq9\" (UID: \"8451a193-e5be-47eb-9e5b-75c5214d3abc\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-4frq9" Oct 30 16:38:04.610131 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:04.610067 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8451a193-e5be-47eb-9e5b-75c5214d3abc-workload-certs\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-4frq9\" (UID: \"8451a193-e5be-47eb-9e5b-75c5214d3abc\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-4frq9" Oct 30 16:38:04.610131 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:04.610118 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8451a193-e5be-47eb-9e5b-75c5214d3abc-workload-socket\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-4frq9\" (UID: \"8451a193-e5be-47eb-9e5b-75c5214d3abc\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-4frq9" Oct 30 16:38:04.610306 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:04.610288 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8451a193-e5be-47eb-9e5b-75c5214d3abc-credential-socket\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-4frq9\" (UID: \"8451a193-e5be-47eb-9e5b-75c5214d3abc\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-4frq9" Oct 30 16:38:04.611466 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:04.611137 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8451a193-e5be-47eb-9e5b-75c5214d3abc-istio-data\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-4frq9\" (UID: \"8451a193-e5be-47eb-9e5b-75c5214d3abc\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-4frq9" Oct 30 16:38:04.611466 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:04.611403 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8451a193-e5be-47eb-9e5b-75c5214d3abc-istiod-ca-cert\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-4frq9\" (UID: \"8451a193-e5be-47eb-9e5b-75c5214d3abc\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-4frq9" Oct 30 16:38:04.613163 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:04.613137 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8451a193-e5be-47eb-9e5b-75c5214d3abc-istio-envoy\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-4frq9\" (UID: \"8451a193-e5be-47eb-9e5b-75c5214d3abc\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-4frq9" Oct 30 16:38:04.613261 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:04.613243 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8451a193-e5be-47eb-9e5b-75c5214d3abc-istio-podinfo\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-4frq9\" (UID: \"8451a193-e5be-47eb-9e5b-75c5214d3abc\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-4frq9" Oct 30 16:38:04.639122 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:04.639098 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8451a193-e5be-47eb-9e5b-75c5214d3abc-istio-token\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-4frq9\" (UID: \"8451a193-e5be-47eb-9e5b-75c5214d3abc\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-4frq9" Oct 30 16:38:04.641768 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:04.641747 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlrrt\" (UniqueName: \"kubernetes.io/projected/8451a193-e5be-47eb-9e5b-75c5214d3abc-kube-api-access-vlrrt\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-4frq9\" (UID: \"8451a193-e5be-47eb-9e5b-75c5214d3abc\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-4frq9" Oct 30 16:38:04.655043 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:04.655022 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-v2jw7" Oct 30 16:38:04.679237 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:04.679211 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-4frq9" Oct 30 16:38:04.984641 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:04.984618 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-v2jw7"] Oct 30 16:38:04.987060 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:38:04.987020 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda40b021f_16e2_4821_ad8b_9a7fb6a4a58f.slice/crio-29dda3bf19c718bd1846e00f562ac38729bb8ee10adf7718a8856492979cd8c0 WatchSource:0}: Error finding container 29dda3bf19c718bd1846e00f562ac38729bb8ee10adf7718a8856492979cd8c0: Status 404 returned error can't find the container with id 29dda3bf19c718bd1846e00f562ac38729bb8ee10adf7718a8856492979cd8c0 Oct 30 16:38:04.989400 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:04.989367 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:38:04.989499 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:04.989447 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:38:04.989563 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:04.989493 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:38:05.032231 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:05.032207 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-4frq9"] Oct 30 16:38:05.035187 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:38:05.035155 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8451a193_e5be_47eb_9e5b_75c5214d3abc.slice/crio-1cf17567e27e0368e714f1934f7236aefe195f2718488e24e6196d744b55b2b7 WatchSource:0}: Error finding container 1cf17567e27e0368e714f1934f7236aefe195f2718488e24e6196d744b55b2b7: Status 404 returned error can't find the container with id 1cf17567e27e0368e714f1934f7236aefe195f2718488e24e6196d744b55b2b7 Oct 30 16:38:05.037509 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:05.037481 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:38:05.037627 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:05.037545 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:38:05.037627 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:05.037572 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:38:05.817050 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:05.814817 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-4frq9" event={"ID":"8451a193-e5be-47eb-9e5b-75c5214d3abc","Type":"ContainerStarted","Data":"a5603fd958409a6b510ee54e6488ae161277b39817552e58289ff5907c5d62b0"} Oct 30 16:38:05.817050 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:05.814880 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-4frq9" event={"ID":"8451a193-e5be-47eb-9e5b-75c5214d3abc","Type":"ContainerStarted","Data":"1cf17567e27e0368e714f1934f7236aefe195f2718488e24e6196d744b55b2b7"} Oct 30 16:38:05.826305 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:05.826272 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-v2jw7" event={"ID":"a40b021f-16e2-4821-ad8b-9a7fb6a4a58f","Type":"ContainerStarted","Data":"c23ab4acd7d104fb93b7a9ff046abf01b183265f52ce398769d1d413d399f552"} Oct 30 16:38:05.826479 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:05.826334 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-v2jw7" event={"ID":"a40b021f-16e2-4821-ad8b-9a7fb6a4a58f","Type":"ContainerStarted","Data":"29dda3bf19c718bd1846e00f562ac38729bb8ee10adf7718a8856492979cd8c0"} Oct 30 16:38:05.944606 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:05.932826 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-4frq9" podStartSLOduration=1.9328034079999998 podStartE2EDuration="1.932803408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:38:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:38:05.909263315 +0000 UTC m=+1041.436598594" watchObservedRunningTime="2026-10-30 16:38:05.932803408 +0000 UTC m=+1041.460138627" Oct 30 16:38:05.988313 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:05.988248 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-v2jw7" podStartSLOduration=1.988226977 podStartE2EDuration="1.988226977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:38:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:38:05.984245098 +0000 UTC m=+1041.511580342" watchObservedRunningTime="2026-10-30 16:38:05.988226977 +0000 UTC m=+1041.515562198" Oct 30 16:38:06.660603 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:06.660083 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-v2jw7" Oct 30 16:38:06.660603 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:06.660222 5052 patch_prober.go:29] interesting pod/e2e-worker-w5-967fac-openshift-default-dffc67487-v2jw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 16:38:06.660603 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:06.660265 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-v2jw7" podUID="a40b021f-16e2-4821-ad8b-9a7fb6a4a58f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 16:38:06.679981 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:06.679945 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-4frq9" Oct 30 16:38:06.681565 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:06.681534 5052 patch_prober.go:29] interesting pod/e2e-worker-w4-263f00-openshift-default-b969d55bc-4frq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 16:38:06.681722 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:06.681619 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-4frq9" podUID="8451a193-e5be-47eb-9e5b-75c5214d3abc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 16:38:07.664603 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:07.664000 5052 patch_prober.go:29] interesting pod/e2e-worker-w5-967fac-openshift-default-dffc67487-v2jw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 16:38:07.664603 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:07.664064 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-v2jw7" podUID="a40b021f-16e2-4821-ad8b-9a7fb6a4a58f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 16:38:07.682210 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:07.681945 5052 patch_prober.go:29] interesting pod/e2e-worker-w4-263f00-openshift-default-b969d55bc-4frq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 16:38:07.682210 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:07.682006 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-4frq9" podUID="8451a193-e5be-47eb-9e5b-75c5214d3abc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 16:38:08.694612 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:08.693707 5052 patch_prober.go:29] interesting pod/e2e-worker-w4-263f00-openshift-default-b969d55bc-4frq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 16:38:08.694612 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:08.693771 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-4frq9" podUID="8451a193-e5be-47eb-9e5b-75c5214d3abc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 16:38:08.694612 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:08.694469 5052 patch_prober.go:29] interesting pod/e2e-worker-w5-967fac-openshift-default-dffc67487-v2jw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 16:38:08.694612 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:08.694598 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-v2jw7" podUID="a40b021f-16e2-4821-ad8b-9a7fb6a4a58f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 16:38:09.194737 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:09.194700 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cdwcj"] Oct 30 16:38:09.657055 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:09.657018 5052 patch_prober.go:29] interesting pod/e2e-worker-w5-967fac-openshift-default-dffc67487-v2jw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 16:38:09.661602 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:09.657274 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-v2jw7" podUID="a40b021f-16e2-4821-ad8b-9a7fb6a4a58f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 16:38:09.696637 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:09.693731 5052 patch_prober.go:29] interesting pod/e2e-worker-w4-263f00-openshift-default-b969d55bc-4frq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 16:38:09.696637 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:09.693799 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-4frq9" podUID="8451a193-e5be-47eb-9e5b-75c5214d3abc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 16:38:10.656006 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:10.655963 5052 patch_prober.go:29] interesting pod/e2e-worker-w5-967fac-openshift-default-dffc67487-v2jw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 16:38:10.656288 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:10.656258 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-v2jw7" podUID="a40b021f-16e2-4821-ad8b-9a7fb6a4a58f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 16:38:10.685636 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:10.679956 5052 patch_prober.go:29] interesting pod/e2e-worker-w4-263f00-openshift-default-b969d55bc-4frq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 16:38:10.685636 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:10.682690 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-4frq9" podUID="8451a193-e5be-47eb-9e5b-75c5214d3abc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 16:38:11.656318 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:11.656275 5052 patch_prober.go:29] interesting pod/e2e-worker-w5-967fac-openshift-default-dffc67487-v2jw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 16:38:11.656956 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:11.656351 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-v2jw7" podUID="a40b021f-16e2-4821-ad8b-9a7fb6a4a58f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 16:38:11.679873 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:11.679829 5052 patch_prober.go:29] interesting pod/e2e-worker-w4-263f00-openshift-default-b969d55bc-4frq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 16:38:11.680035 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:11.679887 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-4frq9" podUID="8451a193-e5be-47eb-9e5b-75c5214d3abc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 16:38:12.666753 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:12.666720 5052 patch_prober.go:29] interesting pod/e2e-worker-w5-967fac-openshift-default-dffc67487-v2jw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 16:38:12.667305 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:12.667275 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-v2jw7" podUID="a40b021f-16e2-4821-ad8b-9a7fb6a4a58f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 16:38:12.681860 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:12.681837 5052 patch_prober.go:29] interesting pod/e2e-worker-w4-263f00-openshift-default-b969d55bc-4frq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 16:38:12.682039 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:12.682017 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-4frq9" podUID="8451a193-e5be-47eb-9e5b-75c5214d3abc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 16:38:13.660706 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:13.660675 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-v2jw7" Oct 30 16:38:13.661146 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:13.661130 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-v2jw7" Oct 30 16:38:13.661752 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:13.661731 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-v2jw7" Oct 30 16:38:14.720725 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:14.720688 5052 patch_prober.go:29] interesting pod/e2e-worker-w4-263f00-openshift-default-b969d55bc-4frq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:38:14.721136 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:14.720743 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-4frq9" podUID="8451a193-e5be-47eb-9e5b-75c5214d3abc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:14.900134 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:14.900100 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-4frq9" Oct 30 16:38:14.900672 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:14.900642 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-4frq9" Oct 30 16:38:14.902981 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:14.902951 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-4frq9" Oct 30 16:38:18.844382 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:38:18.844255 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 16:38:25.581605 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:25.566075 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/distinct-w5-967fac-kserve-69cbcccd8b-b85bn" podUID="8f6b9354-378f-4451-9ff7-be7f7fec12f2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Oct 30 16:38:25.733606 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:25.707666 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx" podUID="bd662f3a-659f-477c-b8d5-94c64288c0c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 16:38:25.733606 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:25.709664 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/distinct-2-w4-263f00-kserve-8464897cf5-q8ck8" podUID="15ed43ef-5cdd-489a-a70e-063239822ff3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:26.611604 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:26.607875 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/distinct-w5-967fac-kserve-69cbcccd8b-b85bn" podUID="8f6b9354-378f-4451-9ff7-be7f7fec12f2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Oct 30 16:38:26.677615 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:26.631296 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-967fac/distinct-w5-967fac-kserve-69cbcccd8b-b85bn" Oct 30 16:38:26.736603 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:26.707866 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx" podUID="bd662f3a-659f-477c-b8d5-94c64288c0c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:26.736603 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:26.710336 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-967fac/distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx" Oct 30 16:38:26.736603 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:26.718306 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/distinct-2-w4-263f00-kserve-8464897cf5-q8ck8" podUID="15ed43ef-5cdd-489a-a70e-063239822ff3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:27.026656 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:27.026620 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-263f00/distinct-2-w4-263f00-kserve-8464897cf5-q8ck8" Oct 30 16:38:27.619508 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:27.619455 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/distinct-w5-967fac-kserve-69cbcccd8b-b85bn" podUID="8f6b9354-378f-4451-9ff7-be7f7fec12f2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:27.628102 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:27.628036 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-263f00/distinct-2-w4-263f00-kserve-8464897cf5-q8ck8" Oct 30 16:38:27.667420 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:27.667025 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-967fac/distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx" Oct 30 16:38:27.861933 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:27.861901 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-967fac/distinct-w5-967fac-kserve-69cbcccd8b-b85bn" Oct 30 16:38:33.988184 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:33.988149 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-kznnt"] Oct 30 16:38:33.993485 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:33.993462 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-kznnt" Oct 30 16:38:34.004809 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:34.004786 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c254bba7-008e-4724-80a8-8292ae42b7e1-istio-data\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-kznnt\" (UID: \"c254bba7-008e-4724-80a8-8292ae42b7e1\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-kznnt" Oct 30 16:38:34.004913 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:34.004858 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dk6pt\" (UniqueName: \"kubernetes.io/projected/c254bba7-008e-4724-80a8-8292ae42b7e1-kube-api-access-dk6pt\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-kznnt\" (UID: \"c254bba7-008e-4724-80a8-8292ae42b7e1\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-kznnt" Oct 30 16:38:34.004980 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:34.004948 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c254bba7-008e-4724-80a8-8292ae42b7e1-workload-certs\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-kznnt\" (UID: \"c254bba7-008e-4724-80a8-8292ae42b7e1\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-kznnt" Oct 30 16:38:34.005082 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:34.005061 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c254bba7-008e-4724-80a8-8292ae42b7e1-istiod-ca-cert\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-kznnt\" (UID: \"c254bba7-008e-4724-80a8-8292ae42b7e1\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-kznnt" Oct 30 16:38:34.005129 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:34.005106 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c254bba7-008e-4724-80a8-8292ae42b7e1-credential-socket\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-kznnt\" (UID: \"c254bba7-008e-4724-80a8-8292ae42b7e1\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-kznnt" Oct 30 16:38:34.005240 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:34.005224 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c254bba7-008e-4724-80a8-8292ae42b7e1-istio-podinfo\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-kznnt\" (UID: \"c254bba7-008e-4724-80a8-8292ae42b7e1\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-kznnt" Oct 30 16:38:34.005305 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:34.005255 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c254bba7-008e-4724-80a8-8292ae42b7e1-istio-token\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-kznnt\" (UID: \"c254bba7-008e-4724-80a8-8292ae42b7e1\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-kznnt" Oct 30 16:38:34.005305 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:34.005282 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c254bba7-008e-4724-80a8-8292ae42b7e1-workload-socket\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-kznnt\" (UID: \"c254bba7-008e-4724-80a8-8292ae42b7e1\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-kznnt" Oct 30 16:38:34.005392 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:34.005372 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c254bba7-008e-4724-80a8-8292ae42b7e1-istio-envoy\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-kznnt\" (UID: \"c254bba7-008e-4724-80a8-8292ae42b7e1\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-kznnt" Oct 30 16:38:34.027080 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:34.027055 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-kznnt"] Oct 30 16:38:34.106469 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:34.106437 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c254bba7-008e-4724-80a8-8292ae42b7e1-istiod-ca-cert\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-kznnt\" (UID: \"c254bba7-008e-4724-80a8-8292ae42b7e1\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-kznnt" Oct 30 16:38:34.106667 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:34.106484 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c254bba7-008e-4724-80a8-8292ae42b7e1-credential-socket\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-kznnt\" (UID: \"c254bba7-008e-4724-80a8-8292ae42b7e1\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-kznnt" Oct 30 16:38:34.106667 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:34.106542 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c254bba7-008e-4724-80a8-8292ae42b7e1-istio-podinfo\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-kznnt\" (UID: \"c254bba7-008e-4724-80a8-8292ae42b7e1\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-kznnt" Oct 30 16:38:34.106667 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:34.106557 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c254bba7-008e-4724-80a8-8292ae42b7e1-istio-token\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-kznnt\" (UID: \"c254bba7-008e-4724-80a8-8292ae42b7e1\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-kznnt" Oct 30 16:38:34.106667 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:34.106573 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c254bba7-008e-4724-80a8-8292ae42b7e1-workload-socket\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-kznnt\" (UID: \"c254bba7-008e-4724-80a8-8292ae42b7e1\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-kznnt" Oct 30 16:38:34.106667 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:34.106629 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c254bba7-008e-4724-80a8-8292ae42b7e1-istio-envoy\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-kznnt\" (UID: \"c254bba7-008e-4724-80a8-8292ae42b7e1\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-kznnt" Oct 30 16:38:34.106926 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:34.106682 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c254bba7-008e-4724-80a8-8292ae42b7e1-istio-data\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-kznnt\" (UID: \"c254bba7-008e-4724-80a8-8292ae42b7e1\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-kznnt" Oct 30 16:38:34.106926 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:34.106760 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk6pt\" (UniqueName: \"kubernetes.io/projected/c254bba7-008e-4724-80a8-8292ae42b7e1-kube-api-access-dk6pt\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-kznnt\" (UID: \"c254bba7-008e-4724-80a8-8292ae42b7e1\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-kznnt" Oct 30 16:38:34.106926 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:34.106791 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c254bba7-008e-4724-80a8-8292ae42b7e1-workload-certs\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-kznnt\" (UID: \"c254bba7-008e-4724-80a8-8292ae42b7e1\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-kznnt" Oct 30 16:38:34.107161 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:34.107136 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c254bba7-008e-4724-80a8-8292ae42b7e1-workload-socket\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-kznnt\" (UID: \"c254bba7-008e-4724-80a8-8292ae42b7e1\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-kznnt" Oct 30 16:38:34.107298 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:34.107202 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c254bba7-008e-4724-80a8-8292ae42b7e1-credential-socket\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-kznnt\" (UID: \"c254bba7-008e-4724-80a8-8292ae42b7e1\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-kznnt" Oct 30 16:38:34.107298 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:34.107272 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c254bba7-008e-4724-80a8-8292ae42b7e1-istio-data\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-kznnt\" (UID: \"c254bba7-008e-4724-80a8-8292ae42b7e1\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-kznnt" Oct 30 16:38:34.107397 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:34.107273 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c254bba7-008e-4724-80a8-8292ae42b7e1-workload-certs\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-kznnt\" (UID: \"c254bba7-008e-4724-80a8-8292ae42b7e1\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-kznnt" Oct 30 16:38:34.107434 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:34.107417 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c254bba7-008e-4724-80a8-8292ae42b7e1-istiod-ca-cert\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-kznnt\" (UID: \"c254bba7-008e-4724-80a8-8292ae42b7e1\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-kznnt" Oct 30 16:38:34.109301 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:34.109278 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c254bba7-008e-4724-80a8-8292ae42b7e1-istio-envoy\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-kznnt\" (UID: \"c254bba7-008e-4724-80a8-8292ae42b7e1\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-kznnt" Oct 30 16:38:34.109626 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:34.109604 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c254bba7-008e-4724-80a8-8292ae42b7e1-istio-podinfo\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-kznnt\" (UID: \"c254bba7-008e-4724-80a8-8292ae42b7e1\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-kznnt" Oct 30 16:38:34.129707 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:34.129684 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c254bba7-008e-4724-80a8-8292ae42b7e1-istio-token\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-kznnt\" (UID: \"c254bba7-008e-4724-80a8-8292ae42b7e1\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-kznnt" Oct 30 16:38:34.129815 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:34.129758 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dk6pt\" (UniqueName: \"kubernetes.io/projected/c254bba7-008e-4724-80a8-8292ae42b7e1-kube-api-access-dk6pt\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-kznnt\" (UID: \"c254bba7-008e-4724-80a8-8292ae42b7e1\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-kznnt" Oct 30 16:38:34.308265 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:34.308189 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-kznnt" Oct 30 16:38:34.630249 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:34.630224 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-kznnt"] Oct 30 16:38:34.632600 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:38:34.632558 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc254bba7_008e_4724_80a8_8292ae42b7e1.slice/crio-8de520285b3e0900170f1e2e37791b4f279bec3b841969b612f0e3a1d88c0206 WatchSource:0}: Error finding container 8de520285b3e0900170f1e2e37791b4f279bec3b841969b612f0e3a1d88c0206: Status 404 returned error can't find the container with id 8de520285b3e0900170f1e2e37791b4f279bec3b841969b612f0e3a1d88c0206 Oct 30 16:38:34.635002 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:34.634972 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:38:34.635079 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:34.635036 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:38:34.635116 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:34.635080 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:38:34.826897 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:34.826865 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-kznnt" event={"ID":"c254bba7-008e-4724-80a8-8292ae42b7e1","Type":"ContainerStarted","Data":"cf30a6a0225cb105f4c7c0096f168d0f62a4cdbc453dbb9fe995055af8d31aef"} Oct 30 16:38:34.827063 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:34.826918 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-kznnt" event={"ID":"c254bba7-008e-4724-80a8-8292ae42b7e1","Type":"ContainerStarted","Data":"8de520285b3e0900170f1e2e37791b4f279bec3b841969b612f0e3a1d88c0206"} Oct 30 16:38:34.875628 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:34.875554 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-kznnt" podStartSLOduration=1.875538094 podStartE2EDuration="1.875538094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:38:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:38:34.871713383 +0000 UTC m=+1070.399048602" watchObservedRunningTime="2026-10-30 16:38:34.875538094 +0000 UTC m=+1070.402873381" Oct 30 16:38:35.275607 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:35.274404 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cdwcj"] Oct 30 16:38:35.309187 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:35.309159 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-kznnt" Oct 30 16:38:35.311097 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:35.311067 5052 patch_prober.go:29] interesting pod/e2e-worker-w3-0b2348-openshift-default-85679bc47c-kznnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 16:38:35.311215 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:35.311114 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-kznnt" podUID="c254bba7-008e-4724-80a8-8292ae42b7e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 16:38:36.309463 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:36.309002 5052 patch_prober.go:29] interesting pod/e2e-worker-w3-0b2348-openshift-default-85679bc47c-kznnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 16:38:36.309463 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:36.309063 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-kznnt" podUID="c254bba7-008e-4724-80a8-8292ae42b7e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 16:38:37.309164 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:37.309113 5052 patch_prober.go:29] interesting pod/e2e-worker-w3-0b2348-openshift-default-85679bc47c-kznnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 16:38:37.309519 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:37.309490 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-kznnt" podUID="c254bba7-008e-4724-80a8-8292ae42b7e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 16:38:37.678603 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:37.675420 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx" podUID="bd662f3a-659f-477c-b8d5-94c64288c0c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:37.678891 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:37.678852 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/distinct-2-w4-263f00-kserve-8464897cf5-q8ck8" podUID="15ed43ef-5cdd-489a-a70e-063239822ff3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:37.876756 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:37.876702 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/distinct-w5-967fac-kserve-69cbcccd8b-b85bn" podUID="8f6b9354-378f-4451-9ff7-be7f7fec12f2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:38.343606 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:38.339988 5052 patch_prober.go:29] interesting pod/e2e-worker-w3-0b2348-openshift-default-85679bc47c-kznnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 16:38:38.379606 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:38.354988 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-kznnt" podUID="c254bba7-008e-4724-80a8-8292ae42b7e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 16:38:38.527606 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:38.516787 5052 patch_prober.go:29] interesting pod/e2e-worker-w3-0b2348-openshift-default-85679bc47c-46brl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:38:38.527606 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:38.516855 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-46brl" podUID="4b8235a9-5724-4354-ad7b-eaef26b3966f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:38.677834 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:38.677790 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx" podUID="bd662f3a-659f-477c-b8d5-94c64288c0c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:38.679264 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:38.679240 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-967fac/distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx" Oct 30 16:38:38.679774 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:38.679424 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/distinct-2-w4-263f00-kserve-8464897cf5-q8ck8" podUID="15ed43ef-5cdd-489a-a70e-063239822ff3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:38.680247 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:38.680228 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-263f00/distinct-2-w4-263f00-kserve-8464897cf5-q8ck8" Oct 30 16:38:38.918622 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:38.881687 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/distinct-w5-967fac-kserve-69cbcccd8b-b85bn" podUID="8f6b9354-378f-4451-9ff7-be7f7fec12f2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:38.918622 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:38.882449 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-967fac/distinct-w5-967fac-kserve-69cbcccd8b-b85bn" Oct 30 16:38:39.190939 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:39.190883 5052 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-263f00/distinct-2-w4-263f00-kserve-8464897cf5-q8ck8" podUID="15ed43ef-5cdd-489a-a70e-063239822ff3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:39.229448 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:39.229418 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-263f00/distinct-2-w4-263f00-kserve-8464897cf5-q8ck8" Oct 30 16:38:39.458838 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:39.458729 5052 patch_prober.go:29] interesting pod/e2e-worker-w3-0b2348-openshift-default-85679bc47c-kznnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 16:38:39.459399 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:39.459354 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-kznnt" podUID="c254bba7-008e-4724-80a8-8292ae42b7e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 16:38:39.495673 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:39.485353 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-967fac/distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx" Oct 30 16:38:39.894678 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:39.893048 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/distinct-w5-967fac-kserve-69cbcccd8b-b85bn" podUID="8f6b9354-378f-4451-9ff7-be7f7fec12f2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:40.007744 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:40.007718 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-967fac/distinct-w5-967fac-kserve-69cbcccd8b-b85bn" Oct 30 16:38:40.308989 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:40.308936 5052 patch_prober.go:29] interesting pod/e2e-worker-w3-0b2348-openshift-default-85679bc47c-kznnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 16:38:40.333441 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:40.333391 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-kznnt" podUID="c254bba7-008e-4724-80a8-8292ae42b7e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 16:38:41.309529 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:41.309495 5052 patch_prober.go:29] interesting pod/e2e-worker-w3-0b2348-openshift-default-85679bc47c-kznnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 16:38:41.309960 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:41.309548 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-kznnt" podUID="c254bba7-008e-4724-80a8-8292ae42b7e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 16:38:42.309160 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:42.309123 5052 patch_prober.go:29] interesting pod/e2e-worker-w3-0b2348-openshift-default-85679bc47c-kznnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 16:38:42.309330 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:42.309200 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-kznnt" podUID="c254bba7-008e-4724-80a8-8292ae42b7e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 16:38:44.309651 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:44.309607 5052 patch_prober.go:29] interesting pod/e2e-worker-w3-0b2348-openshift-default-85679bc47c-kznnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:38:44.310041 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:44.309685 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-kznnt" podUID="c254bba7-008e-4724-80a8-8292ae42b7e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:45.052794 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:45.052709 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-kznnt" Oct 30 16:38:45.053231 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:45.053210 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-kznnt" Oct 30 16:38:45.054290 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:45.054270 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-kznnt" Oct 30 16:38:48.881784 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:38:48.881740 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 16:38:57.607836 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:38:57.607806 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cdwcj"] Oct 30 16:39:00.494860 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:39:00.493672 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx" podUID="bd662f3a-659f-477c-b8d5-94c64288c0c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:00.935602 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:39:00.931717 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/distinct-w5-967fac-kserve-69cbcccd8b-b85bn" podUID="8f6b9354-378f-4451-9ff7-be7f7fec12f2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:00.981605 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:39:00.964662 5052 patch_prober.go:29] interesting pod/e2e-worker-w4-263f00-openshift-default-b969d55bc-4frq9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:39:00.981605 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:39:00.964734 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-4frq9" podUID="8451a193-e5be-47eb-9e5b-75c5214d3abc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:01.109067 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:39:01.108702 5052 patch_prober.go:29] interesting pod/e2e-worker-w3-0b2348-openshift-default-85679bc47c-kznnt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:39:01.109067 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:39:01.108782 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-kznnt" podUID="c254bba7-008e-4724-80a8-8292ae42b7e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:01.238691 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:39:01.237697 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/distinct-2-w4-263f00-kserve-8464897cf5-q8ck8" podUID="15ed43ef-5cdd-489a-a70e-063239822ff3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:01.506946 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:39:01.496907 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx" podUID="bd662f3a-659f-477c-b8d5-94c64288c0c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:01.506946 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:39:01.505827 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-967fac/distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx" Oct 30 16:39:01.942149 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:39:01.942069 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-967fac/distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx" Oct 30 16:39:01.997104 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:39:01.996961 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/distinct-w5-967fac-kserve-69cbcccd8b-b85bn" podUID="8f6b9354-378f-4451-9ff7-be7f7fec12f2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:01.997805 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:39:01.997778 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-967fac/distinct-w5-967fac-kserve-69cbcccd8b-b85bn" Oct 30 16:39:02.022021 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:39:02.021993 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-967fac/distinct-w5-967fac-kserve-69cbcccd8b-b85bn" Oct 30 16:39:14.684762 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:39:14.684667 5052 patch_prober.go:29] interesting pod/e2e-worker-w5-967fac-openshift-default-dffc67487-v2jw7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:39:14.685568 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:39:14.685535 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-v2jw7" podUID="a40b021f-16e2-4821-ad8b-9a7fb6a4a58f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:18.841526 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:39:18.841488 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 16:39:28.928444 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:39:28.928411 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cdwcj"] Oct 30 16:39:48.842317 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:39:48.842266 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 16:39:50.192617 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:39:50.192561 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cdwcj"] Oct 30 16:39:52.843685 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:39:52.843632 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx" podUID="bd662f3a-659f-477c-b8d5-94c64288c0c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:52.874605 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:39:52.871126 5052 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-967fac/distinct-w5-967fac-kserve-69cbcccd8b-b85bn" podUID="8f6b9354-378f-4451-9ff7-be7f7fec12f2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": context deadline exceeded" Oct 30 16:39:53.023606 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:39:53.023044 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/distinct-w5-967fac-kserve-69cbcccd8b-b85bn" podUID="8f6b9354-378f-4451-9ff7-be7f7fec12f2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:53.990606 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:39:53.897877 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx" podUID="bd662f3a-659f-477c-b8d5-94c64288c0c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 16:39:54.051604 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:39:54.025823 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/distinct-w5-967fac-kserve-69cbcccd8b-b85bn" podUID="8f6b9354-378f-4451-9ff7-be7f7fec12f2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Oct 30 16:39:54.063609 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:39:54.052057 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-967fac/distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx" Oct 30 16:39:54.063609 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:39:54.052122 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-967fac/distinct-w5-967fac-kserve-69cbcccd8b-b85bn" Oct 30 16:39:54.263844 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:39:54.263746 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/distinct-2-w4-263f00-kserve-8464897cf5-q8ck8" podUID="15ed43ef-5cdd-489a-a70e-063239822ff3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 30 16:39:54.303312 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:39:54.301483 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-967fac/distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx" Oct 30 16:39:54.453787 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:39:54.452519 5052 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-967fac/distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx" podUID="bd662f3a-659f-477c-b8d5-94c64288c0c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:55.055560 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:39:55.052665 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/distinct-w5-967fac-kserve-69cbcccd8b-b85bn" podUID="8f6b9354-378f-4451-9ff7-be7f7fec12f2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:55.341672 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:39:55.322004 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx" podUID="bd662f3a-659f-477c-b8d5-94c64288c0c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:55.589188 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:39:55.583995 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-967fac/distinct-w5-967fac-kserve-69cbcccd8b-b85bn" Oct 30 16:40:04.356341 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:04.356306 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-v9r87"] Oct 30 16:40:04.360740 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:04.360720 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-v9r87" Oct 30 16:40:04.390248 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:04.390220 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-v9r87"] Oct 30 16:40:04.418434 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:04.418403 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20129ef0-4c2c-498f-9427-985db412bb2e-istio-envoy\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-v9r87\" (UID: \"20129ef0-4c2c-498f-9427-985db412bb2e\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-v9r87" Oct 30 16:40:04.418574 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:04.418457 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20129ef0-4c2c-498f-9427-985db412bb2e-workload-socket\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-v9r87\" (UID: \"20129ef0-4c2c-498f-9427-985db412bb2e\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-v9r87" Oct 30 16:40:04.418694 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:04.418630 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mj6sd\" (UniqueName: \"kubernetes.io/projected/20129ef0-4c2c-498f-9427-985db412bb2e-kube-api-access-mj6sd\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-v9r87\" (UID: \"20129ef0-4c2c-498f-9427-985db412bb2e\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-v9r87" Oct 30 16:40:04.418763 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:04.418734 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20129ef0-4c2c-498f-9427-985db412bb2e-workload-certs\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-v9r87\" (UID: \"20129ef0-4c2c-498f-9427-985db412bb2e\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-v9r87" Oct 30 16:40:04.418812 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:04.418778 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20129ef0-4c2c-498f-9427-985db412bb2e-credential-socket\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-v9r87\" (UID: \"20129ef0-4c2c-498f-9427-985db412bb2e\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-v9r87" Oct 30 16:40:04.418874 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:04.418847 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20129ef0-4c2c-498f-9427-985db412bb2e-istio-podinfo\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-v9r87\" (UID: \"20129ef0-4c2c-498f-9427-985db412bb2e\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-v9r87" Oct 30 16:40:04.419007 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:04.418990 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20129ef0-4c2c-498f-9427-985db412bb2e-istio-data\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-v9r87\" (UID: \"20129ef0-4c2c-498f-9427-985db412bb2e\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-v9r87" Oct 30 16:40:04.419046 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:04.419036 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20129ef0-4c2c-498f-9427-985db412bb2e-istiod-ca-cert\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-v9r87\" (UID: \"20129ef0-4c2c-498f-9427-985db412bb2e\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-v9r87" Oct 30 16:40:04.419159 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:04.419142 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20129ef0-4c2c-498f-9427-985db412bb2e-istio-token\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-v9r87\" (UID: \"20129ef0-4c2c-498f-9427-985db412bb2e\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-v9r87" Oct 30 16:40:04.520671 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:04.520637 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20129ef0-4c2c-498f-9427-985db412bb2e-istio-data\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-v9r87\" (UID: \"20129ef0-4c2c-498f-9427-985db412bb2e\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-v9r87" Oct 30 16:40:04.520835 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:04.520701 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20129ef0-4c2c-498f-9427-985db412bb2e-istiod-ca-cert\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-v9r87\" (UID: \"20129ef0-4c2c-498f-9427-985db412bb2e\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-v9r87" Oct 30 16:40:04.520835 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:04.520771 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20129ef0-4c2c-498f-9427-985db412bb2e-istio-token\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-v9r87\" (UID: \"20129ef0-4c2c-498f-9427-985db412bb2e\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-v9r87" Oct 30 16:40:04.520957 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:04.520862 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20129ef0-4c2c-498f-9427-985db412bb2e-istio-envoy\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-v9r87\" (UID: \"20129ef0-4c2c-498f-9427-985db412bb2e\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-v9r87" Oct 30 16:40:04.520957 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:04.520906 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20129ef0-4c2c-498f-9427-985db412bb2e-workload-socket\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-v9r87\" (UID: \"20129ef0-4c2c-498f-9427-985db412bb2e\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-v9r87" Oct 30 16:40:04.520957 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:04.520945 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mj6sd\" (UniqueName: \"kubernetes.io/projected/20129ef0-4c2c-498f-9427-985db412bb2e-kube-api-access-mj6sd\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-v9r87\" (UID: \"20129ef0-4c2c-498f-9427-985db412bb2e\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-v9r87" Oct 30 16:40:04.521114 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:04.521007 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20129ef0-4c2c-498f-9427-985db412bb2e-workload-certs\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-v9r87\" (UID: \"20129ef0-4c2c-498f-9427-985db412bb2e\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-v9r87" Oct 30 16:40:04.521114 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:04.521015 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20129ef0-4c2c-498f-9427-985db412bb2e-istio-data\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-v9r87\" (UID: \"20129ef0-4c2c-498f-9427-985db412bb2e\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-v9r87" Oct 30 16:40:04.521114 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:04.521052 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20129ef0-4c2c-498f-9427-985db412bb2e-credential-socket\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-v9r87\" (UID: \"20129ef0-4c2c-498f-9427-985db412bb2e\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-v9r87" Oct 30 16:40:04.521114 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:04.521084 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20129ef0-4c2c-498f-9427-985db412bb2e-istio-podinfo\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-v9r87\" (UID: \"20129ef0-4c2c-498f-9427-985db412bb2e\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-v9r87" Oct 30 16:40:04.521451 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:04.521424 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20129ef0-4c2c-498f-9427-985db412bb2e-workload-certs\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-v9r87\" (UID: \"20129ef0-4c2c-498f-9427-985db412bb2e\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-v9r87" Oct 30 16:40:04.521545 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:04.521511 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20129ef0-4c2c-498f-9427-985db412bb2e-credential-socket\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-v9r87\" (UID: \"20129ef0-4c2c-498f-9427-985db412bb2e\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-v9r87" Oct 30 16:40:04.521621 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:04.521541 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20129ef0-4c2c-498f-9427-985db412bb2e-istiod-ca-cert\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-v9r87\" (UID: \"20129ef0-4c2c-498f-9427-985db412bb2e\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-v9r87" Oct 30 16:40:04.521747 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:04.521723 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20129ef0-4c2c-498f-9427-985db412bb2e-workload-socket\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-v9r87\" (UID: \"20129ef0-4c2c-498f-9427-985db412bb2e\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-v9r87" Oct 30 16:40:04.523938 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:04.523916 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20129ef0-4c2c-498f-9427-985db412bb2e-istio-envoy\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-v9r87\" (UID: \"20129ef0-4c2c-498f-9427-985db412bb2e\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-v9r87" Oct 30 16:40:04.524288 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:04.524270 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20129ef0-4c2c-498f-9427-985db412bb2e-istio-podinfo\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-v9r87\" (UID: \"20129ef0-4c2c-498f-9427-985db412bb2e\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-v9r87" Oct 30 16:40:04.543886 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:04.543860 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20129ef0-4c2c-498f-9427-985db412bb2e-istio-token\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-v9r87\" (UID: \"20129ef0-4c2c-498f-9427-985db412bb2e\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-v9r87" Oct 30 16:40:04.544349 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:04.544327 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mj6sd\" (UniqueName: \"kubernetes.io/projected/20129ef0-4c2c-498f-9427-985db412bb2e-kube-api-access-mj6sd\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-v9r87\" (UID: \"20129ef0-4c2c-498f-9427-985db412bb2e\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-v9r87" Oct 30 16:40:04.674806 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:04.674775 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-v9r87" Oct 30 16:40:04.997908 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:04.997884 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-v9r87"] Oct 30 16:40:05.000369 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:40:05.000345 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20129ef0_4c2c_498f_9427_985db412bb2e.slice/crio-05c597d856e44a702ec3f1b620c1e2ab3c3d078b7fb639e4684d275efb2ad377 WatchSource:0}: Error finding container 05c597d856e44a702ec3f1b620c1e2ab3c3d078b7fb639e4684d275efb2ad377: Status 404 returned error can't find the container with id 05c597d856e44a702ec3f1b620c1e2ab3c3d078b7fb639e4684d275efb2ad377 Oct 30 16:40:05.002884 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:05.002854 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:40:05.002951 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:05.002917 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:40:05.002951 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:05.002945 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:40:05.923871 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:05.923833 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-v9r87" event={"ID":"20129ef0-4c2c-498f-9427-985db412bb2e","Type":"ContainerStarted","Data":"95c15eab4330dec0ed5761be3ae5327a9fe786d4cb463872766b7d5204f8fd78"} Oct 30 16:40:05.924374 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:05.923896 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-v9r87" event={"ID":"20129ef0-4c2c-498f-9427-985db412bb2e","Type":"ContainerStarted","Data":"05c597d856e44a702ec3f1b620c1e2ab3c3d078b7fb639e4684d275efb2ad377"} Oct 30 16:40:06.002130 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:06.002070 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-v9r87" podStartSLOduration=2.002053394 podStartE2EDuration="2.002053394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:40:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:40:05.98825989 +0000 UTC m=+1161.515595114" watchObservedRunningTime="2026-10-30 16:40:06.002053394 +0000 UTC m=+1161.529388613" Oct 30 16:40:06.708786 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:06.708745 5052 patch_prober.go:29] interesting pod/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-v9r87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 16:40:06.709823 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:06.709779 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-v9r87" podUID="20129ef0-4c2c-498f-9427-985db412bb2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 16:40:06.744607 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:06.743876 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-v9r87" Oct 30 16:40:07.678604 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:07.678179 5052 patch_prober.go:29] interesting pod/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-v9r87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 16:40:07.678604 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:07.678276 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-v9r87" podUID="20129ef0-4c2c-498f-9427-985db412bb2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 16:40:08.681605 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:08.676669 5052 patch_prober.go:29] interesting pod/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-v9r87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 16:40:08.681605 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:08.676736 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-v9r87" podUID="20129ef0-4c2c-498f-9427-985db412bb2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 16:40:09.060662 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:09.056618 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/distinct-w5-967fac-kserve-69cbcccd8b-b85bn" podUID="8f6b9354-378f-4451-9ff7-be7f7fec12f2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:09.231606 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:09.231455 5052 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-263f00/distinct-2-w4-263f00-kserve-8464897cf5-q8ck8" podUID="15ed43ef-5cdd-489a-a70e-063239822ff3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:09.232277 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:09.232235 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/distinct-2-w4-263f00-kserve-8464897cf5-q8ck8" podUID="15ed43ef-5cdd-489a-a70e-063239822ff3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:09.324660 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:09.310802 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx" podUID="bd662f3a-659f-477c-b8d5-94c64288c0c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:09.769777 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:09.743089 5052 patch_prober.go:29] interesting pod/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-v9r87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 16:40:09.769777 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:09.743152 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-v9r87" podUID="20129ef0-4c2c-498f-9427-985db412bb2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 16:40:10.137658 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:10.112795 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/distinct-w5-967fac-kserve-69cbcccd8b-b85bn" podUID="8f6b9354-378f-4451-9ff7-be7f7fec12f2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:10.137658 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:10.124331 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-967fac/distinct-w5-967fac-kserve-69cbcccd8b-b85bn" Oct 30 16:40:10.251970 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:10.251928 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/distinct-2-w4-263f00-kserve-8464897cf5-q8ck8" podUID="15ed43ef-5cdd-489a-a70e-063239822ff3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:10.300603 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:10.280138 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-263f00/distinct-2-w4-263f00-kserve-8464897cf5-q8ck8" Oct 30 16:40:10.327612 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:10.313884 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx" podUID="bd662f3a-659f-477c-b8d5-94c64288c0c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 16:40:10.327612 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:10.314611 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-967fac/distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx" Oct 30 16:40:10.533259 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:10.532884 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-263f00/distinct-2-w4-263f00-kserve-8464897cf5-q8ck8" Oct 30 16:40:10.599010 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:10.598976 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-967fac/distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx" Oct 30 16:40:10.768702 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:10.697270 5052 patch_prober.go:29] interesting pod/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-v9r87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 16:40:10.768702 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:10.764025 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-v9r87" podUID="20129ef0-4c2c-498f-9427-985db412bb2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 16:40:10.908950 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:10.908918 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-967fac/distinct-w5-967fac-kserve-69cbcccd8b-b85bn" Oct 30 16:40:11.675366 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:11.675332 5052 patch_prober.go:29] interesting pod/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-v9r87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 16:40:11.675568 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:11.675417 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-v9r87" podUID="20129ef0-4c2c-498f-9427-985db412bb2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 16:40:12.675113 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:12.675077 5052 patch_prober.go:29] interesting pod/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-v9r87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 16:40:12.675526 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:12.675131 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-v9r87" podUID="20129ef0-4c2c-498f-9427-985db412bb2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 16:40:13.676110 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:13.676074 5052 patch_prober.go:29] interesting pod/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-v9r87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 16:40:13.676660 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:13.676137 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-v9r87" podUID="20129ef0-4c2c-498f-9427-985db412bb2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 16:40:14.928990 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:14.928952 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-v9r87" Oct 30 16:40:14.929411 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:14.929272 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-v9r87" Oct 30 16:40:14.930551 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:14.930527 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-v9r87" Oct 30 16:40:18.840385 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:40:18.840353 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 16:40:18.898994 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:18.898967 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hkcvj"] Oct 30 16:40:18.903456 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:18.903434 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hkcvj" Oct 30 16:40:18.924048 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:18.924020 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hkcvj"] Oct 30 16:40:18.979112 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:18.979065 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/545986bf-3dc2-4204-9fd1-ac7085df215b-istiod-ca-cert\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-hkcvj\" (UID: \"545986bf-3dc2-4204-9fd1-ac7085df215b\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hkcvj" Oct 30 16:40:18.979112 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:18.979115 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/545986bf-3dc2-4204-9fd1-ac7085df215b-credential-socket\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-hkcvj\" (UID: \"545986bf-3dc2-4204-9fd1-ac7085df215b\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hkcvj" Oct 30 16:40:18.979358 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:18.979331 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/545986bf-3dc2-4204-9fd1-ac7085df215b-istio-podinfo\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-hkcvj\" (UID: \"545986bf-3dc2-4204-9fd1-ac7085df215b\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hkcvj" Oct 30 16:40:18.979409 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:18.979392 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/545986bf-3dc2-4204-9fd1-ac7085df215b-istio-token\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-hkcvj\" (UID: \"545986bf-3dc2-4204-9fd1-ac7085df215b\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hkcvj" Oct 30 16:40:18.979538 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:18.979522 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/545986bf-3dc2-4204-9fd1-ac7085df215b-istio-data\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-hkcvj\" (UID: \"545986bf-3dc2-4204-9fd1-ac7085df215b\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hkcvj" Oct 30 16:40:18.979642 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:18.979626 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/545986bf-3dc2-4204-9fd1-ac7085df215b-workload-certs\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-hkcvj\" (UID: \"545986bf-3dc2-4204-9fd1-ac7085df215b\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hkcvj" Oct 30 16:40:18.979696 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:18.979682 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/545986bf-3dc2-4204-9fd1-ac7085df215b-istio-envoy\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-hkcvj\" (UID: \"545986bf-3dc2-4204-9fd1-ac7085df215b\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hkcvj" Oct 30 16:40:18.979776 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:18.979756 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rm22w\" (UniqueName: \"kubernetes.io/projected/545986bf-3dc2-4204-9fd1-ac7085df215b-kube-api-access-rm22w\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-hkcvj\" (UID: \"545986bf-3dc2-4204-9fd1-ac7085df215b\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hkcvj" Oct 30 16:40:18.979873 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:18.979855 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/545986bf-3dc2-4204-9fd1-ac7085df215b-workload-socket\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-hkcvj\" (UID: \"545986bf-3dc2-4204-9fd1-ac7085df215b\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hkcvj" Oct 30 16:40:19.098949 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:19.084886 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/545986bf-3dc2-4204-9fd1-ac7085df215b-istio-podinfo\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-hkcvj\" (UID: \"545986bf-3dc2-4204-9fd1-ac7085df215b\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hkcvj" Oct 30 16:40:19.098949 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:19.089434 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/545986bf-3dc2-4204-9fd1-ac7085df215b-istio-token\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-hkcvj\" (UID: \"545986bf-3dc2-4204-9fd1-ac7085df215b\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hkcvj" Oct 30 16:40:19.098949 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:19.089555 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/545986bf-3dc2-4204-9fd1-ac7085df215b-istio-data\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-hkcvj\" (UID: \"545986bf-3dc2-4204-9fd1-ac7085df215b\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hkcvj" Oct 30 16:40:19.098949 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:19.089636 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/545986bf-3dc2-4204-9fd1-ac7085df215b-workload-certs\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-hkcvj\" (UID: \"545986bf-3dc2-4204-9fd1-ac7085df215b\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hkcvj" Oct 30 16:40:19.098949 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:19.089691 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/545986bf-3dc2-4204-9fd1-ac7085df215b-istio-envoy\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-hkcvj\" (UID: \"545986bf-3dc2-4204-9fd1-ac7085df215b\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hkcvj" Oct 30 16:40:19.098949 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:19.089731 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rm22w\" (UniqueName: \"kubernetes.io/projected/545986bf-3dc2-4204-9fd1-ac7085df215b-kube-api-access-rm22w\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-hkcvj\" (UID: \"545986bf-3dc2-4204-9fd1-ac7085df215b\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hkcvj" Oct 30 16:40:19.098949 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:19.089827 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/545986bf-3dc2-4204-9fd1-ac7085df215b-workload-socket\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-hkcvj\" (UID: \"545986bf-3dc2-4204-9fd1-ac7085df215b\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hkcvj" Oct 30 16:40:19.098949 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:19.089866 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/545986bf-3dc2-4204-9fd1-ac7085df215b-istiod-ca-cert\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-hkcvj\" (UID: \"545986bf-3dc2-4204-9fd1-ac7085df215b\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hkcvj" Oct 30 16:40:19.098949 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:19.089900 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/545986bf-3dc2-4204-9fd1-ac7085df215b-credential-socket\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-hkcvj\" (UID: \"545986bf-3dc2-4204-9fd1-ac7085df215b\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hkcvj" Oct 30 16:40:19.098949 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:19.090488 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/545986bf-3dc2-4204-9fd1-ac7085df215b-credential-socket\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-hkcvj\" (UID: \"545986bf-3dc2-4204-9fd1-ac7085df215b\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hkcvj" Oct 30 16:40:19.098949 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:19.091708 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/545986bf-3dc2-4204-9fd1-ac7085df215b-workload-certs\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-hkcvj\" (UID: \"545986bf-3dc2-4204-9fd1-ac7085df215b\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hkcvj" Oct 30 16:40:19.098949 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:19.092302 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/545986bf-3dc2-4204-9fd1-ac7085df215b-workload-socket\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-hkcvj\" (UID: \"545986bf-3dc2-4204-9fd1-ac7085df215b\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hkcvj" Oct 30 16:40:19.098949 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:19.096335 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/545986bf-3dc2-4204-9fd1-ac7085df215b-istio-envoy\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-hkcvj\" (UID: \"545986bf-3dc2-4204-9fd1-ac7085df215b\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hkcvj" Oct 30 16:40:19.098949 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:19.097331 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/545986bf-3dc2-4204-9fd1-ac7085df215b-istio-podinfo\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-hkcvj\" (UID: \"545986bf-3dc2-4204-9fd1-ac7085df215b\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hkcvj" Oct 30 16:40:19.098949 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:19.098515 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/545986bf-3dc2-4204-9fd1-ac7085df215b-istiod-ca-cert\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-hkcvj\" (UID: \"545986bf-3dc2-4204-9fd1-ac7085df215b\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hkcvj" Oct 30 16:40:19.098949 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:19.098838 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/545986bf-3dc2-4204-9fd1-ac7085df215b-istio-data\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-hkcvj\" (UID: \"545986bf-3dc2-4204-9fd1-ac7085df215b\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hkcvj" Oct 30 16:40:19.115391 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:19.115342 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rm22w\" (UniqueName: \"kubernetes.io/projected/545986bf-3dc2-4204-9fd1-ac7085df215b-kube-api-access-rm22w\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-hkcvj\" (UID: \"545986bf-3dc2-4204-9fd1-ac7085df215b\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hkcvj" Oct 30 16:40:19.115714 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:19.115692 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/545986bf-3dc2-4204-9fd1-ac7085df215b-istio-token\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-hkcvj\" (UID: \"545986bf-3dc2-4204-9fd1-ac7085df215b\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hkcvj" Oct 30 16:40:19.218386 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:19.218351 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hkcvj" Oct 30 16:40:19.541852 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:19.541824 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hkcvj"] Oct 30 16:40:19.544141 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:40:19.544111 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod545986bf_3dc2_4204_9fd1_ac7085df215b.slice/crio-aac02cdc261f7cffdd80ff723363b3f46bd8f5dc468b97a2fe4efa224955970f WatchSource:0}: Error finding container aac02cdc261f7cffdd80ff723363b3f46bd8f5dc468b97a2fe4efa224955970f: Status 404 returned error can't find the container with id aac02cdc261f7cffdd80ff723363b3f46bd8f5dc468b97a2fe4efa224955970f Oct 30 16:40:19.546697 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:19.546666 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:40:19.546780 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:19.546736 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:40:19.546780 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:19.546764 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:40:20.509908 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:20.509866 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hkcvj" event={"ID":"545986bf-3dc2-4204-9fd1-ac7085df215b","Type":"ContainerStarted","Data":"533bd54ab34a531074d3353f760a16a82f42ac44d0c44ba8d18b40240a57427f"} Oct 30 16:40:20.510344 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:20.509923 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hkcvj" event={"ID":"545986bf-3dc2-4204-9fd1-ac7085df215b","Type":"ContainerStarted","Data":"aac02cdc261f7cffdd80ff723363b3f46bd8f5dc468b97a2fe4efa224955970f"} Oct 30 16:40:20.585492 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:20.585433 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hkcvj" podStartSLOduration=2.585415654 podStartE2EDuration="2.585415654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:40:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:40:20.581499355 +0000 UTC m=+1176.108834575" watchObservedRunningTime="2026-10-30 16:40:20.585415654 +0000 UTC m=+1176.112750873" Oct 30 16:40:21.219032 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:21.218997 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hkcvj" Oct 30 16:40:21.221029 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:21.221001 5052 patch_prober.go:29] interesting pod/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hkcvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 16:40:21.221174 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:21.221048 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hkcvj" podUID="545986bf-3dc2-4204-9fd1-ac7085df215b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 16:40:21.759664 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:21.758767 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-750dcb/distinct-w2-750dcb-kserve-86bbdfcbb-bpn77"] Oct 30 16:40:21.763862 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:21.763840 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-750dcb/distinct-w2-750dcb-kserve-86bbdfcbb-bpn77" Oct 30 16:40:21.779112 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:21.779083 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-750dcb/distinct-w2-750dcb-kserve-86bbdfcbb-bpn77"] Oct 30 16:40:21.782504 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:21.781858 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-750dcb\"/\"openshift-service-ca.crt\"" Oct 30 16:40:21.782504 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:21.782169 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-750dcb\"/\"kube-root-ca.crt\"" Oct 30 16:40:21.820089 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:21.820059 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-750dcb\"/\"distinct-w2-750dcb-kserve-self-signed-certs\"" Oct 30 16:40:21.820270 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:21.820151 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-750dcb\"/\"default-dockercfg-wnttz\"" Oct 30 16:40:21.873392 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:21.873353 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c45190af-d795-4e13-8f2f-670b2c597658-dshm\") pod \"distinct-w2-750dcb-kserve-86bbdfcbb-bpn77\" (UID: \"c45190af-d795-4e13-8f2f-670b2c597658\") " pod="e2e-models-e2e-worker-w2-750dcb/distinct-w2-750dcb-kserve-86bbdfcbb-bpn77" Oct 30 16:40:21.873557 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:21.873470 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnl2p\" (UniqueName: \"kubernetes.io/projected/c45190af-d795-4e13-8f2f-670b2c597658-kube-api-access-rnl2p\") pod \"distinct-w2-750dcb-kserve-86bbdfcbb-bpn77\" (UID: \"c45190af-d795-4e13-8f2f-670b2c597658\") " pod="e2e-models-e2e-worker-w2-750dcb/distinct-w2-750dcb-kserve-86bbdfcbb-bpn77" Oct 30 16:40:21.873644 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:21.873554 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c45190af-d795-4e13-8f2f-670b2c597658-home\") pod \"distinct-w2-750dcb-kserve-86bbdfcbb-bpn77\" (UID: \"c45190af-d795-4e13-8f2f-670b2c597658\") " pod="e2e-models-e2e-worker-w2-750dcb/distinct-w2-750dcb-kserve-86bbdfcbb-bpn77" Oct 30 16:40:21.873644 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:21.873604 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c45190af-d795-4e13-8f2f-670b2c597658-model-cache\") pod \"distinct-w2-750dcb-kserve-86bbdfcbb-bpn77\" (UID: \"c45190af-d795-4e13-8f2f-670b2c597658\") " pod="e2e-models-e2e-worker-w2-750dcb/distinct-w2-750dcb-kserve-86bbdfcbb-bpn77" Oct 30 16:40:21.873776 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:21.873758 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c45190af-d795-4e13-8f2f-670b2c597658-tmp-dir\") pod \"distinct-w2-750dcb-kserve-86bbdfcbb-bpn77\" (UID: \"c45190af-d795-4e13-8f2f-670b2c597658\") " pod="e2e-models-e2e-worker-w2-750dcb/distinct-w2-750dcb-kserve-86bbdfcbb-bpn77" Oct 30 16:40:21.873814 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:21.873785 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c45190af-d795-4e13-8f2f-670b2c597658-tls-certs\") pod \"distinct-w2-750dcb-kserve-86bbdfcbb-bpn77\" (UID: \"c45190af-d795-4e13-8f2f-670b2c597658\") " pod="e2e-models-e2e-worker-w2-750dcb/distinct-w2-750dcb-kserve-86bbdfcbb-bpn77" Oct 30 16:40:21.979677 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:21.978745 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c45190af-d795-4e13-8f2f-670b2c597658-dshm\") pod \"distinct-w2-750dcb-kserve-86bbdfcbb-bpn77\" (UID: \"c45190af-d795-4e13-8f2f-670b2c597658\") " pod="e2e-models-e2e-worker-w2-750dcb/distinct-w2-750dcb-kserve-86bbdfcbb-bpn77" Oct 30 16:40:21.979677 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:21.978841 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnl2p\" (UniqueName: \"kubernetes.io/projected/c45190af-d795-4e13-8f2f-670b2c597658-kube-api-access-rnl2p\") pod \"distinct-w2-750dcb-kserve-86bbdfcbb-bpn77\" (UID: \"c45190af-d795-4e13-8f2f-670b2c597658\") " pod="e2e-models-e2e-worker-w2-750dcb/distinct-w2-750dcb-kserve-86bbdfcbb-bpn77" Oct 30 16:40:21.979677 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:21.978887 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c45190af-d795-4e13-8f2f-670b2c597658-home\") pod \"distinct-w2-750dcb-kserve-86bbdfcbb-bpn77\" (UID: \"c45190af-d795-4e13-8f2f-670b2c597658\") " pod="e2e-models-e2e-worker-w2-750dcb/distinct-w2-750dcb-kserve-86bbdfcbb-bpn77" Oct 30 16:40:21.979677 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:21.978907 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c45190af-d795-4e13-8f2f-670b2c597658-model-cache\") pod \"distinct-w2-750dcb-kserve-86bbdfcbb-bpn77\" (UID: \"c45190af-d795-4e13-8f2f-670b2c597658\") " pod="e2e-models-e2e-worker-w2-750dcb/distinct-w2-750dcb-kserve-86bbdfcbb-bpn77" Oct 30 16:40:21.979677 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:21.979041 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c45190af-d795-4e13-8f2f-670b2c597658-tmp-dir\") pod \"distinct-w2-750dcb-kserve-86bbdfcbb-bpn77\" (UID: \"c45190af-d795-4e13-8f2f-670b2c597658\") " pod="e2e-models-e2e-worker-w2-750dcb/distinct-w2-750dcb-kserve-86bbdfcbb-bpn77" Oct 30 16:40:21.979677 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:21.979065 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c45190af-d795-4e13-8f2f-670b2c597658-tls-certs\") pod \"distinct-w2-750dcb-kserve-86bbdfcbb-bpn77\" (UID: \"c45190af-d795-4e13-8f2f-670b2c597658\") " pod="e2e-models-e2e-worker-w2-750dcb/distinct-w2-750dcb-kserve-86bbdfcbb-bpn77" Oct 30 16:40:21.980823 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:21.980492 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c45190af-d795-4e13-8f2f-670b2c597658-home\") pod \"distinct-w2-750dcb-kserve-86bbdfcbb-bpn77\" (UID: \"c45190af-d795-4e13-8f2f-670b2c597658\") " pod="e2e-models-e2e-worker-w2-750dcb/distinct-w2-750dcb-kserve-86bbdfcbb-bpn77" Oct 30 16:40:21.998168 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:21.998133 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c45190af-d795-4e13-8f2f-670b2c597658-model-cache\") pod \"distinct-w2-750dcb-kserve-86bbdfcbb-bpn77\" (UID: \"c45190af-d795-4e13-8f2f-670b2c597658\") " pod="e2e-models-e2e-worker-w2-750dcb/distinct-w2-750dcb-kserve-86bbdfcbb-bpn77" Oct 30 16:40:21.998852 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:21.998825 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c45190af-d795-4e13-8f2f-670b2c597658-tmp-dir\") pod \"distinct-w2-750dcb-kserve-86bbdfcbb-bpn77\" (UID: \"c45190af-d795-4e13-8f2f-670b2c597658\") " pod="e2e-models-e2e-worker-w2-750dcb/distinct-w2-750dcb-kserve-86bbdfcbb-bpn77" Oct 30 16:40:22.048362 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:22.048283 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c45190af-d795-4e13-8f2f-670b2c597658-tls-certs\") pod \"distinct-w2-750dcb-kserve-86bbdfcbb-bpn77\" (UID: \"c45190af-d795-4e13-8f2f-670b2c597658\") " pod="e2e-models-e2e-worker-w2-750dcb/distinct-w2-750dcb-kserve-86bbdfcbb-bpn77" Oct 30 16:40:22.057464 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:22.057428 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnl2p\" (UniqueName: \"kubernetes.io/projected/c45190af-d795-4e13-8f2f-670b2c597658-kube-api-access-rnl2p\") pod \"distinct-w2-750dcb-kserve-86bbdfcbb-bpn77\" (UID: \"c45190af-d795-4e13-8f2f-670b2c597658\") " pod="e2e-models-e2e-worker-w2-750dcb/distinct-w2-750dcb-kserve-86bbdfcbb-bpn77" Oct 30 16:40:22.064040 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:22.063975 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c45190af-d795-4e13-8f2f-670b2c597658-dshm\") pod \"distinct-w2-750dcb-kserve-86bbdfcbb-bpn77\" (UID: \"c45190af-d795-4e13-8f2f-670b2c597658\") " pod="e2e-models-e2e-worker-w2-750dcb/distinct-w2-750dcb-kserve-86bbdfcbb-bpn77" Oct 30 16:40:22.088972 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:22.088888 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-750dcb/distinct-w2-750dcb-kserve-86bbdfcbb-bpn77" Oct 30 16:40:22.219767 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:22.219732 5052 patch_prober.go:29] interesting pod/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hkcvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 16:40:22.219949 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:22.219804 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hkcvj" podUID="545986bf-3dc2-4204-9fd1-ac7085df215b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 16:40:22.524380 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:22.524358 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-750dcb/distinct-w2-750dcb-kserve-86bbdfcbb-bpn77"] Oct 30 16:40:22.526207 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:40:22.526180 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc45190af_d795_4e13_8f2f_670b2c597658.slice/crio-7a10e73f1b0dd38922611c4e7282b93b247bb01afef8b735c63af8e18536dc43 WatchSource:0}: Error finding container 7a10e73f1b0dd38922611c4e7282b93b247bb01afef8b735c63af8e18536dc43: Status 404 returned error can't find the container with id 7a10e73f1b0dd38922611c4e7282b93b247bb01afef8b735c63af8e18536dc43 Oct 30 16:40:23.219451 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:23.219414 5052 patch_prober.go:29] interesting pod/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hkcvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 16:40:23.219883 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:23.219499 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hkcvj" podUID="545986bf-3dc2-4204-9fd1-ac7085df215b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 16:40:23.551379 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:23.551276 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-750dcb/distinct-w2-750dcb-kserve-86bbdfcbb-bpn77" event={"ID":"c45190af-d795-4e13-8f2f-670b2c597658","Type":"ContainerStarted","Data":"947e5ececd27de1adc121ca653afa4e886b8756c0317abac23b33b2f748435c6"} Oct 30 16:40:23.551646 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:23.551616 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-750dcb/distinct-w2-750dcb-kserve-86bbdfcbb-bpn77" event={"ID":"c45190af-d795-4e13-8f2f-670b2c597658","Type":"ContainerStarted","Data":"7a10e73f1b0dd38922611c4e7282b93b247bb01afef8b735c63af8e18536dc43"} Oct 30 16:40:23.595075 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:23.595022 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-750dcb/distinct-w2-750dcb-kserve-86bbdfcbb-bpn77" podStartSLOduration=2.595004736 podStartE2EDuration="2.595004736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:40:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:40:23.591824512 +0000 UTC m=+1179.119159758" watchObservedRunningTime="2026-10-30 16:40:23.595004736 +0000 UTC m=+1179.122339963" Oct 30 16:40:24.090297 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:24.090264 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-750dcb/distinct-w2-750dcb-kserve-86bbdfcbb-bpn77" Oct 30 16:40:24.218725 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:24.218692 5052 patch_prober.go:29] interesting pod/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hkcvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 16:40:24.218901 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:24.218745 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hkcvj" podUID="545986bf-3dc2-4204-9fd1-ac7085df215b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 16:40:25.218742 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:25.218709 5052 patch_prober.go:29] interesting pod/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hkcvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 16:40:25.219135 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:25.218764 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hkcvj" podUID="545986bf-3dc2-4204-9fd1-ac7085df215b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 16:40:25.814445 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:25.814405 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cdwcj"] Oct 30 16:40:26.392355 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:26.392319 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hkcvj" Oct 30 16:40:26.393169 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:26.393141 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hkcvj" Oct 30 16:40:26.394124 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:26.394092 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hkcvj" Oct 30 16:40:28.584743 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:28.584696 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/distinct-2-w4-263f00-kserve-8464897cf5-q8ck8" podUID="15ed43ef-5cdd-489a-a70e-063239822ff3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:28.622607 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:28.618738 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx" podUID="bd662f3a-659f-477c-b8d5-94c64288c0c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 16:40:29.007607 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:29.006682 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/distinct-w5-967fac-kserve-69cbcccd8b-b85bn" podUID="8f6b9354-378f-4451-9ff7-be7f7fec12f2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:29.202605 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:29.201662 5052 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-263f00/distinct-2-w4-263f00-kserve-8464897cf5-q8ck8" podUID="15ed43ef-5cdd-489a-a70e-063239822ff3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:29.602605 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:29.602044 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/distinct-2-w4-263f00-kserve-8464897cf5-q8ck8" podUID="15ed43ef-5cdd-489a-a70e-063239822ff3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:29.603128 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:29.602833 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-263f00/distinct-2-w4-263f00-kserve-8464897cf5-q8ck8" Oct 30 16:40:29.671752 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:29.662268 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx" podUID="bd662f3a-659f-477c-b8d5-94c64288c0c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 16:40:29.671752 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:29.662999 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-967fac/distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx" Oct 30 16:40:30.023625 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:30.011882 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/distinct-w5-967fac-kserve-69cbcccd8b-b85bn" podUID="8f6b9354-378f-4451-9ff7-be7f7fec12f2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:30.023625 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:30.012754 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-967fac/distinct-w5-967fac-kserve-69cbcccd8b-b85bn" Oct 30 16:40:30.382757 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:30.382650 5052 patch_prober.go:29] interesting pod/e2e-worker-w4-263f00-openshift-default-b969d55bc-55x7p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:40:30.383286 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:30.382991 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-55x7p" podUID="52ba3acd-4de6-4ae2-8a20-49bf7f966ab0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:30.435125 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:30.434818 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-263f00/distinct-2-w4-263f00-kserve-8464897cf5-q8ck8" Oct 30 16:40:30.436903 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:30.436875 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-967fac/distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx" Oct 30 16:40:30.525374 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:30.525328 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-967fac/distinct-w5-967fac-kserve-69cbcccd8b-b85bn" Oct 30 16:40:32.096298 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:32.095331 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-750dcb/distinct-w2-750dcb-kserve-86bbdfcbb-bpn77" Oct 30 16:40:32.112149 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:32.112121 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-750dcb/distinct-w2-750dcb-kserve-86bbdfcbb-bpn77" Oct 30 16:40:32.954312 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:32.954284 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-750dcb/distinct-w2-750dcb-kserve-86bbdfcbb-bpn77" Oct 30 16:40:34.068569 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:34.068537 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-8cxdf"] Oct 30 16:40:34.073282 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:34.073261 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-8cxdf" Oct 30 16:40:34.116859 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:34.116822 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-8cxdf"] Oct 30 16:40:34.174996 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:34.174954 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd213278-0e61-4779-a644-abfed99fbb9a-istio-data\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-8cxdf\" (UID: \"dd213278-0e61-4779-a644-abfed99fbb9a\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-8cxdf" Oct 30 16:40:34.175166 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:34.175017 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crjgx\" (UniqueName: \"kubernetes.io/projected/dd213278-0e61-4779-a644-abfed99fbb9a-kube-api-access-crjgx\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-8cxdf\" (UID: \"dd213278-0e61-4779-a644-abfed99fbb9a\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-8cxdf" Oct 30 16:40:34.175166 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:34.175132 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd213278-0e61-4779-a644-abfed99fbb9a-workload-socket\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-8cxdf\" (UID: \"dd213278-0e61-4779-a644-abfed99fbb9a\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-8cxdf" Oct 30 16:40:34.175240 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:34.175181 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd213278-0e61-4779-a644-abfed99fbb9a-istiod-ca-cert\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-8cxdf\" (UID: \"dd213278-0e61-4779-a644-abfed99fbb9a\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-8cxdf" Oct 30 16:40:34.175240 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:34.175203 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd213278-0e61-4779-a644-abfed99fbb9a-credential-socket\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-8cxdf\" (UID: \"dd213278-0e61-4779-a644-abfed99fbb9a\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-8cxdf" Oct 30 16:40:34.175304 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:34.175289 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd213278-0e61-4779-a644-abfed99fbb9a-istio-token\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-8cxdf\" (UID: \"dd213278-0e61-4779-a644-abfed99fbb9a\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-8cxdf" Oct 30 16:40:34.175440 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:34.175418 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd213278-0e61-4779-a644-abfed99fbb9a-workload-certs\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-8cxdf\" (UID: \"dd213278-0e61-4779-a644-abfed99fbb9a\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-8cxdf" Oct 30 16:40:34.175492 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:34.175481 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd213278-0e61-4779-a644-abfed99fbb9a-istio-envoy\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-8cxdf\" (UID: \"dd213278-0e61-4779-a644-abfed99fbb9a\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-8cxdf" Oct 30 16:40:34.175553 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:34.175540 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd213278-0e61-4779-a644-abfed99fbb9a-istio-podinfo\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-8cxdf\" (UID: \"dd213278-0e61-4779-a644-abfed99fbb9a\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-8cxdf" Oct 30 16:40:34.276707 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:34.276678 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd213278-0e61-4779-a644-abfed99fbb9a-istio-data\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-8cxdf\" (UID: \"dd213278-0e61-4779-a644-abfed99fbb9a\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-8cxdf" Oct 30 16:40:34.276887 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:34.276720 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crjgx\" (UniqueName: \"kubernetes.io/projected/dd213278-0e61-4779-a644-abfed99fbb9a-kube-api-access-crjgx\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-8cxdf\" (UID: \"dd213278-0e61-4779-a644-abfed99fbb9a\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-8cxdf" Oct 30 16:40:34.276887 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:34.276745 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd213278-0e61-4779-a644-abfed99fbb9a-workload-socket\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-8cxdf\" (UID: \"dd213278-0e61-4779-a644-abfed99fbb9a\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-8cxdf" Oct 30 16:40:34.276887 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:34.276765 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd213278-0e61-4779-a644-abfed99fbb9a-istiod-ca-cert\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-8cxdf\" (UID: \"dd213278-0e61-4779-a644-abfed99fbb9a\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-8cxdf" Oct 30 16:40:34.276887 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:34.276789 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd213278-0e61-4779-a644-abfed99fbb9a-credential-socket\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-8cxdf\" (UID: \"dd213278-0e61-4779-a644-abfed99fbb9a\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-8cxdf" Oct 30 16:40:34.276887 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:34.276824 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd213278-0e61-4779-a644-abfed99fbb9a-istio-token\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-8cxdf\" (UID: \"dd213278-0e61-4779-a644-abfed99fbb9a\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-8cxdf" Oct 30 16:40:34.277157 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:34.276899 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd213278-0e61-4779-a644-abfed99fbb9a-workload-certs\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-8cxdf\" (UID: \"dd213278-0e61-4779-a644-abfed99fbb9a\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-8cxdf" Oct 30 16:40:34.277157 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:34.276971 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd213278-0e61-4779-a644-abfed99fbb9a-istio-envoy\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-8cxdf\" (UID: \"dd213278-0e61-4779-a644-abfed99fbb9a\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-8cxdf" Oct 30 16:40:34.277157 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:34.277029 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd213278-0e61-4779-a644-abfed99fbb9a-istio-podinfo\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-8cxdf\" (UID: \"dd213278-0e61-4779-a644-abfed99fbb9a\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-8cxdf" Oct 30 16:40:34.277157 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:34.277143 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd213278-0e61-4779-a644-abfed99fbb9a-workload-socket\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-8cxdf\" (UID: \"dd213278-0e61-4779-a644-abfed99fbb9a\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-8cxdf" Oct 30 16:40:34.277335 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:34.277201 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd213278-0e61-4779-a644-abfed99fbb9a-istio-data\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-8cxdf\" (UID: \"dd213278-0e61-4779-a644-abfed99fbb9a\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-8cxdf" Oct 30 16:40:34.277374 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:34.277331 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd213278-0e61-4779-a644-abfed99fbb9a-workload-certs\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-8cxdf\" (UID: \"dd213278-0e61-4779-a644-abfed99fbb9a\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-8cxdf" Oct 30 16:40:34.277545 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:34.277526 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd213278-0e61-4779-a644-abfed99fbb9a-credential-socket\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-8cxdf\" (UID: \"dd213278-0e61-4779-a644-abfed99fbb9a\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-8cxdf" Oct 30 16:40:34.277678 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:34.277574 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd213278-0e61-4779-a644-abfed99fbb9a-istiod-ca-cert\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-8cxdf\" (UID: \"dd213278-0e61-4779-a644-abfed99fbb9a\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-8cxdf" Oct 30 16:40:34.279217 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:34.279195 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd213278-0e61-4779-a644-abfed99fbb9a-istio-envoy\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-8cxdf\" (UID: \"dd213278-0e61-4779-a644-abfed99fbb9a\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-8cxdf" Oct 30 16:40:34.279478 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:34.279462 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd213278-0e61-4779-a644-abfed99fbb9a-istio-podinfo\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-8cxdf\" (UID: \"dd213278-0e61-4779-a644-abfed99fbb9a\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-8cxdf" Oct 30 16:40:34.299393 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:34.299368 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd213278-0e61-4779-a644-abfed99fbb9a-istio-token\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-8cxdf\" (UID: \"dd213278-0e61-4779-a644-abfed99fbb9a\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-8cxdf" Oct 30 16:40:34.299663 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:34.299642 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crjgx\" (UniqueName: \"kubernetes.io/projected/dd213278-0e61-4779-a644-abfed99fbb9a-kube-api-access-crjgx\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-8cxdf\" (UID: \"dd213278-0e61-4779-a644-abfed99fbb9a\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-8cxdf" Oct 30 16:40:34.388879 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:34.388808 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-8cxdf" Oct 30 16:40:34.711295 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:34.711210 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-8cxdf"] Oct 30 16:40:34.713740 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:40:34.713713 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd213278_0e61_4779_a644_abfed99fbb9a.slice/crio-ef6b6c3b26a22d318ee4a4e89a3682754e4bbda33aaedc11922a1635ba756638 WatchSource:0}: Error finding container ef6b6c3b26a22d318ee4a4e89a3682754e4bbda33aaedc11922a1635ba756638: Status 404 returned error can't find the container with id ef6b6c3b26a22d318ee4a4e89a3682754e4bbda33aaedc11922a1635ba756638 Oct 30 16:40:34.716458 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:34.716427 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:40:34.716556 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:34.716494 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:40:34.716556 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:34.716534 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:40:34.958606 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:34.958019 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-8cxdf" event={"ID":"dd213278-0e61-4779-a644-abfed99fbb9a","Type":"ContainerStarted","Data":"e9052ffa09f683f1ff1770365e18bfcd37e8a9ba23795838e0125f2898f05cd4"} Oct 30 16:40:34.958606 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:34.958088 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-8cxdf" event={"ID":"dd213278-0e61-4779-a644-abfed99fbb9a","Type":"ContainerStarted","Data":"ef6b6c3b26a22d318ee4a4e89a3682754e4bbda33aaedc11922a1635ba756638"} Oct 30 16:40:35.003990 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:35.003867 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-8cxdf" podStartSLOduration=1.003851111 podStartE2EDuration="1.003851111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:40:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:40:35.001415685 +0000 UTC m=+1190.528750933" watchObservedRunningTime="2026-10-30 16:40:35.003851111 +0000 UTC m=+1190.531186330" Oct 30 16:40:35.389070 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:35.388983 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-8cxdf" Oct 30 16:40:35.390194 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:35.390164 5052 patch_prober.go:29] interesting pod/e2e-worker-w0-3e2f96-openshift-default-84865466dc-8cxdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 16:40:35.390301 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:35.390216 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-8cxdf" podUID="dd213278-0e61-4779-a644-abfed99fbb9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 16:40:36.389926 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:36.389502 5052 patch_prober.go:29] interesting pod/e2e-worker-w0-3e2f96-openshift-default-84865466dc-8cxdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 16:40:36.389926 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:36.389561 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-8cxdf" podUID="dd213278-0e61-4779-a644-abfed99fbb9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 16:40:37.389803 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:37.389766 5052 patch_prober.go:29] interesting pod/e2e-worker-w0-3e2f96-openshift-default-84865466dc-8cxdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 16:40:37.390001 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:37.389817 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-8cxdf" podUID="dd213278-0e61-4779-a644-abfed99fbb9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 16:40:38.389277 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:38.389243 5052 patch_prober.go:29] interesting pod/e2e-worker-w0-3e2f96-openshift-default-84865466dc-8cxdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 16:40:38.389459 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:38.389294 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-8cxdf" podUID="dd213278-0e61-4779-a644-abfed99fbb9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 16:40:39.389370 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:39.389334 5052 patch_prober.go:29] interesting pod/e2e-worker-w0-3e2f96-openshift-default-84865466dc-8cxdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 16:40:39.389788 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:39.389412 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-8cxdf" podUID="dd213278-0e61-4779-a644-abfed99fbb9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 16:40:40.390615 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:40.390080 5052 patch_prober.go:29] interesting pod/e2e-worker-w0-3e2f96-openshift-default-84865466dc-8cxdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 16:40:40.390615 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:40.390167 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-8cxdf" podUID="dd213278-0e61-4779-a644-abfed99fbb9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 16:40:41.392606 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:41.391645 5052 patch_prober.go:29] interesting pod/e2e-worker-w0-3e2f96-openshift-default-84865466dc-8cxdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 16:40:41.392606 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:41.391715 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-8cxdf" podUID="dd213278-0e61-4779-a644-abfed99fbb9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 16:40:41.439691 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:41.439644 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx" podUID="bd662f3a-659f-477c-b8d5-94c64288c0c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:41.462999 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:41.462670 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/distinct-2-w4-263f00-kserve-8464897cf5-q8ck8" podUID="15ed43ef-5cdd-489a-a70e-063239822ff3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:41.537846 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:41.536669 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/distinct-w5-967fac-kserve-69cbcccd8b-b85bn" podUID="8f6b9354-378f-4451-9ff7-be7f7fec12f2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:41.958660 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:41.957669 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-750dcb/distinct-w2-750dcb-kserve-86bbdfcbb-bpn77" podUID="c45190af-d795-4e13-8f2f-670b2c597658" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:42.395662 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:42.395415 5052 patch_prober.go:29] interesting pod/e2e-worker-w0-3e2f96-openshift-default-84865466dc-8cxdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 16:40:42.395662 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:42.395475 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-8cxdf" podUID="dd213278-0e61-4779-a644-abfed99fbb9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 16:40:42.444604 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:42.443821 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx" podUID="bd662f3a-659f-477c-b8d5-94c64288c0c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:42.444604 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:42.444269 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-967fac/distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx" Oct 30 16:40:42.471605 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:42.467616 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/distinct-2-w4-263f00-kserve-8464897cf5-q8ck8" podUID="15ed43ef-5cdd-489a-a70e-063239822ff3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:42.471605 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:42.468030 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-263f00/distinct-2-w4-263f00-kserve-8464897cf5-q8ck8" Oct 30 16:40:42.544718 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:42.544672 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/distinct-w5-967fac-kserve-69cbcccd8b-b85bn" podUID="8f6b9354-378f-4451-9ff7-be7f7fec12f2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:42.551605 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:42.545364 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-967fac/distinct-w5-967fac-kserve-69cbcccd8b-b85bn" Oct 30 16:40:42.873173 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:42.872684 5052 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-967fac/distinct-w5-967fac-kserve-69cbcccd8b-b85bn" podUID="8f6b9354-378f-4451-9ff7-be7f7fec12f2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:42.959117 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:42.958690 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-750dcb/distinct-w2-750dcb-kserve-86bbdfcbb-bpn77" podUID="c45190af-d795-4e13-8f2f-670b2c597658" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:43.121612 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:43.114094 5052 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-750dcb/distinct-w2-750dcb-kserve-86bbdfcbb-bpn77" podUID="c45190af-d795-4e13-8f2f-670b2c597658" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:43.393791 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:43.393754 5052 patch_prober.go:29] interesting pod/e2e-worker-w0-3e2f96-openshift-default-84865466dc-8cxdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 16:40:43.394051 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:43.394020 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-8cxdf" podUID="dd213278-0e61-4779-a644-abfed99fbb9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 16:40:43.460492 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:43.448728 5052 patch_prober.go:29] interesting pod/e2e-worker-w2-750dcb-openshift-default-777f567b9f-dnwt8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:40:43.460492 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:43.448797 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-dnwt8" podUID="c8947ffd-7539-43d8-86d5-fc17dd8c7d6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:43.693607 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:40:43.691985 5052 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.068s" Oct 30 16:40:43.693607 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:43.692107 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-750dcb/distinct-w2-750dcb-kserve-86bbdfcbb-bpn77" Oct 30 16:40:43.693607 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:43.692332 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-967fac/distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx" Oct 30 16:40:43.776673 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:43.701947 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/distinct-w5-967fac-kserve-69cbcccd8b-b85bn" podUID="8f6b9354-378f-4451-9ff7-be7f7fec12f2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:43.887721 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:43.729340 5052 patch_prober.go:29] interesting pod/e2e-worker-w3-0b2348-openshift-default-85679bc47c-vbvvh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:40:43.944636 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:43.944517 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-vbvvh" podUID="0a5f0a2a-f2f2-4bea-98e8-16e6794028de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:43.944899 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:43.887805 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/distinct-2-w4-263f00-kserve-8464897cf5-q8ck8" podUID="15ed43ef-5cdd-489a-a70e-063239822ff3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:44.067976 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:44.067927 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-750dcb/distinct-w2-750dcb-kserve-86bbdfcbb-bpn77" podUID="c45190af-d795-4e13-8f2f-670b2c597658" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded" Oct 30 16:40:44.342664 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:44.311183 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-967fac/distinct-w5-967fac-kserve-69cbcccd8b-b85bn" Oct 30 16:40:44.392606 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:44.391772 5052 patch_prober.go:29] interesting pod/e2e-worker-w0-3e2f96-openshift-default-84865466dc-8cxdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 16:40:44.392606 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:44.391833 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-8cxdf" podUID="dd213278-0e61-4779-a644-abfed99fbb9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 16:40:44.403685 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:44.393536 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-750dcb/distinct-w2-750dcb-kserve-86bbdfcbb-bpn77" Oct 30 16:40:44.403685 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:44.395438 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-263f00/distinct-2-w4-263f00-kserve-8464897cf5-q8ck8" Oct 30 16:40:44.661765 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:44.661731 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8965-9edf49"] Oct 30 16:40:44.668136 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:44.668110 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8965-9edf49" Oct 30 16:40:44.684362 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:44.684330 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-xr4c7\"" Oct 30 16:40:44.696156 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:44.691894 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8965-9edf49"] Oct 30 16:40:44.762132 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:44.762090 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4wrf8_4f0f46f4-5cda-4d67-b6f3-47df06eb844c/console-operator/2.log" Oct 30 16:40:44.762632 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:44.762520 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4wrf8_4f0f46f4-5cda-4d67-b6f3-47df06eb844c/console-operator/2.log" Oct 30 16:40:44.781618 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:44.769877 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrjwx\" (UniqueName: \"kubernetes.io/projected/a02dfcb0-009c-4a26-8d1c-c8af56c2b05a-kube-api-access-vrjwx\") pod \"test-curl-8965-9edf49\" (UID: \"a02dfcb0-009c-4a26-8d1c-c8af56c2b05a\") " pod="opendatahub/test-curl-8965-9edf49" Oct 30 16:40:44.871693 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:44.871662 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrjwx\" (UniqueName: \"kubernetes.io/projected/a02dfcb0-009c-4a26-8d1c-c8af56c2b05a-kube-api-access-vrjwx\") pod \"test-curl-8965-9edf49\" (UID: \"a02dfcb0-009c-4a26-8d1c-c8af56c2b05a\") " pod="opendatahub/test-curl-8965-9edf49" Oct 30 16:40:44.948662 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:44.944981 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrjwx\" (UniqueName: \"kubernetes.io/projected/a02dfcb0-009c-4a26-8d1c-c8af56c2b05a-kube-api-access-vrjwx\") pod \"test-curl-8965-9edf49\" (UID: \"a02dfcb0-009c-4a26-8d1c-c8af56c2b05a\") " pod="opendatahub/test-curl-8965-9edf49" Oct 30 16:40:44.999961 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:44.999926 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8965-9edf49" Oct 30 16:40:45.394314 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:45.394163 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8965-9edf49"] Oct 30 16:40:45.399430 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:40:45.399244 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda02dfcb0_009c_4a26_8d1c_c8af56c2b05a.slice/crio-9e926d5e865ba789ac1a28d4ebb3196104ac4ee3ffcb13663ed14a2750a94059 WatchSource:0}: Error finding container 9e926d5e865ba789ac1a28d4ebb3196104ac4ee3ffcb13663ed14a2750a94059: Status 404 returned error can't find the container with id 9e926d5e865ba789ac1a28d4ebb3196104ac4ee3ffcb13663ed14a2750a94059 Oct 30 16:40:45.402303 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:45.402284 5052 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 16:40:45.468883 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:45.468850 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-8cxdf" Oct 30 16:40:45.469289 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:45.469266 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-8cxdf" Oct 30 16:40:45.470045 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:45.470022 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-8cxdf" Oct 30 16:40:46.412606 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:46.412024 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8965-9edf49" event={"ID":"a02dfcb0-009c-4a26-8d1c-c8af56c2b05a","Type":"ContainerStarted","Data":"9e926d5e865ba789ac1a28d4ebb3196104ac4ee3ffcb13663ed14a2750a94059"} Oct 30 16:40:48.427011 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:48.426973 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8965-9edf49" event={"ID":"a02dfcb0-009c-4a26-8d1c-c8af56c2b05a","Type":"ContainerStarted","Data":"fa99d8be1582d810a4cd78c8863dfc86f5b873e0563464781e09195d0d27cc88"} Oct 30 16:40:48.465237 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:48.465177 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8965-9edf49" podStartSLOduration=2.128693027 podStartE2EDuration="4.465162666s" totalImagesPullingTime="2.336469639s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:40:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:40:48.463497938 +0000 UTC m=+1203.990833158" watchObservedRunningTime="2026-10-30 16:40:48.465162666 +0000 UTC m=+1203.992497885" Oct 30 16:40:48.843844 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:40:48.843763 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 16:40:49.474539 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:49.474502 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8965-9edf49"] Oct 30 16:40:49.479917 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:49.479888 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8965-9edf49"] Oct 30 16:40:49.647135 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:49.647102 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8965-cbe3f3"] Oct 30 16:40:49.647762 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:49.647746 5052 cpu_manager.go:414] "containerMap: removing container" podUID="a02dfcb0-009c-4a26-8d1c-c8af56c2b05a" containerName="test-curl-8965-9edf49" Oct 30 16:40:49.647817 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:49.647763 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a02dfcb0-009c-4a26-8d1c-c8af56c2b05a" containerName="test-curl-8965-9edf49" Oct 30 16:40:49.647878 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:49.647870 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="a02dfcb0-009c-4a26-8d1c-c8af56c2b05a" containerName="test-curl-8965-9edf49" Oct 30 16:40:49.651468 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:49.651453 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8965-cbe3f3" Oct 30 16:40:49.668327 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:49.668303 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8965-cbe3f3"] Oct 30 16:40:49.791902 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:49.791828 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lg2kr\" (UniqueName: \"kubernetes.io/projected/8816f256-7e84-48cd-8f05-25bcff341ea9-kube-api-access-lg2kr\") pod \"test-curl-8965-cbe3f3\" (UID: \"8816f256-7e84-48cd-8f05-25bcff341ea9\") " pod="opendatahub/test-curl-8965-cbe3f3" Oct 30 16:40:49.893479 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:49.893450 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lg2kr\" (UniqueName: \"kubernetes.io/projected/8816f256-7e84-48cd-8f05-25bcff341ea9-kube-api-access-lg2kr\") pod \"test-curl-8965-cbe3f3\" (UID: \"8816f256-7e84-48cd-8f05-25bcff341ea9\") " pod="opendatahub/test-curl-8965-cbe3f3" Oct 30 16:40:49.916426 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:49.916394 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lg2kr\" (UniqueName: \"kubernetes.io/projected/8816f256-7e84-48cd-8f05-25bcff341ea9-kube-api-access-lg2kr\") pod \"test-curl-8965-cbe3f3\" (UID: \"8816f256-7e84-48cd-8f05-25bcff341ea9\") " pod="opendatahub/test-curl-8965-cbe3f3" Oct 30 16:40:49.968054 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:49.968024 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8965-cbe3f3" Oct 30 16:40:50.324257 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:50.324219 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8965-cbe3f3"] Oct 30 16:40:50.442003 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:50.441967 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8965-cbe3f3" event={"ID":"8816f256-7e84-48cd-8f05-25bcff341ea9","Type":"ContainerStarted","Data":"5d83aec419b95e3e943b4032098d9ee798827ed57fae83f582f1a9627c505869"} Oct 30 16:40:50.442148 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:50.442018 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8965-cbe3f3" event={"ID":"8816f256-7e84-48cd-8f05-25bcff341ea9","Type":"ContainerStarted","Data":"3b60f7b73e3ef95582a606ce3227b358b3b306d568c05b5a34574bbe6e5e6d55"} Oct 30 16:40:50.442148 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:50.442088 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8965-9edf49" podUID="a02dfcb0-009c-4a26-8d1c-c8af56c2b05a" containerName="test-curl-8965-9edf49" containerID="cri-o://fa99d8be1582d810a4cd78c8863dfc86f5b873e0563464781e09195d0d27cc88" gracePeriod=2 Oct 30 16:40:50.483198 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:50.483138 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8965-cbe3f3" podStartSLOduration=1.483124766 podStartE2EDuration="1.483124766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:40:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:40:50.481714715 +0000 UTC m=+1206.009049936" watchObservedRunningTime="2026-10-30 16:40:50.483124766 +0000 UTC m=+1206.010459984" Oct 30 16:40:50.494608 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:40:50.494552 5052 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8965-9edf49\" is forbidden: User \"system:node:ip-10-0-131-136.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-136.ec2.internal' and this object" podUID="a02dfcb0-009c-4a26-8d1c-c8af56c2b05a" pod="opendatahub/test-curl-8965-9edf49" Oct 30 16:40:50.507509 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:40:50.507479 5052 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8965-9edf49\" is forbidden: User \"system:node:ip-10-0-131-136.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-136.ec2.internal' and this object" podUID="a02dfcb0-009c-4a26-8d1c-c8af56c2b05a" pod="opendatahub/test-curl-8965-9edf49" Oct 30 16:40:51.436813 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:51.436780 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8965-cbe3f3"] Oct 30 16:40:51.444013 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:51.443975 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8965-cbe3f3"] Oct 30 16:40:52.459009 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:52.458957 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8965-cbe3f3" podUID="8816f256-7e84-48cd-8f05-25bcff341ea9" containerName="test-curl-8965-cbe3f3" containerID="cri-o://5d83aec419b95e3e943b4032098d9ee798827ed57fae83f582f1a9627c505869" gracePeriod=2 Oct 30 16:40:52.471858 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:40:52.471821 5052 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8965-cbe3f3\" is forbidden: User \"system:node:ip-10-0-131-136.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-136.ec2.internal' and this object" podUID="8816f256-7e84-48cd-8f05-25bcff341ea9" pod="opendatahub/test-curl-8965-cbe3f3" Oct 30 16:40:52.633486 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:52.633457 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8965-9edf49" Oct 30 16:40:52.742975 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:52.742905 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a02dfcb0-009c-4a26-8d1c-c8af56c2b05a-kube-api-access-vrjwx\" (UniqueName: \"kubernetes.io/projected/a02dfcb0-009c-4a26-8d1c-c8af56c2b05a-kube-api-access-vrjwx\") pod \"a02dfcb0-009c-4a26-8d1c-c8af56c2b05a\" (UID: \"a02dfcb0-009c-4a26-8d1c-c8af56c2b05a\") " Oct 30 16:40:52.745992 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:52.745969 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a02dfcb0-009c-4a26-8d1c-c8af56c2b05a-kube-api-access-vrjwx" pod "a02dfcb0-009c-4a26-8d1c-c8af56c2b05a" (UID: "a02dfcb0-009c-4a26-8d1c-c8af56c2b05a"). InnerVolumeSpecName "kube-api-access-vrjwx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:40:52.844538 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:52.844493 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrjwx\" (UniqueName: \"kubernetes.io/projected/a02dfcb0-009c-4a26-8d1c-c8af56c2b05a-kube-api-access-vrjwx\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:40:53.467747 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:53.467712 5052 generic.go:356] "Generic (PLEG): container finished" podID="a02dfcb0-009c-4a26-8d1c-c8af56c2b05a" containerID="fa99d8be1582d810a4cd78c8863dfc86f5b873e0563464781e09195d0d27cc88" exitCode=137 Oct 30 16:40:53.468178 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:53.467770 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8965-9edf49" Oct 30 16:40:53.468178 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:53.467797 5052 scope.go:122] "RemoveContainer" containerID="fa99d8be1582d810a4cd78c8863dfc86f5b873e0563464781e09195d0d27cc88" Oct 30 16:40:53.478528 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:53.478511 5052 scope.go:122] "RemoveContainer" containerID="fa99d8be1582d810a4cd78c8863dfc86f5b873e0563464781e09195d0d27cc88" Oct 30 16:40:53.478895 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:40:53.478870 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa99d8be1582d810a4cd78c8863dfc86f5b873e0563464781e09195d0d27cc88\": container with ID starting with fa99d8be1582d810a4cd78c8863dfc86f5b873e0563464781e09195d0d27cc88 not found: ID does not exist" containerID="fa99d8be1582d810a4cd78c8863dfc86f5b873e0563464781e09195d0d27cc88" Oct 30 16:40:53.479015 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:53.478907 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa99d8be1582d810a4cd78c8863dfc86f5b873e0563464781e09195d0d27cc88"} err="failed to get container status \"fa99d8be1582d810a4cd78c8863dfc86f5b873e0563464781e09195d0d27cc88\": rpc error: code = NotFound desc = could not find container \"fa99d8be1582d810a4cd78c8863dfc86f5b873e0563464781e09195d0d27cc88\": container with ID starting with fa99d8be1582d810a4cd78c8863dfc86f5b873e0563464781e09195d0d27cc88 not found: ID does not exist" Oct 30 16:40:53.491712 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:40:53.491686 5052 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8965-9edf49\" is forbidden: User \"system:node:ip-10-0-131-136.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-136.ec2.internal' and this object" podUID="a02dfcb0-009c-4a26-8d1c-c8af56c2b05a" pod="opendatahub/test-curl-8965-9edf49" Oct 30 16:40:54.533793 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:54.533759 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cdwcj"] Oct 30 16:40:54.612929 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:54.612892 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a02dfcb0-009c-4a26-8d1c-c8af56c2b05a" path="/var/lib/kubelet/pods/a02dfcb0-009c-4a26-8d1c-c8af56c2b05a/volumes" Oct 30 16:40:54.621566 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:40:54.621532 5052 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8965-9edf49\" is forbidden: User \"system:node:ip-10-0-131-136.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-136.ec2.internal' and this object" podUID="a02dfcb0-009c-4a26-8d1c-c8af56c2b05a" pod="opendatahub/test-curl-8965-9edf49" Oct 30 16:40:54.670420 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:54.670390 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8965-cbe3f3" Oct 30 16:40:54.683390 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:40:54.683353 5052 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8965-cbe3f3\" is forbidden: User \"system:node:ip-10-0-131-136.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-136.ec2.internal' and this object" podUID="8816f256-7e84-48cd-8f05-25bcff341ea9" pod="opendatahub/test-curl-8965-cbe3f3" Oct 30 16:40:54.785305 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:54.785215 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8816f256-7e84-48cd-8f05-25bcff341ea9-kube-api-access-lg2kr\" (UniqueName: \"kubernetes.io/projected/8816f256-7e84-48cd-8f05-25bcff341ea9-kube-api-access-lg2kr\") pod \"8816f256-7e84-48cd-8f05-25bcff341ea9\" (UID: \"8816f256-7e84-48cd-8f05-25bcff341ea9\") " Oct 30 16:40:54.787416 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:54.787383 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8816f256-7e84-48cd-8f05-25bcff341ea9-kube-api-access-lg2kr" pod "8816f256-7e84-48cd-8f05-25bcff341ea9" (UID: "8816f256-7e84-48cd-8f05-25bcff341ea9"). InnerVolumeSpecName "kube-api-access-lg2kr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:40:54.887453 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:54.887421 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lg2kr\" (UniqueName: \"kubernetes.io/projected/8816f256-7e84-48cd-8f05-25bcff341ea9-kube-api-access-lg2kr\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:40:55.516302 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:55.514230 5052 generic.go:356] "Generic (PLEG): container finished" podID="8816f256-7e84-48cd-8f05-25bcff341ea9" containerID="5d83aec419b95e3e943b4032098d9ee798827ed57fae83f582f1a9627c505869" exitCode=137 Oct 30 16:40:55.516302 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:55.514430 5052 scope.go:122] "RemoveContainer" containerID="5d83aec419b95e3e943b4032098d9ee798827ed57fae83f582f1a9627c505869" Oct 30 16:40:55.516302 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:55.514575 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8965-cbe3f3" Oct 30 16:40:55.537104 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:40:55.536860 5052 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8965-cbe3f3\" is forbidden: User \"system:node:ip-10-0-131-136.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-136.ec2.internal' and this object" podUID="8816f256-7e84-48cd-8f05-25bcff341ea9" pod="opendatahub/test-curl-8965-cbe3f3" Oct 30 16:40:55.544532 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:55.543888 5052 scope.go:122] "RemoveContainer" containerID="5d83aec419b95e3e943b4032098d9ee798827ed57fae83f582f1a9627c505869" Oct 30 16:40:55.544532 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:40:55.544243 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d83aec419b95e3e943b4032098d9ee798827ed57fae83f582f1a9627c505869\": container with ID starting with 5d83aec419b95e3e943b4032098d9ee798827ed57fae83f582f1a9627c505869 not found: ID does not exist" containerID="5d83aec419b95e3e943b4032098d9ee798827ed57fae83f582f1a9627c505869" Oct 30 16:40:55.544532 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:55.544276 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d83aec419b95e3e943b4032098d9ee798827ed57fae83f582f1a9627c505869"} err="failed to get container status \"5d83aec419b95e3e943b4032098d9ee798827ed57fae83f582f1a9627c505869\": rpc error: code = NotFound desc = could not find container \"5d83aec419b95e3e943b4032098d9ee798827ed57fae83f582f1a9627c505869\": container with ID starting with 5d83aec419b95e3e943b4032098d9ee798827ed57fae83f582f1a9627c505869 not found: ID does not exist" Oct 30 16:40:55.556175 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:40:55.553330 5052 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8965-cbe3f3\" is forbidden: User \"system:node:ip-10-0-131-136.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-136.ec2.internal' and this object" podUID="8816f256-7e84-48cd-8f05-25bcff341ea9" pod="opendatahub/test-curl-8965-cbe3f3" Oct 30 16:40:56.637605 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:56.634419 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8816f256-7e84-48cd-8f05-25bcff341ea9" path="/var/lib/kubelet/pods/8816f256-7e84-48cd-8f05-25bcff341ea9/volumes" Oct 30 16:40:56.889801 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:56.889688 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/distinct-w5-967fac-kserve-69cbcccd8b-b85bn" podUID="8f6b9354-378f-4451-9ff7-be7f7fec12f2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:57.123827 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:57.120794 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-750dcb/distinct-w2-750dcb-kserve-86bbdfcbb-bpn77" podUID="c45190af-d795-4e13-8f2f-670b2c597658" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:57.330605 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:57.323859 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx" podUID="bd662f3a-659f-477c-b8d5-94c64288c0c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:57.402606 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:57.402005 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/distinct-2-w4-263f00-kserve-8464897cf5-q8ck8" podUID="15ed43ef-5cdd-489a-a70e-063239822ff3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:57.895608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:57.894890 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/distinct-w5-967fac-kserve-69cbcccd8b-b85bn" podUID="8f6b9354-378f-4451-9ff7-be7f7fec12f2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:57.920305 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:57.920249 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-967fac/distinct-w5-967fac-kserve-69cbcccd8b-b85bn" Oct 30 16:40:58.005938 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:58.005900 5052 patch_prober.go:29] interesting pod/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-98gfq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:40:58.006221 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:58.006191 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-98gfq" podUID="c22edc73-463a-4439-92aa-601b4cddd714" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:58.152608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:58.151240 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-750dcb/distinct-w2-750dcb-kserve-86bbdfcbb-bpn77" podUID="c45190af-d795-4e13-8f2f-670b2c597658" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded" Oct 30 16:40:58.152608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:58.152406 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-750dcb/distinct-w2-750dcb-kserve-86bbdfcbb-bpn77" Oct 30 16:40:58.234910 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:58.234869 5052 patch_prober.go:29] interesting pod/e2e-worker-w0-3e2f96-openshift-default-84865466dc-7lh24 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:40:58.235187 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:58.235155 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-7lh24" podUID="0e461432-f429-4153-9e32-954f2ca3fb61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:58.423719 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:58.423620 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx" podUID="bd662f3a-659f-477c-b8d5-94c64288c0c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:58.429919 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:58.429893 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-967fac/distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx" Oct 30 16:40:58.534609 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:58.465382 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/distinct-2-w4-263f00-kserve-8464897cf5-q8ck8" podUID="15ed43ef-5cdd-489a-a70e-063239822ff3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:58.627611 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:58.552786 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-263f00/distinct-2-w4-263f00-kserve-8464897cf5-q8ck8" Oct 30 16:40:58.997761 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:58.960806 5052 patch_prober.go:29] interesting pod/e2e-worker-w3-0b2348-openshift-default-85679bc47c-xjhj4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:40:59.058743 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:59.058697 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-xjhj4" podUID="029d295c-a65a-4191-9754-a421c21464ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:59.080061 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:59.017099 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/distinct-w5-967fac-kserve-69cbcccd8b-b85bn" podUID="8f6b9354-378f-4451-9ff7-be7f7fec12f2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:59.248662 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:59.223833 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-750dcb/distinct-w2-750dcb-kserve-86bbdfcbb-bpn77" podUID="c45190af-d795-4e13-8f2f-670b2c597658" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:59.418606 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:59.407394 5052 patch_prober.go:29] interesting pod/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2slrc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:40:59.458723 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:59.458680 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2slrc" podUID="69a65b89-4d81-476b-b4b7-06736537fa4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:59.459291 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:59.453941 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx" podUID="bd662f3a-659f-477c-b8d5-94c64288c0c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:59.518678 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:59.518563 5052 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-263f00/distinct-2-w4-263f00-kserve-8464897cf5-q8ck8" podUID="15ed43ef-5cdd-489a-a70e-063239822ff3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": context deadline exceeded" Oct 30 16:40:59.652163 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:40:59.652033 5052 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.054s" Oct 30 16:40:59.657735 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:40:59.657691 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/distinct-2-w4-263f00-kserve-8464897cf5-q8ck8" podUID="15ed43ef-5cdd-489a-a70e-063239822ff3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 30 16:41:00.088572 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:00.088521 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/distinct-w5-967fac-kserve-69cbcccd8b-b85bn" podUID="8f6b9354-378f-4451-9ff7-be7f7fec12f2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:00.256063 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:00.256006 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-750dcb/distinct-w2-750dcb-kserve-86bbdfcbb-bpn77" podUID="c45190af-d795-4e13-8f2f-670b2c597658" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:00.444151 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:00.444118 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-967fac/distinct-w5-967fac-kserve-69cbcccd8b-b85bn" Oct 30 16:41:00.499458 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:00.499352 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx" podUID="bd662f3a-659f-477c-b8d5-94c64288c0c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:00.557264 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:00.557232 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-967fac/distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx" Oct 30 16:41:00.689110 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:00.679751 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-750dcb/distinct-w2-750dcb-kserve-86bbdfcbb-bpn77" Oct 30 16:41:00.754557 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:00.754470 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-263f00/distinct-2-w4-263f00-kserve-8464897cf5-q8ck8" Oct 30 16:41:02.288836 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:02.288802 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-f654570d-openshift-default-76cdfc5fcf-kwdnz"] Oct 30 16:41:02.289481 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:02.289463 5052 cpu_manager.go:414] "containerMap: removing container" podUID="8816f256-7e84-48cd-8f05-25bcff341ea9" containerName="test-curl-8965-cbe3f3" Oct 30 16:41:02.289481 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:02.289481 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8816f256-7e84-48cd-8f05-25bcff341ea9" containerName="test-curl-8965-cbe3f3" Oct 30 16:41:02.289693 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:02.289680 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="8816f256-7e84-48cd-8f05-25bcff341ea9" containerName="test-curl-8965-cbe3f3" Oct 30 16:41:02.293397 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:02.293378 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f654570d-openshift-default-76cdfc5fcf-kwdnz" Oct 30 16:41:02.312983 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:02.312959 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-f654570d-openshift-default-76cdfc5fcf-kwdnz"] Oct 30 16:41:02.331917 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:02.331894 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-f654570d-openshift-default-dockercfg-wbmbj\"" Oct 30 16:41:02.390482 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:02.390444 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6092ad4-fa04-4813-aebb-655353769fc1-istio-token\") pod \"e2e-ait-f654570d-openshift-default-76cdfc5fcf-kwdnz\" (UID: \"f6092ad4-fa04-4813-aebb-655353769fc1\") " pod="openshift-ingress/e2e-ait-f654570d-openshift-default-76cdfc5fcf-kwdnz" Oct 30 16:41:02.390682 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:02.390497 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6092ad4-fa04-4813-aebb-655353769fc1-istio-data\") pod \"e2e-ait-f654570d-openshift-default-76cdfc5fcf-kwdnz\" (UID: \"f6092ad4-fa04-4813-aebb-655353769fc1\") " pod="openshift-ingress/e2e-ait-f654570d-openshift-default-76cdfc5fcf-kwdnz" Oct 30 16:41:02.390682 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:02.390645 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6092ad4-fa04-4813-aebb-655353769fc1-istio-envoy\") pod \"e2e-ait-f654570d-openshift-default-76cdfc5fcf-kwdnz\" (UID: \"f6092ad4-fa04-4813-aebb-655353769fc1\") " pod="openshift-ingress/e2e-ait-f654570d-openshift-default-76cdfc5fcf-kwdnz" Oct 30 16:41:02.390813 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:02.390708 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6092ad4-fa04-4813-aebb-655353769fc1-istiod-ca-cert\") pod \"e2e-ait-f654570d-openshift-default-76cdfc5fcf-kwdnz\" (UID: \"f6092ad4-fa04-4813-aebb-655353769fc1\") " pod="openshift-ingress/e2e-ait-f654570d-openshift-default-76cdfc5fcf-kwdnz" Oct 30 16:41:02.390813 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:02.390785 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6092ad4-fa04-4813-aebb-655353769fc1-credential-socket\") pod \"e2e-ait-f654570d-openshift-default-76cdfc5fcf-kwdnz\" (UID: \"f6092ad4-fa04-4813-aebb-655353769fc1\") " pod="openshift-ingress/e2e-ait-f654570d-openshift-default-76cdfc5fcf-kwdnz" Oct 30 16:41:02.390927 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:02.390819 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6092ad4-fa04-4813-aebb-655353769fc1-istio-podinfo\") pod \"e2e-ait-f654570d-openshift-default-76cdfc5fcf-kwdnz\" (UID: \"f6092ad4-fa04-4813-aebb-655353769fc1\") " pod="openshift-ingress/e2e-ait-f654570d-openshift-default-76cdfc5fcf-kwdnz" Oct 30 16:41:02.390927 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:02.390907 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6092ad4-fa04-4813-aebb-655353769fc1-workload-certs\") pod \"e2e-ait-f654570d-openshift-default-76cdfc5fcf-kwdnz\" (UID: \"f6092ad4-fa04-4813-aebb-655353769fc1\") " pod="openshift-ingress/e2e-ait-f654570d-openshift-default-76cdfc5fcf-kwdnz" Oct 30 16:41:02.390998 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:02.390965 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6092ad4-fa04-4813-aebb-655353769fc1-workload-socket\") pod \"e2e-ait-f654570d-openshift-default-76cdfc5fcf-kwdnz\" (UID: \"f6092ad4-fa04-4813-aebb-655353769fc1\") " pod="openshift-ingress/e2e-ait-f654570d-openshift-default-76cdfc5fcf-kwdnz" Oct 30 16:41:02.391054 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:02.390997 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msjqp\" (UniqueName: \"kubernetes.io/projected/f6092ad4-fa04-4813-aebb-655353769fc1-kube-api-access-msjqp\") pod \"e2e-ait-f654570d-openshift-default-76cdfc5fcf-kwdnz\" (UID: \"f6092ad4-fa04-4813-aebb-655353769fc1\") " pod="openshift-ingress/e2e-ait-f654570d-openshift-default-76cdfc5fcf-kwdnz" Oct 30 16:41:02.493604 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:02.493473 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6092ad4-fa04-4813-aebb-655353769fc1-istio-token\") pod \"e2e-ait-f654570d-openshift-default-76cdfc5fcf-kwdnz\" (UID: \"f6092ad4-fa04-4813-aebb-655353769fc1\") " pod="openshift-ingress/e2e-ait-f654570d-openshift-default-76cdfc5fcf-kwdnz" Oct 30 16:41:02.493604 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:02.493545 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6092ad4-fa04-4813-aebb-655353769fc1-istio-data\") pod \"e2e-ait-f654570d-openshift-default-76cdfc5fcf-kwdnz\" (UID: \"f6092ad4-fa04-4813-aebb-655353769fc1\") " pod="openshift-ingress/e2e-ait-f654570d-openshift-default-76cdfc5fcf-kwdnz" Oct 30 16:41:02.493917 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:02.493647 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6092ad4-fa04-4813-aebb-655353769fc1-istio-envoy\") pod \"e2e-ait-f654570d-openshift-default-76cdfc5fcf-kwdnz\" (UID: \"f6092ad4-fa04-4813-aebb-655353769fc1\") " pod="openshift-ingress/e2e-ait-f654570d-openshift-default-76cdfc5fcf-kwdnz" Oct 30 16:41:02.493917 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:02.493738 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6092ad4-fa04-4813-aebb-655353769fc1-istiod-ca-cert\") pod \"e2e-ait-f654570d-openshift-default-76cdfc5fcf-kwdnz\" (UID: \"f6092ad4-fa04-4813-aebb-655353769fc1\") " pod="openshift-ingress/e2e-ait-f654570d-openshift-default-76cdfc5fcf-kwdnz" Oct 30 16:41:02.493917 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:02.493773 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6092ad4-fa04-4813-aebb-655353769fc1-credential-socket\") pod \"e2e-ait-f654570d-openshift-default-76cdfc5fcf-kwdnz\" (UID: \"f6092ad4-fa04-4813-aebb-655353769fc1\") " pod="openshift-ingress/e2e-ait-f654570d-openshift-default-76cdfc5fcf-kwdnz" Oct 30 16:41:02.493917 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:02.493799 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6092ad4-fa04-4813-aebb-655353769fc1-istio-podinfo\") pod \"e2e-ait-f654570d-openshift-default-76cdfc5fcf-kwdnz\" (UID: \"f6092ad4-fa04-4813-aebb-655353769fc1\") " pod="openshift-ingress/e2e-ait-f654570d-openshift-default-76cdfc5fcf-kwdnz" Oct 30 16:41:02.493917 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:02.493865 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6092ad4-fa04-4813-aebb-655353769fc1-workload-certs\") pod \"e2e-ait-f654570d-openshift-default-76cdfc5fcf-kwdnz\" (UID: \"f6092ad4-fa04-4813-aebb-655353769fc1\") " pod="openshift-ingress/e2e-ait-f654570d-openshift-default-76cdfc5fcf-kwdnz" Oct 30 16:41:02.494357 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:02.493938 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6092ad4-fa04-4813-aebb-655353769fc1-workload-socket\") pod \"e2e-ait-f654570d-openshift-default-76cdfc5fcf-kwdnz\" (UID: \"f6092ad4-fa04-4813-aebb-655353769fc1\") " pod="openshift-ingress/e2e-ait-f654570d-openshift-default-76cdfc5fcf-kwdnz" Oct 30 16:41:02.494357 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:02.493997 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msjqp\" (UniqueName: \"kubernetes.io/projected/f6092ad4-fa04-4813-aebb-655353769fc1-kube-api-access-msjqp\") pod \"e2e-ait-f654570d-openshift-default-76cdfc5fcf-kwdnz\" (UID: \"f6092ad4-fa04-4813-aebb-655353769fc1\") " pod="openshift-ingress/e2e-ait-f654570d-openshift-default-76cdfc5fcf-kwdnz" Oct 30 16:41:02.498602 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:02.495099 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6092ad4-fa04-4813-aebb-655353769fc1-istio-data\") pod \"e2e-ait-f654570d-openshift-default-76cdfc5fcf-kwdnz\" (UID: \"f6092ad4-fa04-4813-aebb-655353769fc1\") " pod="openshift-ingress/e2e-ait-f654570d-openshift-default-76cdfc5fcf-kwdnz" Oct 30 16:41:02.502608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:02.502438 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6092ad4-fa04-4813-aebb-655353769fc1-workload-certs\") pod \"e2e-ait-f654570d-openshift-default-76cdfc5fcf-kwdnz\" (UID: \"f6092ad4-fa04-4813-aebb-655353769fc1\") " pod="openshift-ingress/e2e-ait-f654570d-openshift-default-76cdfc5fcf-kwdnz" Oct 30 16:41:02.506602 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:02.502790 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6092ad4-fa04-4813-aebb-655353769fc1-workload-socket\") pod \"e2e-ait-f654570d-openshift-default-76cdfc5fcf-kwdnz\" (UID: \"f6092ad4-fa04-4813-aebb-655353769fc1\") " pod="openshift-ingress/e2e-ait-f654570d-openshift-default-76cdfc5fcf-kwdnz" Oct 30 16:41:02.506602 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:02.503388 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6092ad4-fa04-4813-aebb-655353769fc1-istiod-ca-cert\") pod \"e2e-ait-f654570d-openshift-default-76cdfc5fcf-kwdnz\" (UID: \"f6092ad4-fa04-4813-aebb-655353769fc1\") " pod="openshift-ingress/e2e-ait-f654570d-openshift-default-76cdfc5fcf-kwdnz" Oct 30 16:41:02.506602 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:02.503643 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6092ad4-fa04-4813-aebb-655353769fc1-credential-socket\") pod \"e2e-ait-f654570d-openshift-default-76cdfc5fcf-kwdnz\" (UID: \"f6092ad4-fa04-4813-aebb-655353769fc1\") " pod="openshift-ingress/e2e-ait-f654570d-openshift-default-76cdfc5fcf-kwdnz" Oct 30 16:41:02.540396 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:02.540317 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msjqp\" (UniqueName: \"kubernetes.io/projected/f6092ad4-fa04-4813-aebb-655353769fc1-kube-api-access-msjqp\") pod \"e2e-ait-f654570d-openshift-default-76cdfc5fcf-kwdnz\" (UID: \"f6092ad4-fa04-4813-aebb-655353769fc1\") " pod="openshift-ingress/e2e-ait-f654570d-openshift-default-76cdfc5fcf-kwdnz" Oct 30 16:41:02.546904 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:02.545916 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6092ad4-fa04-4813-aebb-655353769fc1-istio-envoy\") pod \"e2e-ait-f654570d-openshift-default-76cdfc5fcf-kwdnz\" (UID: \"f6092ad4-fa04-4813-aebb-655353769fc1\") " pod="openshift-ingress/e2e-ait-f654570d-openshift-default-76cdfc5fcf-kwdnz" Oct 30 16:41:02.546904 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:02.546381 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6092ad4-fa04-4813-aebb-655353769fc1-istio-token\") pod \"e2e-ait-f654570d-openshift-default-76cdfc5fcf-kwdnz\" (UID: \"f6092ad4-fa04-4813-aebb-655353769fc1\") " pod="openshift-ingress/e2e-ait-f654570d-openshift-default-76cdfc5fcf-kwdnz" Oct 30 16:41:02.546904 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:02.546408 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6092ad4-fa04-4813-aebb-655353769fc1-istio-podinfo\") pod \"e2e-ait-f654570d-openshift-default-76cdfc5fcf-kwdnz\" (UID: \"f6092ad4-fa04-4813-aebb-655353769fc1\") " pod="openshift-ingress/e2e-ait-f654570d-openshift-default-76cdfc5fcf-kwdnz" Oct 30 16:41:02.638191 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:02.621227 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f654570d-openshift-default-76cdfc5fcf-kwdnz" Oct 30 16:41:03.043255 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:03.043228 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-f654570d-openshift-default-76cdfc5fcf-kwdnz"] Oct 30 16:41:03.045700 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:41:03.045671 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6092ad4_fa04_4813_aebb_655353769fc1.slice/crio-85f92d6dd4362f1034f060cde1940b8c9fa66e240c407e2f09570d4beec251ff WatchSource:0}: Error finding container 85f92d6dd4362f1034f060cde1940b8c9fa66e240c407e2f09570d4beec251ff: Status 404 returned error can't find the container with id 85f92d6dd4362f1034f060cde1940b8c9fa66e240c407e2f09570d4beec251ff Oct 30 16:41:03.048096 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:03.048065 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:41:03.048159 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:03.048131 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:41:03.048193 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:03.048160 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:41:03.232748 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:03.232714 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f654570d-openshift-default-76cdfc5fcf-kwdnz" event={"ID":"f6092ad4-fa04-4813-aebb-655353769fc1","Type":"ContainerStarted","Data":"6939f75159ba401a04fb22fb3ae142dfd65077ddfe28f96281a7b01d0de70471"} Oct 30 16:41:03.232911 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:03.232765 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f654570d-openshift-default-76cdfc5fcf-kwdnz" event={"ID":"f6092ad4-fa04-4813-aebb-655353769fc1","Type":"ContainerStarted","Data":"85f92d6dd4362f1034f060cde1940b8c9fa66e240c407e2f09570d4beec251ff"} Oct 30 16:41:03.281917 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:03.281863 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-f654570d-openshift-default-76cdfc5fcf-kwdnz" podStartSLOduration=1.281850796 podStartE2EDuration="1.281850796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:41:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:41:03.277214521 +0000 UTC m=+1218.804549740" watchObservedRunningTime="2026-10-30 16:41:03.281850796 +0000 UTC m=+1218.809186018" Oct 30 16:41:03.625543 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:03.625506 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-f654570d-openshift-default-76cdfc5fcf-kwdnz" Oct 30 16:41:04.054595 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:04.054549 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-bg8br"] Oct 30 16:41:04.059183 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:04.059156 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-bg8br" Oct 30 16:41:04.079623 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:04.079573 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-bg8br"] Oct 30 16:41:04.146405 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:04.146371 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/597c97ef-00c9-46b1-afbb-8b05363df559-workload-certs\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-bg8br\" (UID: \"597c97ef-00c9-46b1-afbb-8b05363df559\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-bg8br" Oct 30 16:41:04.146594 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:04.146432 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/597c97ef-00c9-46b1-afbb-8b05363df559-istio-envoy\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-bg8br\" (UID: \"597c97ef-00c9-46b1-afbb-8b05363df559\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-bg8br" Oct 30 16:41:04.146594 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:04.146535 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/597c97ef-00c9-46b1-afbb-8b05363df559-istio-podinfo\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-bg8br\" (UID: \"597c97ef-00c9-46b1-afbb-8b05363df559\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-bg8br" Oct 30 16:41:04.146688 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:04.146617 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqhz2\" (UniqueName: \"kubernetes.io/projected/597c97ef-00c9-46b1-afbb-8b05363df559-kube-api-access-zqhz2\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-bg8br\" (UID: \"597c97ef-00c9-46b1-afbb-8b05363df559\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-bg8br" Oct 30 16:41:04.146688 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:04.146643 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/597c97ef-00c9-46b1-afbb-8b05363df559-istio-token\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-bg8br\" (UID: \"597c97ef-00c9-46b1-afbb-8b05363df559\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-bg8br" Oct 30 16:41:04.146688 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:04.146674 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/597c97ef-00c9-46b1-afbb-8b05363df559-workload-socket\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-bg8br\" (UID: \"597c97ef-00c9-46b1-afbb-8b05363df559\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-bg8br" Oct 30 16:41:04.146855 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:04.146830 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/597c97ef-00c9-46b1-afbb-8b05363df559-istio-data\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-bg8br\" (UID: \"597c97ef-00c9-46b1-afbb-8b05363df559\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-bg8br" Oct 30 16:41:04.147003 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:04.146977 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/597c97ef-00c9-46b1-afbb-8b05363df559-credential-socket\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-bg8br\" (UID: \"597c97ef-00c9-46b1-afbb-8b05363df559\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-bg8br" Oct 30 16:41:04.147039 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:04.147020 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/597c97ef-00c9-46b1-afbb-8b05363df559-istiod-ca-cert\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-bg8br\" (UID: \"597c97ef-00c9-46b1-afbb-8b05363df559\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-bg8br" Oct 30 16:41:04.285621 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:04.256698 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/597c97ef-00c9-46b1-afbb-8b05363df559-credential-socket\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-bg8br\" (UID: \"597c97ef-00c9-46b1-afbb-8b05363df559\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-bg8br" Oct 30 16:41:04.285621 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:04.256759 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/597c97ef-00c9-46b1-afbb-8b05363df559-istiod-ca-cert\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-bg8br\" (UID: \"597c97ef-00c9-46b1-afbb-8b05363df559\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-bg8br" Oct 30 16:41:04.285621 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:04.256845 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/597c97ef-00c9-46b1-afbb-8b05363df559-workload-certs\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-bg8br\" (UID: \"597c97ef-00c9-46b1-afbb-8b05363df559\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-bg8br" Oct 30 16:41:04.285621 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:04.256925 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/597c97ef-00c9-46b1-afbb-8b05363df559-istio-envoy\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-bg8br\" (UID: \"597c97ef-00c9-46b1-afbb-8b05363df559\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-bg8br" Oct 30 16:41:04.285621 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:04.256974 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/597c97ef-00c9-46b1-afbb-8b05363df559-istio-podinfo\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-bg8br\" (UID: \"597c97ef-00c9-46b1-afbb-8b05363df559\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-bg8br" Oct 30 16:41:04.285621 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:04.257083 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqhz2\" (UniqueName: \"kubernetes.io/projected/597c97ef-00c9-46b1-afbb-8b05363df559-kube-api-access-zqhz2\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-bg8br\" (UID: \"597c97ef-00c9-46b1-afbb-8b05363df559\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-bg8br" Oct 30 16:41:04.285621 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:04.257123 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/597c97ef-00c9-46b1-afbb-8b05363df559-istio-token\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-bg8br\" (UID: \"597c97ef-00c9-46b1-afbb-8b05363df559\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-bg8br" Oct 30 16:41:04.285621 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:04.257172 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/597c97ef-00c9-46b1-afbb-8b05363df559-workload-socket\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-bg8br\" (UID: \"597c97ef-00c9-46b1-afbb-8b05363df559\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-bg8br" Oct 30 16:41:04.285621 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:04.257229 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/597c97ef-00c9-46b1-afbb-8b05363df559-istio-data\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-bg8br\" (UID: \"597c97ef-00c9-46b1-afbb-8b05363df559\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-bg8br" Oct 30 16:41:04.286230 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:04.286044 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/597c97ef-00c9-46b1-afbb-8b05363df559-credential-socket\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-bg8br\" (UID: \"597c97ef-00c9-46b1-afbb-8b05363df559\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-bg8br" Oct 30 16:41:04.286797 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:04.286770 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/597c97ef-00c9-46b1-afbb-8b05363df559-istiod-ca-cert\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-bg8br\" (UID: \"597c97ef-00c9-46b1-afbb-8b05363df559\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-bg8br" Oct 30 16:41:04.287243 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:04.287221 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/597c97ef-00c9-46b1-afbb-8b05363df559-workload-certs\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-bg8br\" (UID: \"597c97ef-00c9-46b1-afbb-8b05363df559\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-bg8br" Oct 30 16:41:04.309276 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:04.309202 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/597c97ef-00c9-46b1-afbb-8b05363df559-workload-socket\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-bg8br\" (UID: \"597c97ef-00c9-46b1-afbb-8b05363df559\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-bg8br" Oct 30 16:41:04.329691 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:04.329133 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/597c97ef-00c9-46b1-afbb-8b05363df559-istio-data\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-bg8br\" (UID: \"597c97ef-00c9-46b1-afbb-8b05363df559\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-bg8br" Oct 30 16:41:04.357715 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:04.357685 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqhz2\" (UniqueName: \"kubernetes.io/projected/597c97ef-00c9-46b1-afbb-8b05363df559-kube-api-access-zqhz2\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-bg8br\" (UID: \"597c97ef-00c9-46b1-afbb-8b05363df559\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-bg8br" Oct 30 16:41:04.358921 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:04.358478 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/597c97ef-00c9-46b1-afbb-8b05363df559-istio-token\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-bg8br\" (UID: \"597c97ef-00c9-46b1-afbb-8b05363df559\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-bg8br" Oct 30 16:41:04.411981 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:04.410423 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/597c97ef-00c9-46b1-afbb-8b05363df559-istio-envoy\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-bg8br\" (UID: \"597c97ef-00c9-46b1-afbb-8b05363df559\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-bg8br" Oct 30 16:41:04.414668 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:04.414633 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/597c97ef-00c9-46b1-afbb-8b05363df559-istio-podinfo\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-bg8br\" (UID: \"597c97ef-00c9-46b1-afbb-8b05363df559\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-bg8br" Oct 30 16:41:04.665606 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:04.664173 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-f654570d-openshift-default-76cdfc5fcf-kwdnz" Oct 30 16:41:04.678870 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:04.677683 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-bg8br" Oct 30 16:41:05.026653 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:05.026623 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-bg8br"] Oct 30 16:41:05.027819 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:41:05.027789 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod597c97ef_00c9_46b1_afbb_8b05363df559.slice/crio-2d9d77889e9fcbeb3b6fc6479700c5142ed040f2d59de00cea38c3dfb968b0cc WatchSource:0}: Error finding container 2d9d77889e9fcbeb3b6fc6479700c5142ed040f2d59de00cea38c3dfb968b0cc: Status 404 returned error can't find the container with id 2d9d77889e9fcbeb3b6fc6479700c5142ed040f2d59de00cea38c3dfb968b0cc Oct 30 16:41:05.030549 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:05.030518 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:41:05.030663 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:05.030597 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:41:05.030663 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:05.030631 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:41:05.357992 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:05.357959 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-bg8br" event={"ID":"597c97ef-00c9-46b1-afbb-8b05363df559","Type":"ContainerStarted","Data":"2d9d77889e9fcbeb3b6fc6479700c5142ed040f2d59de00cea38c3dfb968b0cc"} Oct 30 16:41:05.358151 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:05.358034 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-f654570d-openshift-default-76cdfc5fcf-kwdnz" Oct 30 16:41:05.359234 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:05.359206 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-f654570d-openshift-default-76cdfc5fcf-kwdnz" Oct 30 16:41:05.418552 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:05.418502 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-bg8br" podStartSLOduration=1.418488978 podStartE2EDuration="1.418488978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:41:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:41:05.414489332 +0000 UTC m=+1220.941824579" watchObservedRunningTime="2026-10-30 16:41:05.418488978 +0000 UTC m=+1220.945824196" Oct 30 16:41:05.681694 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:05.681654 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-bg8br" Oct 30 16:41:06.365739 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:06.365698 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-bg8br" event={"ID":"597c97ef-00c9-46b1-afbb-8b05363df559","Type":"ContainerStarted","Data":"3dcd2c17c569e4716e96bf386a1abf72d5329514b6ae4a6a3fde68a0fbce0037"} Oct 30 16:41:06.682564 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:06.682529 5052 patch_prober.go:29] interesting pod/e2e-worker-w2-750dcb-openshift-default-777f567b9f-bg8br container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 30 16:41:06.682985 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:06.682596 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-bg8br" podUID="597c97ef-00c9-46b1-afbb-8b05363df559" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 30 16:41:06.779684 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:06.779651 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f654570d-7b49ffb9f-2s4q4"] Oct 30 16:41:06.784255 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:06.784236 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f654570d-7b49ffb9f-2s4q4" Oct 30 16:41:06.802666 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:06.802642 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-f654570d\"" Oct 30 16:41:06.805233 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:06.805207 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f654570d-7b49ffb9f-2s4q4"] Oct 30 16:41:06.812475 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:06.812451 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-f654570d-dockercfg-zpdq9\"" Oct 30 16:41:06.996764 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:06.989253 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7c5678b-63b4-4fc1-89ca-f43a6ca9926a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f654570d-7b49ffb9f-2s4q4\" (UID: \"d7c5678b-63b4-4fc1-89ca-f43a6ca9926a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f654570d-7b49ffb9f-2s4q4" Oct 30 16:41:06.996764 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:06.989348 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwc24\" (UniqueName: \"kubernetes.io/projected/d7c5678b-63b4-4fc1-89ca-f43a6ca9926a-kube-api-access-nwc24\") pod \"payload-pre-processing-e2e-ait-f654570d-7b49ffb9f-2s4q4\" (UID: \"d7c5678b-63b4-4fc1-89ca-f43a6ca9926a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f654570d-7b49ffb9f-2s4q4" Oct 30 16:41:07.131480 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:07.130092 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7c5678b-63b4-4fc1-89ca-f43a6ca9926a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f654570d-7b49ffb9f-2s4q4\" (UID: \"d7c5678b-63b4-4fc1-89ca-f43a6ca9926a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f654570d-7b49ffb9f-2s4q4" Oct 30 16:41:07.131480 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:07.130181 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwc24\" (UniqueName: \"kubernetes.io/projected/d7c5678b-63b4-4fc1-89ca-f43a6ca9926a-kube-api-access-nwc24\") pod \"payload-pre-processing-e2e-ait-f654570d-7b49ffb9f-2s4q4\" (UID: \"d7c5678b-63b4-4fc1-89ca-f43a6ca9926a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f654570d-7b49ffb9f-2s4q4" Oct 30 16:41:07.131480 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:07.131474 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7c5678b-63b4-4fc1-89ca-f43a6ca9926a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f654570d-7b49ffb9f-2s4q4\" (UID: \"d7c5678b-63b4-4fc1-89ca-f43a6ca9926a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f654570d-7b49ffb9f-2s4q4" Oct 30 16:41:07.156208 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:07.156172 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f654570d-56d5957948-lh66x"] Oct 30 16:41:07.189302 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:07.189261 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwc24\" (UniqueName: \"kubernetes.io/projected/d7c5678b-63b4-4fc1-89ca-f43a6ca9926a-kube-api-access-nwc24\") pod \"payload-pre-processing-e2e-ait-f654570d-7b49ffb9f-2s4q4\" (UID: \"d7c5678b-63b4-4fc1-89ca-f43a6ca9926a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f654570d-7b49ffb9f-2s4q4" Oct 30 16:41:07.195811 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:07.195776 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f654570d-56d5957948-lh66x"] Oct 30 16:41:07.196111 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:07.196088 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f654570d-56d5957948-lh66x" Oct 30 16:41:07.250662 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:07.235127 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf3426ff-3476-4dbe-8fbf-8196cabedfa4-plugins-config-volume\") pod \"payload-processing-e2e-ait-f654570d-56d5957948-lh66x\" (UID: \"cf3426ff-3476-4dbe-8fbf-8196cabedfa4\") " pod="openshift-ingress/payload-processing-e2e-ait-f654570d-56d5957948-lh66x" Oct 30 16:41:07.250662 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:07.235302 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvkn7\" (UniqueName: \"kubernetes.io/projected/cf3426ff-3476-4dbe-8fbf-8196cabedfa4-kube-api-access-cvkn7\") pod \"payload-processing-e2e-ait-f654570d-56d5957948-lh66x\" (UID: \"cf3426ff-3476-4dbe-8fbf-8196cabedfa4\") " pod="openshift-ingress/payload-processing-e2e-ait-f654570d-56d5957948-lh66x" Oct 30 16:41:07.351606 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:07.338071 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvkn7\" (UniqueName: \"kubernetes.io/projected/cf3426ff-3476-4dbe-8fbf-8196cabedfa4-kube-api-access-cvkn7\") pod \"payload-processing-e2e-ait-f654570d-56d5957948-lh66x\" (UID: \"cf3426ff-3476-4dbe-8fbf-8196cabedfa4\") " pod="openshift-ingress/payload-processing-e2e-ait-f654570d-56d5957948-lh66x" Oct 30 16:41:07.351606 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:07.338317 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf3426ff-3476-4dbe-8fbf-8196cabedfa4-plugins-config-volume\") pod \"payload-processing-e2e-ait-f654570d-56d5957948-lh66x\" (UID: \"cf3426ff-3476-4dbe-8fbf-8196cabedfa4\") " pod="openshift-ingress/payload-processing-e2e-ait-f654570d-56d5957948-lh66x" Oct 30 16:41:07.351606 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:07.339273 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf3426ff-3476-4dbe-8fbf-8196cabedfa4-plugins-config-volume\") pod \"payload-processing-e2e-ait-f654570d-56d5957948-lh66x\" (UID: \"cf3426ff-3476-4dbe-8fbf-8196cabedfa4\") " pod="openshift-ingress/payload-processing-e2e-ait-f654570d-56d5957948-lh66x" Oct 30 16:41:07.362530 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:07.362500 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvkn7\" (UniqueName: \"kubernetes.io/projected/cf3426ff-3476-4dbe-8fbf-8196cabedfa4-kube-api-access-cvkn7\") pod \"payload-processing-e2e-ait-f654570d-56d5957948-lh66x\" (UID: \"cf3426ff-3476-4dbe-8fbf-8196cabedfa4\") " pod="openshift-ingress/payload-processing-e2e-ait-f654570d-56d5957948-lh66x" Oct 30 16:41:07.398660 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:07.398636 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f654570d-7b49ffb9f-2s4q4" Oct 30 16:41:07.586471 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:07.586386 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f654570d-56d5957948-lh66x" Oct 30 16:41:07.696814 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:07.695377 5052 patch_prober.go:29] interesting pod/e2e-worker-w2-750dcb-openshift-default-777f567b9f-bg8br container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 30 16:41:07.696814 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:07.695438 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-bg8br" podUID="597c97ef-00c9-46b1-afbb-8b05363df559" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 30 16:41:07.772542 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:07.770801 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f654570d-7b49ffb9f-2s4q4"] Oct 30 16:41:07.775998 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:41:07.774974 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7c5678b_63b4_4fc1_89ca_f43a6ca9926a.slice/crio-4a4ed501b6ac50154aecf1e1e2292064239ac041ca81c4f3505a80812e2c1676 WatchSource:0}: Error finding container 4a4ed501b6ac50154aecf1e1e2292064239ac041ca81c4f3505a80812e2c1676: Status 404 returned error can't find the container with id 4a4ed501b6ac50154aecf1e1e2292064239ac041ca81c4f3505a80812e2c1676 Oct 30 16:41:07.964222 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:07.964183 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f654570d-56d5957948-lh66x"] Oct 30 16:41:07.964568 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:41:07.964534 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf3426ff_3476_4dbe_8fbf_8196cabedfa4.slice/crio-bb3c34a7011c41e3d509cfb9bb8b96b83e09e7910b24a54aa100b18a45a9e54e WatchSource:0}: Error finding container bb3c34a7011c41e3d509cfb9bb8b96b83e09e7910b24a54aa100b18a45a9e54e: Status 404 returned error can't find the container with id bb3c34a7011c41e3d509cfb9bb8b96b83e09e7910b24a54aa100b18a45a9e54e Oct 30 16:41:08.382750 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:08.382706 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f654570d-56d5957948-lh66x" event={"ID":"cf3426ff-3476-4dbe-8fbf-8196cabedfa4","Type":"ContainerStarted","Data":"e84bdf7f621343f796307bed370d4da876f8582157df6f2132fe164ade1eff4a"} Oct 30 16:41:08.382953 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:08.382782 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-f654570d-56d5957948-lh66x" Oct 30 16:41:08.382953 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:08.382804 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f654570d-56d5957948-lh66x" event={"ID":"cf3426ff-3476-4dbe-8fbf-8196cabedfa4","Type":"ContainerStarted","Data":"bb3c34a7011c41e3d509cfb9bb8b96b83e09e7910b24a54aa100b18a45a9e54e"} Oct 30 16:41:08.384371 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:08.384340 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f654570d-7b49ffb9f-2s4q4" event={"ID":"d7c5678b-63b4-4fc1-89ca-f43a6ca9926a","Type":"ContainerStarted","Data":"ee15b979f6adbc9b8c73abfd0de8a167491e5998f763042c82058b632b83e968"} Oct 30 16:41:08.384495 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:08.384389 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f654570d-7b49ffb9f-2s4q4" event={"ID":"d7c5678b-63b4-4fc1-89ca-f43a6ca9926a","Type":"ContainerStarted","Data":"4a4ed501b6ac50154aecf1e1e2292064239ac041ca81c4f3505a80812e2c1676"} Oct 30 16:41:08.384495 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:08.384435 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-f654570d-7b49ffb9f-2s4q4" Oct 30 16:41:08.422729 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:08.422683 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-f654570d-56d5957948-lh66x" podStartSLOduration=2.42267086 podStartE2EDuration="2.42267086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:41:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:41:08.421519478 +0000 UTC m=+1223.948854715" watchObservedRunningTime="2026-10-30 16:41:08.42267086 +0000 UTC m=+1223.950006079" Oct 30 16:41:08.462906 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:08.462506 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-f654570d-7b49ffb9f-2s4q4" podStartSLOduration=2.462489285 podStartE2EDuration="2.462489285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:41:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:41:08.461976962 +0000 UTC m=+1223.989312181" watchObservedRunningTime="2026-10-30 16:41:08.462489285 +0000 UTC m=+1223.989824505" Oct 30 16:41:08.683150 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:08.683113 5052 patch_prober.go:29] interesting pod/e2e-worker-w2-750dcb-openshift-default-777f567b9f-bg8br container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 30 16:41:08.683435 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:08.683402 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-bg8br" podUID="597c97ef-00c9-46b1-afbb-8b05363df559" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 30 16:41:09.681340 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:09.681303 5052 patch_prober.go:29] interesting pod/e2e-worker-w2-750dcb-openshift-default-777f567b9f-bg8br container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 30 16:41:09.681723 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:09.681363 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-bg8br" podUID="597c97ef-00c9-46b1-afbb-8b05363df559" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 30 16:41:10.681778 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:10.681745 5052 patch_prober.go:29] interesting pod/e2e-worker-w2-750dcb-openshift-default-777f567b9f-bg8br container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 30 16:41:10.682174 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:10.681796 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-bg8br" podUID="597c97ef-00c9-46b1-afbb-8b05363df559" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 30 16:41:11.168176 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:11.168141 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-4db2924d-openshift-default-bc6f66585-r4b8b"] Oct 30 16:41:11.172850 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:11.172827 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-4db2924d-openshift-default-bc6f66585-r4b8b" Oct 30 16:41:11.203819 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:11.203787 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-4db2924d-openshift-default-dockercfg-fgggn\"" Oct 30 16:41:11.209643 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:11.209618 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-4db2924d-openshift-default-bc6f66585-r4b8b"] Oct 30 16:41:11.260623 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:11.260530 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62e3ed6d-3cc2-4e79-9738-43a802b6d58b-workload-socket\") pod \"e2e-derive-4db2924d-openshift-default-bc6f66585-r4b8b\" (UID: \"62e3ed6d-3cc2-4e79-9738-43a802b6d58b\") " pod="openshift-ingress/e2e-derive-4db2924d-openshift-default-bc6f66585-r4b8b" Oct 30 16:41:11.260623 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:11.260621 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62e3ed6d-3cc2-4e79-9738-43a802b6d58b-istio-envoy\") pod \"e2e-derive-4db2924d-openshift-default-bc6f66585-r4b8b\" (UID: \"62e3ed6d-3cc2-4e79-9738-43a802b6d58b\") " pod="openshift-ingress/e2e-derive-4db2924d-openshift-default-bc6f66585-r4b8b" Oct 30 16:41:11.260880 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:11.260648 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62e3ed6d-3cc2-4e79-9738-43a802b6d58b-istio-token\") pod \"e2e-derive-4db2924d-openshift-default-bc6f66585-r4b8b\" (UID: \"62e3ed6d-3cc2-4e79-9738-43a802b6d58b\") " pod="openshift-ingress/e2e-derive-4db2924d-openshift-default-bc6f66585-r4b8b" Oct 30 16:41:11.260880 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:11.260769 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62e3ed6d-3cc2-4e79-9738-43a802b6d58b-istio-data\") pod \"e2e-derive-4db2924d-openshift-default-bc6f66585-r4b8b\" (UID: \"62e3ed6d-3cc2-4e79-9738-43a802b6d58b\") " pod="openshift-ingress/e2e-derive-4db2924d-openshift-default-bc6f66585-r4b8b" Oct 30 16:41:11.260880 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:11.260866 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62e3ed6d-3cc2-4e79-9738-43a802b6d58b-workload-certs\") pod \"e2e-derive-4db2924d-openshift-default-bc6f66585-r4b8b\" (UID: \"62e3ed6d-3cc2-4e79-9738-43a802b6d58b\") " pod="openshift-ingress/e2e-derive-4db2924d-openshift-default-bc6f66585-r4b8b" Oct 30 16:41:11.261057 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:11.260917 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtqfw\" (UniqueName: \"kubernetes.io/projected/62e3ed6d-3cc2-4e79-9738-43a802b6d58b-kube-api-access-qtqfw\") pod \"e2e-derive-4db2924d-openshift-default-bc6f66585-r4b8b\" (UID: \"62e3ed6d-3cc2-4e79-9738-43a802b6d58b\") " pod="openshift-ingress/e2e-derive-4db2924d-openshift-default-bc6f66585-r4b8b" Oct 30 16:41:11.261057 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:11.261041 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62e3ed6d-3cc2-4e79-9738-43a802b6d58b-istio-podinfo\") pod \"e2e-derive-4db2924d-openshift-default-bc6f66585-r4b8b\" (UID: \"62e3ed6d-3cc2-4e79-9738-43a802b6d58b\") " pod="openshift-ingress/e2e-derive-4db2924d-openshift-default-bc6f66585-r4b8b" Oct 30 16:41:11.261164 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:11.261081 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62e3ed6d-3cc2-4e79-9738-43a802b6d58b-istiod-ca-cert\") pod \"e2e-derive-4db2924d-openshift-default-bc6f66585-r4b8b\" (UID: \"62e3ed6d-3cc2-4e79-9738-43a802b6d58b\") " pod="openshift-ingress/e2e-derive-4db2924d-openshift-default-bc6f66585-r4b8b" Oct 30 16:41:11.261164 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:11.261109 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62e3ed6d-3cc2-4e79-9738-43a802b6d58b-credential-socket\") pod \"e2e-derive-4db2924d-openshift-default-bc6f66585-r4b8b\" (UID: \"62e3ed6d-3cc2-4e79-9738-43a802b6d58b\") " pod="openshift-ingress/e2e-derive-4db2924d-openshift-default-bc6f66585-r4b8b" Oct 30 16:41:11.423664 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:11.423409 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62e3ed6d-3cc2-4e79-9738-43a802b6d58b-istio-podinfo\") pod \"e2e-derive-4db2924d-openshift-default-bc6f66585-r4b8b\" (UID: \"62e3ed6d-3cc2-4e79-9738-43a802b6d58b\") " pod="openshift-ingress/e2e-derive-4db2924d-openshift-default-bc6f66585-r4b8b" Oct 30 16:41:11.423664 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:11.423468 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62e3ed6d-3cc2-4e79-9738-43a802b6d58b-istiod-ca-cert\") pod \"e2e-derive-4db2924d-openshift-default-bc6f66585-r4b8b\" (UID: \"62e3ed6d-3cc2-4e79-9738-43a802b6d58b\") " pod="openshift-ingress/e2e-derive-4db2924d-openshift-default-bc6f66585-r4b8b" Oct 30 16:41:11.423664 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:11.423501 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62e3ed6d-3cc2-4e79-9738-43a802b6d58b-credential-socket\") pod \"e2e-derive-4db2924d-openshift-default-bc6f66585-r4b8b\" (UID: \"62e3ed6d-3cc2-4e79-9738-43a802b6d58b\") " pod="openshift-ingress/e2e-derive-4db2924d-openshift-default-bc6f66585-r4b8b" Oct 30 16:41:11.423664 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:11.423552 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62e3ed6d-3cc2-4e79-9738-43a802b6d58b-workload-socket\") pod \"e2e-derive-4db2924d-openshift-default-bc6f66585-r4b8b\" (UID: \"62e3ed6d-3cc2-4e79-9738-43a802b6d58b\") " pod="openshift-ingress/e2e-derive-4db2924d-openshift-default-bc6f66585-r4b8b" Oct 30 16:41:11.423664 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:11.423627 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62e3ed6d-3cc2-4e79-9738-43a802b6d58b-istio-envoy\") pod \"e2e-derive-4db2924d-openshift-default-bc6f66585-r4b8b\" (UID: \"62e3ed6d-3cc2-4e79-9738-43a802b6d58b\") " pod="openshift-ingress/e2e-derive-4db2924d-openshift-default-bc6f66585-r4b8b" Oct 30 16:41:11.423664 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:11.423650 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62e3ed6d-3cc2-4e79-9738-43a802b6d58b-istio-token\") pod \"e2e-derive-4db2924d-openshift-default-bc6f66585-r4b8b\" (UID: \"62e3ed6d-3cc2-4e79-9738-43a802b6d58b\") " pod="openshift-ingress/e2e-derive-4db2924d-openshift-default-bc6f66585-r4b8b" Oct 30 16:41:11.424046 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:11.423772 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62e3ed6d-3cc2-4e79-9738-43a802b6d58b-istio-data\") pod \"e2e-derive-4db2924d-openshift-default-bc6f66585-r4b8b\" (UID: \"62e3ed6d-3cc2-4e79-9738-43a802b6d58b\") " pod="openshift-ingress/e2e-derive-4db2924d-openshift-default-bc6f66585-r4b8b" Oct 30 16:41:11.424046 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:11.423892 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62e3ed6d-3cc2-4e79-9738-43a802b6d58b-workload-certs\") pod \"e2e-derive-4db2924d-openshift-default-bc6f66585-r4b8b\" (UID: \"62e3ed6d-3cc2-4e79-9738-43a802b6d58b\") " pod="openshift-ingress/e2e-derive-4db2924d-openshift-default-bc6f66585-r4b8b" Oct 30 16:41:11.424046 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:11.423952 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtqfw\" (UniqueName: \"kubernetes.io/projected/62e3ed6d-3cc2-4e79-9738-43a802b6d58b-kube-api-access-qtqfw\") pod \"e2e-derive-4db2924d-openshift-default-bc6f66585-r4b8b\" (UID: \"62e3ed6d-3cc2-4e79-9738-43a802b6d58b\") " pod="openshift-ingress/e2e-derive-4db2924d-openshift-default-bc6f66585-r4b8b" Oct 30 16:41:11.429610 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:11.427264 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62e3ed6d-3cc2-4e79-9738-43a802b6d58b-credential-socket\") pod \"e2e-derive-4db2924d-openshift-default-bc6f66585-r4b8b\" (UID: \"62e3ed6d-3cc2-4e79-9738-43a802b6d58b\") " pod="openshift-ingress/e2e-derive-4db2924d-openshift-default-bc6f66585-r4b8b" Oct 30 16:41:11.429610 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:11.427966 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62e3ed6d-3cc2-4e79-9738-43a802b6d58b-istio-data\") pod \"e2e-derive-4db2924d-openshift-default-bc6f66585-r4b8b\" (UID: \"62e3ed6d-3cc2-4e79-9738-43a802b6d58b\") " pod="openshift-ingress/e2e-derive-4db2924d-openshift-default-bc6f66585-r4b8b" Oct 30 16:41:11.429610 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:11.428286 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62e3ed6d-3cc2-4e79-9738-43a802b6d58b-istiod-ca-cert\") pod \"e2e-derive-4db2924d-openshift-default-bc6f66585-r4b8b\" (UID: \"62e3ed6d-3cc2-4e79-9738-43a802b6d58b\") " pod="openshift-ingress/e2e-derive-4db2924d-openshift-default-bc6f66585-r4b8b" Oct 30 16:41:11.429610 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:11.428609 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62e3ed6d-3cc2-4e79-9738-43a802b6d58b-workload-certs\") pod \"e2e-derive-4db2924d-openshift-default-bc6f66585-r4b8b\" (UID: \"62e3ed6d-3cc2-4e79-9738-43a802b6d58b\") " pod="openshift-ingress/e2e-derive-4db2924d-openshift-default-bc6f66585-r4b8b" Oct 30 16:41:11.455921 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:11.455886 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62e3ed6d-3cc2-4e79-9738-43a802b6d58b-workload-socket\") pod \"e2e-derive-4db2924d-openshift-default-bc6f66585-r4b8b\" (UID: \"62e3ed6d-3cc2-4e79-9738-43a802b6d58b\") " pod="openshift-ingress/e2e-derive-4db2924d-openshift-default-bc6f66585-r4b8b" Oct 30 16:41:11.549605 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:11.548802 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtqfw\" (UniqueName: \"kubernetes.io/projected/62e3ed6d-3cc2-4e79-9738-43a802b6d58b-kube-api-access-qtqfw\") pod \"e2e-derive-4db2924d-openshift-default-bc6f66585-r4b8b\" (UID: \"62e3ed6d-3cc2-4e79-9738-43a802b6d58b\") " pod="openshift-ingress/e2e-derive-4db2924d-openshift-default-bc6f66585-r4b8b" Oct 30 16:41:11.566167 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:11.566000 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62e3ed6d-3cc2-4e79-9738-43a802b6d58b-istio-token\") pod \"e2e-derive-4db2924d-openshift-default-bc6f66585-r4b8b\" (UID: \"62e3ed6d-3cc2-4e79-9738-43a802b6d58b\") " pod="openshift-ingress/e2e-derive-4db2924d-openshift-default-bc6f66585-r4b8b" Oct 30 16:41:11.587347 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:11.580986 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62e3ed6d-3cc2-4e79-9738-43a802b6d58b-istio-envoy\") pod \"e2e-derive-4db2924d-openshift-default-bc6f66585-r4b8b\" (UID: \"62e3ed6d-3cc2-4e79-9738-43a802b6d58b\") " pod="openshift-ingress/e2e-derive-4db2924d-openshift-default-bc6f66585-r4b8b" Oct 30 16:41:11.610480 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:11.610448 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62e3ed6d-3cc2-4e79-9738-43a802b6d58b-istio-podinfo\") pod \"e2e-derive-4db2924d-openshift-default-bc6f66585-r4b8b\" (UID: \"62e3ed6d-3cc2-4e79-9738-43a802b6d58b\") " pod="openshift-ingress/e2e-derive-4db2924d-openshift-default-bc6f66585-r4b8b" Oct 30 16:41:11.692658 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:11.684376 5052 patch_prober.go:29] interesting pod/e2e-worker-w2-750dcb-openshift-default-777f567b9f-bg8br container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 30 16:41:11.692658 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:11.684459 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-bg8br" podUID="597c97ef-00c9-46b1-afbb-8b05363df559" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 30 16:41:11.806654 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:11.806618 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-4db2924d-openshift-default-bc6f66585-r4b8b" Oct 30 16:41:12.207715 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:12.199654 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-4db2924d-openshift-default-bc6f66585-r4b8b"] Oct 30 16:41:12.238960 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:12.237961 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:41:12.238960 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:12.238084 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:41:12.238960 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:12.238158 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:41:12.585238 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:12.585148 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-4db2924d-openshift-default-bc6f66585-r4b8b" event={"ID":"62e3ed6d-3cc2-4e79-9738-43a802b6d58b","Type":"ContainerStarted","Data":"38c010033fc2bc3d8d86c019978d27604a98087fef69dd5b1df3144af7f05b6d"} Oct 30 16:41:12.585238 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:12.585216 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-4db2924d-openshift-default-bc6f66585-r4b8b" event={"ID":"62e3ed6d-3cc2-4e79-9738-43a802b6d58b","Type":"ContainerStarted","Data":"ecc7ea57c684fc75a985e62f77715ebcdad2e7d244f244796860a91e43295491"} Oct 30 16:41:12.643385 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:12.643322 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-4db2924d-openshift-default-bc6f66585-r4b8b" podStartSLOduration=1.6433046500000001 podStartE2EDuration="1.64330465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:41:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:41:12.642660722 +0000 UTC m=+1228.169995970" watchObservedRunningTime="2026-10-30 16:41:12.64330465 +0000 UTC m=+1228.170639870" Oct 30 16:41:12.806842 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:12.806808 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-4db2924d-openshift-default-bc6f66585-r4b8b" Oct 30 16:41:13.155601 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:13.155559 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-bg8br" Oct 30 16:41:13.155953 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:13.155931 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-bg8br" Oct 30 16:41:13.156870 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:13.156836 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-bg8br" Oct 30 16:41:13.812598 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:13.812555 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-4db2924d-openshift-default-bc6f66585-r4b8b" Oct 30 16:41:14.600635 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:14.600604 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-4db2924d-openshift-default-bc6f66585-r4b8b" Oct 30 16:41:14.601393 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:14.601375 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-4db2924d-openshift-default-bc6f66585-r4b8b" Oct 30 16:41:16.026425 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:16.026392 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-d693d0/distinct-2-w6-d693d0-kserve-6498fdc474-dpbjj"] Oct 30 16:41:16.030898 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:16.030877 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-d693d0/distinct-2-w6-d693d0-kserve-6498fdc474-dpbjj" Oct 30 16:41:16.057453 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:16.057417 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-d693d0\"/\"kube-root-ca.crt\"" Oct 30 16:41:16.057665 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:16.057509 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-d693d0\"/\"openshift-service-ca.crt\"" Oct 30 16:41:16.057665 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:16.057619 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-d693d0\"/\"default-dockercfg-xznkg\"" Oct 30 16:41:16.059310 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:16.059286 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-d693d0\"/\"distinct-2-w6-d693d0-kserve-self-signed-certs\"" Oct 30 16:41:16.061826 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:16.061805 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-d693d0/distinct-2-w6-d693d0-kserve-6498fdc474-dpbjj"] Oct 30 16:41:16.094787 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:16.094753 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b98b8ff-72cd-4ddb-a4b8-67078162e611-home\") pod \"distinct-2-w6-d693d0-kserve-6498fdc474-dpbjj\" (UID: \"4b98b8ff-72cd-4ddb-a4b8-67078162e611\") " pod="e2e-models-e2e-worker-w6-d693d0/distinct-2-w6-d693d0-kserve-6498fdc474-dpbjj" Oct 30 16:41:16.094943 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:16.094843 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b98b8ff-72cd-4ddb-a4b8-67078162e611-model-cache\") pod \"distinct-2-w6-d693d0-kserve-6498fdc474-dpbjj\" (UID: \"4b98b8ff-72cd-4ddb-a4b8-67078162e611\") " pod="e2e-models-e2e-worker-w6-d693d0/distinct-2-w6-d693d0-kserve-6498fdc474-dpbjj" Oct 30 16:41:16.095015 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:16.094974 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b98b8ff-72cd-4ddb-a4b8-67078162e611-dshm\") pod \"distinct-2-w6-d693d0-kserve-6498fdc474-dpbjj\" (UID: \"4b98b8ff-72cd-4ddb-a4b8-67078162e611\") " pod="e2e-models-e2e-worker-w6-d693d0/distinct-2-w6-d693d0-kserve-6498fdc474-dpbjj" Oct 30 16:41:16.095071 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:16.095027 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzqt2\" (UniqueName: \"kubernetes.io/projected/4b98b8ff-72cd-4ddb-a4b8-67078162e611-kube-api-access-xzqt2\") pod \"distinct-2-w6-d693d0-kserve-6498fdc474-dpbjj\" (UID: \"4b98b8ff-72cd-4ddb-a4b8-67078162e611\") " pod="e2e-models-e2e-worker-w6-d693d0/distinct-2-w6-d693d0-kserve-6498fdc474-dpbjj" Oct 30 16:41:16.095133 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:16.095110 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b98b8ff-72cd-4ddb-a4b8-67078162e611-tls-certs\") pod \"distinct-2-w6-d693d0-kserve-6498fdc474-dpbjj\" (UID: \"4b98b8ff-72cd-4ddb-a4b8-67078162e611\") " pod="e2e-models-e2e-worker-w6-d693d0/distinct-2-w6-d693d0-kserve-6498fdc474-dpbjj" Oct 30 16:41:16.095187 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:16.095145 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b98b8ff-72cd-4ddb-a4b8-67078162e611-tmp-dir\") pod \"distinct-2-w6-d693d0-kserve-6498fdc474-dpbjj\" (UID: \"4b98b8ff-72cd-4ddb-a4b8-67078162e611\") " pod="e2e-models-e2e-worker-w6-d693d0/distinct-2-w6-d693d0-kserve-6498fdc474-dpbjj" Oct 30 16:41:16.196505 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:16.196466 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b98b8ff-72cd-4ddb-a4b8-67078162e611-model-cache\") pod \"distinct-2-w6-d693d0-kserve-6498fdc474-dpbjj\" (UID: \"4b98b8ff-72cd-4ddb-a4b8-67078162e611\") " pod="e2e-models-e2e-worker-w6-d693d0/distinct-2-w6-d693d0-kserve-6498fdc474-dpbjj" Oct 30 16:41:16.196702 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:16.196609 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b98b8ff-72cd-4ddb-a4b8-67078162e611-dshm\") pod \"distinct-2-w6-d693d0-kserve-6498fdc474-dpbjj\" (UID: \"4b98b8ff-72cd-4ddb-a4b8-67078162e611\") " pod="e2e-models-e2e-worker-w6-d693d0/distinct-2-w6-d693d0-kserve-6498fdc474-dpbjj" Oct 30 16:41:16.196702 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:16.196666 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzqt2\" (UniqueName: \"kubernetes.io/projected/4b98b8ff-72cd-4ddb-a4b8-67078162e611-kube-api-access-xzqt2\") pod \"distinct-2-w6-d693d0-kserve-6498fdc474-dpbjj\" (UID: \"4b98b8ff-72cd-4ddb-a4b8-67078162e611\") " pod="e2e-models-e2e-worker-w6-d693d0/distinct-2-w6-d693d0-kserve-6498fdc474-dpbjj" Oct 30 16:41:16.196827 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:16.196798 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b98b8ff-72cd-4ddb-a4b8-67078162e611-tls-certs\") pod \"distinct-2-w6-d693d0-kserve-6498fdc474-dpbjj\" (UID: \"4b98b8ff-72cd-4ddb-a4b8-67078162e611\") " pod="e2e-models-e2e-worker-w6-d693d0/distinct-2-w6-d693d0-kserve-6498fdc474-dpbjj" Oct 30 16:41:16.196869 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:16.196852 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b98b8ff-72cd-4ddb-a4b8-67078162e611-tmp-dir\") pod \"distinct-2-w6-d693d0-kserve-6498fdc474-dpbjj\" (UID: \"4b98b8ff-72cd-4ddb-a4b8-67078162e611\") " pod="e2e-models-e2e-worker-w6-d693d0/distinct-2-w6-d693d0-kserve-6498fdc474-dpbjj" Oct 30 16:41:16.196922 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:16.196875 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b98b8ff-72cd-4ddb-a4b8-67078162e611-model-cache\") pod \"distinct-2-w6-d693d0-kserve-6498fdc474-dpbjj\" (UID: \"4b98b8ff-72cd-4ddb-a4b8-67078162e611\") " pod="e2e-models-e2e-worker-w6-d693d0/distinct-2-w6-d693d0-kserve-6498fdc474-dpbjj" Oct 30 16:41:16.196922 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:16.196908 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b98b8ff-72cd-4ddb-a4b8-67078162e611-home\") pod \"distinct-2-w6-d693d0-kserve-6498fdc474-dpbjj\" (UID: \"4b98b8ff-72cd-4ddb-a4b8-67078162e611\") " pod="e2e-models-e2e-worker-w6-d693d0/distinct-2-w6-d693d0-kserve-6498fdc474-dpbjj" Oct 30 16:41:16.197184 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:16.197157 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b98b8ff-72cd-4ddb-a4b8-67078162e611-tmp-dir\") pod \"distinct-2-w6-d693d0-kserve-6498fdc474-dpbjj\" (UID: \"4b98b8ff-72cd-4ddb-a4b8-67078162e611\") " pod="e2e-models-e2e-worker-w6-d693d0/distinct-2-w6-d693d0-kserve-6498fdc474-dpbjj" Oct 30 16:41:16.197261 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:16.197194 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b98b8ff-72cd-4ddb-a4b8-67078162e611-home\") pod \"distinct-2-w6-d693d0-kserve-6498fdc474-dpbjj\" (UID: \"4b98b8ff-72cd-4ddb-a4b8-67078162e611\") " pod="e2e-models-e2e-worker-w6-d693d0/distinct-2-w6-d693d0-kserve-6498fdc474-dpbjj" Oct 30 16:41:16.198856 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:16.198828 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b98b8ff-72cd-4ddb-a4b8-67078162e611-dshm\") pod \"distinct-2-w6-d693d0-kserve-6498fdc474-dpbjj\" (UID: \"4b98b8ff-72cd-4ddb-a4b8-67078162e611\") " pod="e2e-models-e2e-worker-w6-d693d0/distinct-2-w6-d693d0-kserve-6498fdc474-dpbjj" Oct 30 16:41:16.199337 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:16.199306 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b98b8ff-72cd-4ddb-a4b8-67078162e611-tls-certs\") pod \"distinct-2-w6-d693d0-kserve-6498fdc474-dpbjj\" (UID: \"4b98b8ff-72cd-4ddb-a4b8-67078162e611\") " pod="e2e-models-e2e-worker-w6-d693d0/distinct-2-w6-d693d0-kserve-6498fdc474-dpbjj" Oct 30 16:41:16.220853 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:16.220828 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzqt2\" (UniqueName: \"kubernetes.io/projected/4b98b8ff-72cd-4ddb-a4b8-67078162e611-kube-api-access-xzqt2\") pod \"distinct-2-w6-d693d0-kserve-6498fdc474-dpbjj\" (UID: \"4b98b8ff-72cd-4ddb-a4b8-67078162e611\") " pod="e2e-models-e2e-worker-w6-d693d0/distinct-2-w6-d693d0-kserve-6498fdc474-dpbjj" Oct 30 16:41:16.455782 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:16.393857 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-4db2924d-6ff4bd685d-mkkgw"] Oct 30 16:41:16.455782 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:16.399790 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-4db2924d-6ff4bd685d-mkkgw" Oct 30 16:41:16.455782 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:16.405851 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-d693d0/distinct-2-w6-d693d0-kserve-6498fdc474-dpbjj" Oct 30 16:41:16.455782 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:16.417650 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-4db2924d-dockercfg-5s6qs\"" Oct 30 16:41:16.455782 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:16.418347 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-4db2924d\"" Oct 30 16:41:16.455782 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:16.438014 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kj5v8\" (UniqueName: \"kubernetes.io/projected/d00a8c79-7b28-4ea2-97ea-d76e2df5e441-kube-api-access-kj5v8\") pod \"payload-processing-e2e-derive-4db2924d-6ff4bd685d-mkkgw\" (UID: \"d00a8c79-7b28-4ea2-97ea-d76e2df5e441\") " pod="openshift-ingress/payload-processing-e2e-derive-4db2924d-6ff4bd685d-mkkgw" Oct 30 16:41:16.455782 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:16.438199 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d00a8c79-7b28-4ea2-97ea-d76e2df5e441-plugins-config-volume\") pod \"payload-processing-e2e-derive-4db2924d-6ff4bd685d-mkkgw\" (UID: \"d00a8c79-7b28-4ea2-97ea-d76e2df5e441\") " pod="openshift-ingress/payload-processing-e2e-derive-4db2924d-6ff4bd685d-mkkgw" Oct 30 16:41:16.471543 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:16.471451 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-4db2924d-6ff4bd685d-mkkgw"] Oct 30 16:41:16.539935 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:16.539906 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kj5v8\" (UniqueName: \"kubernetes.io/projected/d00a8c79-7b28-4ea2-97ea-d76e2df5e441-kube-api-access-kj5v8\") pod \"payload-processing-e2e-derive-4db2924d-6ff4bd685d-mkkgw\" (UID: \"d00a8c79-7b28-4ea2-97ea-d76e2df5e441\") " pod="openshift-ingress/payload-processing-e2e-derive-4db2924d-6ff4bd685d-mkkgw" Oct 30 16:41:16.540123 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:16.540037 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d00a8c79-7b28-4ea2-97ea-d76e2df5e441-plugins-config-volume\") pod \"payload-processing-e2e-derive-4db2924d-6ff4bd685d-mkkgw\" (UID: \"d00a8c79-7b28-4ea2-97ea-d76e2df5e441\") " pod="openshift-ingress/payload-processing-e2e-derive-4db2924d-6ff4bd685d-mkkgw" Oct 30 16:41:16.542928 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:16.541482 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d00a8c79-7b28-4ea2-97ea-d76e2df5e441-plugins-config-volume\") pod \"payload-processing-e2e-derive-4db2924d-6ff4bd685d-mkkgw\" (UID: \"d00a8c79-7b28-4ea2-97ea-d76e2df5e441\") " pod="openshift-ingress/payload-processing-e2e-derive-4db2924d-6ff4bd685d-mkkgw" Oct 30 16:41:16.590855 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:16.569739 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kj5v8\" (UniqueName: \"kubernetes.io/projected/d00a8c79-7b28-4ea2-97ea-d76e2df5e441-kube-api-access-kj5v8\") pod \"payload-processing-e2e-derive-4db2924d-6ff4bd685d-mkkgw\" (UID: \"d00a8c79-7b28-4ea2-97ea-d76e2df5e441\") " pod="openshift-ingress/payload-processing-e2e-derive-4db2924d-6ff4bd685d-mkkgw" Oct 30 16:41:16.853025 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:16.852940 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-4db2924d-6ff4bd685d-mkkgw" Oct 30 16:41:16.936754 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:16.936720 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-d693d0/distinct-2-w6-d693d0-kserve-6498fdc474-dpbjj"] Oct 30 16:41:17.170641 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:17.170613 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-4db2924d-6ff4bd685d-mkkgw"] Oct 30 16:41:17.172292 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:41:17.172264 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd00a8c79_7b28_4ea2_97ea_d76e2df5e441.slice/crio-fc041e09ce4b5e87f1441bb1e7a6f6da176e19e45668c6522bb8553d2f74502c WatchSource:0}: Error finding container fc041e09ce4b5e87f1441bb1e7a6f6da176e19e45668c6522bb8553d2f74502c: Status 404 returned error can't find the container with id fc041e09ce4b5e87f1441bb1e7a6f6da176e19e45668c6522bb8553d2f74502c Oct 30 16:41:17.641935 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:17.641856 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-4db2924d-6ff4bd685d-mkkgw" event={"ID":"d00a8c79-7b28-4ea2-97ea-d76e2df5e441","Type":"ContainerStarted","Data":"dd5c94607bc92e2b64cc06f85b82178f2f5c9655850f826d7c1e92a50b47f269"} Oct 30 16:41:17.642095 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:17.641957 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-4db2924d-6ff4bd685d-mkkgw" Oct 30 16:41:17.642095 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:17.641983 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-4db2924d-6ff4bd685d-mkkgw" event={"ID":"d00a8c79-7b28-4ea2-97ea-d76e2df5e441","Type":"ContainerStarted","Data":"fc041e09ce4b5e87f1441bb1e7a6f6da176e19e45668c6522bb8553d2f74502c"} Oct 30 16:41:17.643478 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:17.643450 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-d693d0/distinct-2-w6-d693d0-kserve-6498fdc474-dpbjj" event={"ID":"4b98b8ff-72cd-4ddb-a4b8-67078162e611","Type":"ContainerStarted","Data":"b03a9b24cafd8486dcb27b366c6f9a29cd35a431dee7b63a90e5706373940915"} Oct 30 16:41:17.643478 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:17.643479 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-d693d0/distinct-2-w6-d693d0-kserve-6498fdc474-dpbjj" event={"ID":"4b98b8ff-72cd-4ddb-a4b8-67078162e611","Type":"ContainerStarted","Data":"2ac477a7bd28303720ff3fcdff4e5036f0a74c2c96f6362e51462efc5b7365bd"} Oct 30 16:41:17.685049 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:17.685004 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-4db2924d-6ff4bd685d-mkkgw" podStartSLOduration=1.684991766 podStartE2EDuration="1.684991766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:41:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:41:17.680747278 +0000 UTC m=+1233.208082508" watchObservedRunningTime="2026-10-30 16:41:17.684991766 +0000 UTC m=+1233.212326985" Oct 30 16:41:18.411256 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:18.411227 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d693d0/distinct-2-w6-d693d0-kserve-6498fdc474-dpbjj" Oct 30 16:41:18.842318 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:41:18.842237 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 16:41:19.212931 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:19.212874 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-d693d0/distinct-2-w6-d693d0-kserve-6498fdc474-dpbjj" podStartSLOduration=3.212860035 podStartE2EDuration="3.212860035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:41:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:41:17.720888924 +0000 UTC m=+1233.248224145" watchObservedRunningTime="2026-10-30 16:41:19.212860035 +0000 UTC m=+1234.740195271" Oct 30 16:41:19.213867 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:19.213850 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cp9m8"] Oct 30 16:41:19.218339 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:19.218317 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cp9m8" Oct 30 16:41:19.248862 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:19.248832 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cp9m8"] Oct 30 16:41:19.287288 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:19.287247 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee44984b-c448-4580-aa5a-637c73d09e19-workload-certs\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-cp9m8\" (UID: \"ee44984b-c448-4580-aa5a-637c73d09e19\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cp9m8" Oct 30 16:41:19.287459 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:19.287298 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee44984b-c448-4580-aa5a-637c73d09e19-istio-podinfo\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-cp9m8\" (UID: \"ee44984b-c448-4580-aa5a-637c73d09e19\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cp9m8" Oct 30 16:41:19.287459 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:19.287375 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee44984b-c448-4580-aa5a-637c73d09e19-credential-socket\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-cp9m8\" (UID: \"ee44984b-c448-4580-aa5a-637c73d09e19\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cp9m8" Oct 30 16:41:19.287459 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:19.287440 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee44984b-c448-4580-aa5a-637c73d09e19-workload-socket\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-cp9m8\" (UID: \"ee44984b-c448-4580-aa5a-637c73d09e19\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cp9m8" Oct 30 16:41:19.287702 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:19.287524 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69n2v\" (UniqueName: \"kubernetes.io/projected/ee44984b-c448-4580-aa5a-637c73d09e19-kube-api-access-69n2v\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-cp9m8\" (UID: \"ee44984b-c448-4580-aa5a-637c73d09e19\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cp9m8" Oct 30 16:41:19.287702 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:19.287637 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee44984b-c448-4580-aa5a-637c73d09e19-istiod-ca-cert\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-cp9m8\" (UID: \"ee44984b-c448-4580-aa5a-637c73d09e19\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cp9m8" Oct 30 16:41:19.287702 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:19.287689 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee44984b-c448-4580-aa5a-637c73d09e19-istio-data\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-cp9m8\" (UID: \"ee44984b-c448-4580-aa5a-637c73d09e19\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cp9m8" Oct 30 16:41:19.287852 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:19.287708 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee44984b-c448-4580-aa5a-637c73d09e19-istio-envoy\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-cp9m8\" (UID: \"ee44984b-c448-4580-aa5a-637c73d09e19\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cp9m8" Oct 30 16:41:19.287852 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:19.287739 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee44984b-c448-4580-aa5a-637c73d09e19-istio-token\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-cp9m8\" (UID: \"ee44984b-c448-4580-aa5a-637c73d09e19\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cp9m8" Oct 30 16:41:19.389398 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:19.389357 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee44984b-c448-4580-aa5a-637c73d09e19-workload-certs\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-cp9m8\" (UID: \"ee44984b-c448-4580-aa5a-637c73d09e19\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cp9m8" Oct 30 16:41:19.389574 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:19.389417 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee44984b-c448-4580-aa5a-637c73d09e19-istio-podinfo\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-cp9m8\" (UID: \"ee44984b-c448-4580-aa5a-637c73d09e19\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cp9m8" Oct 30 16:41:19.389574 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:19.389442 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee44984b-c448-4580-aa5a-637c73d09e19-credential-socket\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-cp9m8\" (UID: \"ee44984b-c448-4580-aa5a-637c73d09e19\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cp9m8" Oct 30 16:41:19.389574 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:19.389502 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee44984b-c448-4580-aa5a-637c73d09e19-workload-socket\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-cp9m8\" (UID: \"ee44984b-c448-4580-aa5a-637c73d09e19\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cp9m8" Oct 30 16:41:19.389777 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:19.389658 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69n2v\" (UniqueName: \"kubernetes.io/projected/ee44984b-c448-4580-aa5a-637c73d09e19-kube-api-access-69n2v\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-cp9m8\" (UID: \"ee44984b-c448-4580-aa5a-637c73d09e19\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cp9m8" Oct 30 16:41:19.389836 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:19.389779 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee44984b-c448-4580-aa5a-637c73d09e19-istiod-ca-cert\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-cp9m8\" (UID: \"ee44984b-c448-4580-aa5a-637c73d09e19\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cp9m8" Oct 30 16:41:19.389886 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:19.389845 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee44984b-c448-4580-aa5a-637c73d09e19-workload-certs\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-cp9m8\" (UID: \"ee44984b-c448-4580-aa5a-637c73d09e19\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cp9m8" Oct 30 16:41:19.389886 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:19.389852 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee44984b-c448-4580-aa5a-637c73d09e19-istio-data\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-cp9m8\" (UID: \"ee44984b-c448-4580-aa5a-637c73d09e19\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cp9m8" Oct 30 16:41:19.389988 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:19.389886 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee44984b-c448-4580-aa5a-637c73d09e19-credential-socket\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-cp9m8\" (UID: \"ee44984b-c448-4580-aa5a-637c73d09e19\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cp9m8" Oct 30 16:41:19.389988 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:19.389923 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee44984b-c448-4580-aa5a-637c73d09e19-istio-envoy\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-cp9m8\" (UID: \"ee44984b-c448-4580-aa5a-637c73d09e19\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cp9m8" Oct 30 16:41:19.389988 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:19.389970 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee44984b-c448-4580-aa5a-637c73d09e19-istio-token\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-cp9m8\" (UID: \"ee44984b-c448-4580-aa5a-637c73d09e19\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cp9m8" Oct 30 16:41:19.390134 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:19.389925 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee44984b-c448-4580-aa5a-637c73d09e19-workload-socket\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-cp9m8\" (UID: \"ee44984b-c448-4580-aa5a-637c73d09e19\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cp9m8" Oct 30 16:41:19.390134 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:19.390125 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee44984b-c448-4580-aa5a-637c73d09e19-istio-data\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-cp9m8\" (UID: \"ee44984b-c448-4580-aa5a-637c73d09e19\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cp9m8" Oct 30 16:41:19.390988 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:19.390959 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee44984b-c448-4580-aa5a-637c73d09e19-istiod-ca-cert\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-cp9m8\" (UID: \"ee44984b-c448-4580-aa5a-637c73d09e19\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cp9m8" Oct 30 16:41:19.392203 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:19.392166 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee44984b-c448-4580-aa5a-637c73d09e19-istio-podinfo\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-cp9m8\" (UID: \"ee44984b-c448-4580-aa5a-637c73d09e19\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cp9m8" Oct 30 16:41:19.392203 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:19.392181 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee44984b-c448-4580-aa5a-637c73d09e19-istio-envoy\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-cp9m8\" (UID: \"ee44984b-c448-4580-aa5a-637c73d09e19\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cp9m8" Oct 30 16:41:19.394618 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:19.394393 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-f654570d-7b49ffb9f-2s4q4" Oct 30 16:41:19.394714 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:19.394654 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-f654570d-56d5957948-lh66x" Oct 30 16:41:19.413967 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:19.413948 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee44984b-c448-4580-aa5a-637c73d09e19-istio-token\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-cp9m8\" (UID: \"ee44984b-c448-4580-aa5a-637c73d09e19\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cp9m8" Oct 30 16:41:19.414269 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:19.414100 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69n2v\" (UniqueName: \"kubernetes.io/projected/ee44984b-c448-4580-aa5a-637c73d09e19-kube-api-access-69n2v\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-cp9m8\" (UID: \"ee44984b-c448-4580-aa5a-637c73d09e19\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cp9m8" Oct 30 16:41:19.547433 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:19.534927 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cp9m8" Oct 30 16:41:19.994150 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:19.994118 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cp9m8"] Oct 30 16:41:19.994626 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:41:19.994597 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee44984b_c448_4580_aa5a_637c73d09e19.slice/crio-92f82870a9f063cacf8016a3fb8ef10ad6b9615e4b43bf5dc0970c414ee94074 WatchSource:0}: Error finding container 92f82870a9f063cacf8016a3fb8ef10ad6b9615e4b43bf5dc0970c414ee94074: Status 404 returned error can't find the container with id 92f82870a9f063cacf8016a3fb8ef10ad6b9615e4b43bf5dc0970c414ee94074 Oct 30 16:41:19.997203 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:19.997167 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:41:19.997268 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:19.997252 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:41:19.997308 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:19.997285 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:41:20.820337 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:20.818031 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cp9m8" event={"ID":"ee44984b-c448-4580-aa5a-637c73d09e19","Type":"ContainerStarted","Data":"d46b8129f7de87e1ddf19c8dd248d0ec3e2f04a3ccf0bf6d3f1a6876fbb75a06"} Oct 30 16:41:20.820337 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:20.818091 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cp9m8" event={"ID":"ee44984b-c448-4580-aa5a-637c73d09e19","Type":"ContainerStarted","Data":"92f82870a9f063cacf8016a3fb8ef10ad6b9615e4b43bf5dc0970c414ee94074"} Oct 30 16:41:20.890441 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:20.890382 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cp9m8" podStartSLOduration=1.890364618 podStartE2EDuration="1.890364618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:41:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:41:20.8749277 +0000 UTC m=+1236.402262920" watchObservedRunningTime="2026-10-30 16:41:20.890364618 +0000 UTC m=+1236.417699853" Oct 30 16:41:21.543939 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:21.543904 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cp9m8" Oct 30 16:41:21.544337 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:21.544312 5052 patch_prober.go:29] interesting pod/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cp9m8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 16:41:21.544546 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:21.544519 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cp9m8" podUID="ee44984b-c448-4580-aa5a-637c73d09e19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 16:41:22.545605 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:22.545024 5052 patch_prober.go:29] interesting pod/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cp9m8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 16:41:22.545605 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:22.545105 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cp9m8" podUID="ee44984b-c448-4580-aa5a-637c73d09e19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 16:41:23.557603 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:23.544572 5052 patch_prober.go:29] interesting pod/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cp9m8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 16:41:23.557603 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:23.544668 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cp9m8" podUID="ee44984b-c448-4580-aa5a-637c73d09e19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 16:41:24.453737 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:24.453678 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/distinct-w5-967fac-kserve-69cbcccd8b-b85bn" podUID="8f6b9354-378f-4451-9ff7-be7f7fec12f2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:24.455939 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:24.454928 5052 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-967fac/distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx" podUID="bd662f3a-659f-477c-b8d5-94c64288c0c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:24.504903 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:24.504858 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx" podUID="bd662f3a-659f-477c-b8d5-94c64288c0c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 16:41:24.545901 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:24.545867 5052 patch_prober.go:29] interesting pod/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cp9m8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 16:41:24.546165 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:24.546139 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cp9m8" podUID="ee44984b-c448-4580-aa5a-637c73d09e19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 16:41:24.765666 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:24.760695 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/distinct-2-w4-263f00-kserve-8464897cf5-q8ck8" podUID="15ed43ef-5cdd-489a-a70e-063239822ff3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:25.262747 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:25.262707 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-750dcb/distinct-w2-750dcb-kserve-86bbdfcbb-bpn77" podUID="c45190af-d795-4e13-8f2f-670b2c597658" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:25.462779 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:25.462733 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/distinct-w5-967fac-kserve-69cbcccd8b-b85bn" podUID="8f6b9354-378f-4451-9ff7-be7f7fec12f2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Oct 30 16:41:25.463607 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:25.463559 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-967fac/distinct-w5-967fac-kserve-69cbcccd8b-b85bn" Oct 30 16:41:25.506744 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:25.506701 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx" podUID="bd662f3a-659f-477c-b8d5-94c64288c0c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:25.507417 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:25.507394 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-967fac/distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx" Oct 30 16:41:25.539224 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:25.539139 5052 patch_prober.go:29] interesting pod/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cp9m8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 16:41:25.540278 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:25.540243 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cp9m8" podUID="ee44984b-c448-4580-aa5a-637c73d09e19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 16:41:25.857664 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:25.839026 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/distinct-2-w4-263f00-kserve-8464897cf5-q8ck8" podUID="15ed43ef-5cdd-489a-a70e-063239822ff3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 30 16:41:25.923636 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:25.885364 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-263f00/distinct-2-w4-263f00-kserve-8464897cf5-q8ck8" Oct 30 16:41:26.268607 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:26.264636 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-750dcb/distinct-w2-750dcb-kserve-86bbdfcbb-bpn77" podUID="c45190af-d795-4e13-8f2f-670b2c597658" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:26.352604 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:26.351953 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-750dcb/distinct-w2-750dcb-kserve-86bbdfcbb-bpn77" Oct 30 16:41:26.482480 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:26.432669 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-d693d0/distinct-2-w6-d693d0-kserve-6498fdc474-dpbjj" Oct 30 16:41:26.482480 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:26.468879 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/distinct-w5-967fac-kserve-69cbcccd8b-b85bn" podUID="8f6b9354-378f-4451-9ff7-be7f7fec12f2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:26.509607 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:26.507923 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx" podUID="bd662f3a-659f-477c-b8d5-94c64288c0c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 16:41:26.579659 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:26.547880 5052 patch_prober.go:29] interesting pod/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cp9m8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 16:41:26.579659 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:26.547944 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cp9m8" podUID="ee44984b-c448-4580-aa5a-637c73d09e19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 16:41:26.862661 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:26.857301 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/distinct-2-w4-263f00-kserve-8464897cf5-q8ck8" podUID="15ed43ef-5cdd-489a-a70e-063239822ff3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:27.335605 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:27.314720 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-750dcb/distinct-w2-750dcb-kserve-86bbdfcbb-bpn77" podUID="c45190af-d795-4e13-8f2f-670b2c597658" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded" Oct 30 16:41:27.438605 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:27.402686 5052 patch_prober.go:29] interesting pod/e2e-worker-w4-263f00-openshift-default-b969d55bc-f7blz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:41:27.438949 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:27.438912 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-f7blz" podUID="1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:27.458664 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:27.424176 5052 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-models-e2e-worker-w6-d693d0/distinct-2-w6-d693d0-kserve-6498fdc474-dpbjj" podUID="4b98b8ff-72cd-4ddb-a4b8-67078162e611" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:27.499612 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:27.496927 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/distinct-w5-967fac-kserve-69cbcccd8b-b85bn" podUID="8f6b9354-378f-4451-9ff7-be7f7fec12f2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:27.499612 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:27.498875 5052 patch_prober.go:29] interesting pod/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hkcvj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:41:27.499612 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:27.498947 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hkcvj" podUID="545986bf-3dc2-4204-9fd1-ac7085df215b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:27.536606 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:27.527900 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx" podUID="bd662f3a-659f-477c-b8d5-94c64288c0c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 16:41:27.579607 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:27.578835 5052 patch_prober.go:29] interesting pod/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cp9m8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 16:41:27.579607 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:27.578900 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cp9m8" podUID="ee44984b-c448-4580-aa5a-637c73d09e19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 16:41:27.925219 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:27.916725 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/distinct-2-w4-263f00-kserve-8464897cf5-q8ck8" podUID="15ed43ef-5cdd-489a-a70e-063239822ff3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:28.472981 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:28.472936 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-750dcb/distinct-w2-750dcb-kserve-86bbdfcbb-bpn77" podUID="c45190af-d795-4e13-8f2f-670b2c597658" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded" Oct 30 16:41:28.519562 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:28.519514 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/distinct-w5-967fac-kserve-69cbcccd8b-b85bn" podUID="8f6b9354-378f-4451-9ff7-be7f7fec12f2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Oct 30 16:41:28.701134 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:28.695336 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx" podUID="bd662f3a-659f-477c-b8d5-94c64288c0c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 16:41:28.701134 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:28.698337 5052 patch_prober.go:29] interesting pod/e2e-worker-w2-750dcb-openshift-default-777f567b9f-dnwt8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:41:28.701134 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:28.698396 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-dnwt8" podUID="c8947ffd-7539-43d8-86d5-fc17dd8c7d6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:28.709027 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:28.708528 5052 patch_prober.go:29] interesting pod/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cp9m8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 16:41:28.709294 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:28.709261 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cp9m8" podUID="ee44984b-c448-4580-aa5a-637c73d09e19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 16:41:28.841730 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:28.841268 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-4db2924d-6ff4bd685d-mkkgw" Oct 30 16:41:28.842006 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:28.841988 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-967fac/distinct-w5-967fac-kserve-69cbcccd8b-b85bn" Oct 30 16:41:28.843293 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:28.843274 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-263f00/distinct-2-w4-263f00-kserve-8464897cf5-q8ck8" Oct 30 16:41:28.873684 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:28.873647 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-750dcb/distinct-w2-750dcb-kserve-86bbdfcbb-bpn77" Oct 30 16:41:28.885876 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:28.885841 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-967fac/distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx" Oct 30 16:41:29.560016 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:29.559978 5052 patch_prober.go:29] interesting pod/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cp9m8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 16:41:29.560639 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:29.560606 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cp9m8" podUID="ee44984b-c448-4580-aa5a-637c73d09e19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 16:41:30.536552 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:30.536514 5052 patch_prober.go:29] interesting pod/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cp9m8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 16:41:30.536762 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:30.536572 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cp9m8" podUID="ee44984b-c448-4580-aa5a-637c73d09e19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 16:41:32.072606 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:32.072555 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cp9m8" Oct 30 16:41:32.073694 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:32.073665 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cp9m8" Oct 30 16:41:32.074494 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:32.074474 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cp9m8" Oct 30 16:41:32.909807 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:32.909772 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-4db2924d-7fdc89f884-6t9w8"] Oct 30 16:41:32.914527 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:32.914508 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4db2924d-7fdc89f884-6t9w8" Oct 30 16:41:32.936044 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:32.935998 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-4db2924d-7fdc89f884-6t9w8"] Oct 30 16:41:32.942263 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:32.942240 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-4db2924d-serving-cert\"" Oct 30 16:41:32.964178 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:32.964151 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69a28b3b-3990-46f7-b2da-42824fb9af4a-maas-api-tls\") pod \"maas-api-e2e-derive-4db2924d-7fdc89f884-6t9w8\" (UID: \"69a28b3b-3990-46f7-b2da-42824fb9af4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4db2924d-7fdc89f884-6t9w8" Oct 30 16:41:32.964329 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:32.964222 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69a28b3b-3990-46f7-b2da-42824fb9af4a-metrics-certs\") pod \"maas-api-e2e-derive-4db2924d-7fdc89f884-6t9w8\" (UID: \"69a28b3b-3990-46f7-b2da-42824fb9af4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4db2924d-7fdc89f884-6t9w8" Oct 30 16:41:32.964379 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:32.964354 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69a28b3b-3990-46f7-b2da-42824fb9af4a-openshift-service-ca\") pod \"maas-api-e2e-derive-4db2924d-7fdc89f884-6t9w8\" (UID: \"69a28b3b-3990-46f7-b2da-42824fb9af4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4db2924d-7fdc89f884-6t9w8" Oct 30 16:41:32.964433 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:32.964421 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nljpz\" (UniqueName: \"kubernetes.io/projected/69a28b3b-3990-46f7-b2da-42824fb9af4a-kube-api-access-nljpz\") pod \"maas-api-e2e-derive-4db2924d-7fdc89f884-6t9w8\" (UID: \"69a28b3b-3990-46f7-b2da-42824fb9af4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4db2924d-7fdc89f884-6t9w8" Oct 30 16:41:33.066934 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:33.066898 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69a28b3b-3990-46f7-b2da-42824fb9af4a-metrics-certs\") pod \"maas-api-e2e-derive-4db2924d-7fdc89f884-6t9w8\" (UID: \"69a28b3b-3990-46f7-b2da-42824fb9af4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4db2924d-7fdc89f884-6t9w8" Oct 30 16:41:33.067131 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:33.066973 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69a28b3b-3990-46f7-b2da-42824fb9af4a-openshift-service-ca\") pod \"maas-api-e2e-derive-4db2924d-7fdc89f884-6t9w8\" (UID: \"69a28b3b-3990-46f7-b2da-42824fb9af4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4db2924d-7fdc89f884-6t9w8" Oct 30 16:41:33.067131 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:33.067022 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nljpz\" (UniqueName: \"kubernetes.io/projected/69a28b3b-3990-46f7-b2da-42824fb9af4a-kube-api-access-nljpz\") pod \"maas-api-e2e-derive-4db2924d-7fdc89f884-6t9w8\" (UID: \"69a28b3b-3990-46f7-b2da-42824fb9af4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4db2924d-7fdc89f884-6t9w8" Oct 30 16:41:33.067324 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:33.067307 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69a28b3b-3990-46f7-b2da-42824fb9af4a-maas-api-tls\") pod \"maas-api-e2e-derive-4db2924d-7fdc89f884-6t9w8\" (UID: \"69a28b3b-3990-46f7-b2da-42824fb9af4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4db2924d-7fdc89f884-6t9w8" Oct 30 16:41:33.067479 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:41:33.067456 5052 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-4db2924d-serving-cert: secret "maas-api-e2e-derive-4db2924d-serving-cert" not found Oct 30 16:41:33.067548 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:41:33.067534 5052 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69a28b3b-3990-46f7-b2da-42824fb9af4a-maas-api-tls podName:69a28b3b-3990-46f7-b2da-42824fb9af4a nodeName:}" failed. No retries permitted until 2026-10-30 16:41:33.567512894 +0000 UTC m=+1249.094848114 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/69a28b3b-3990-46f7-b2da-42824fb9af4a-maas-api-tls") pod "maas-api-e2e-derive-4db2924d-7fdc89f884-6t9w8" (UID: "69a28b3b-3990-46f7-b2da-42824fb9af4a") : secret "maas-api-e2e-derive-4db2924d-serving-cert" not found Oct 30 16:41:33.067740 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:33.067718 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69a28b3b-3990-46f7-b2da-42824fb9af4a-openshift-service-ca\") pod \"maas-api-e2e-derive-4db2924d-7fdc89f884-6t9w8\" (UID: \"69a28b3b-3990-46f7-b2da-42824fb9af4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4db2924d-7fdc89f884-6t9w8" Oct 30 16:41:33.069357 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:33.069336 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69a28b3b-3990-46f7-b2da-42824fb9af4a-metrics-certs\") pod \"maas-api-e2e-derive-4db2924d-7fdc89f884-6t9w8\" (UID: \"69a28b3b-3990-46f7-b2da-42824fb9af4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4db2924d-7fdc89f884-6t9w8" Oct 30 16:41:33.092882 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:33.092858 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nljpz\" (UniqueName: \"kubernetes.io/projected/69a28b3b-3990-46f7-b2da-42824fb9af4a-kube-api-access-nljpz\") pod \"maas-api-e2e-derive-4db2924d-7fdc89f884-6t9w8\" (UID: \"69a28b3b-3990-46f7-b2da-42824fb9af4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4db2924d-7fdc89f884-6t9w8" Oct 30 16:41:33.591187 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:33.591148 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69a28b3b-3990-46f7-b2da-42824fb9af4a-maas-api-tls\") pod \"maas-api-e2e-derive-4db2924d-7fdc89f884-6t9w8\" (UID: \"69a28b3b-3990-46f7-b2da-42824fb9af4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4db2924d-7fdc89f884-6t9w8" Oct 30 16:41:33.594965 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:33.594934 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69a28b3b-3990-46f7-b2da-42824fb9af4a-maas-api-tls\") pod \"maas-api-e2e-derive-4db2924d-7fdc89f884-6t9w8\" (UID: \"69a28b3b-3990-46f7-b2da-42824fb9af4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4db2924d-7fdc89f884-6t9w8" Oct 30 16:41:33.831241 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:33.831207 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4db2924d-7fdc89f884-6t9w8" Oct 30 16:41:34.340570 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:34.340544 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-4db2924d-7fdc89f884-6t9w8"] Oct 30 16:41:34.862761 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:34.862720 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4db2924d-7fdc89f884-6t9w8" event={"ID":"69a28b3b-3990-46f7-b2da-42824fb9af4a","Type":"ContainerStarted","Data":"a878e945f6c5f98516e3bdbe9402375111e16491e7dbc4d661c3062ddc3b9067"} Oct 30 16:41:35.871645 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:35.871607 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4db2924d-7fdc89f884-6t9w8" event={"ID":"69a28b3b-3990-46f7-b2da-42824fb9af4a","Type":"ContainerStarted","Data":"44a5a72becfb9491d652be2e66d73855d2bc8a6f4732a9a3f25b2b4a97c23f52"} Oct 30 16:41:35.872136 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:35.871690 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4db2924d-7fdc89f884-6t9w8" Oct 30 16:41:35.917982 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:35.917933 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4db2924d-7fdc89f884-6t9w8" podStartSLOduration=3.530503441 podStartE2EDuration="3.917920829s" totalImagesPullingTime="387.417388ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:41:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:41:35.914632108 +0000 UTC m=+1251.441967352" watchObservedRunningTime="2026-10-30 16:41:35.917920829 +0000 UTC m=+1251.445256048" Oct 30 16:41:36.422776 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:36.422741 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-d693d0/distinct-2-w6-d693d0-kserve-6498fdc474-dpbjj" Oct 30 16:41:36.434491 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:36.434465 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d693d0/distinct-2-w6-d693d0-kserve-6498fdc474-dpbjj" Oct 30 16:41:38.430433 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:38.430398 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-a475e0dd-577fdb5999-dwtjd"] Oct 30 16:41:38.436262 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:38.436233 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a475e0dd-577fdb5999-dwtjd" Oct 30 16:41:38.535349 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:38.535314 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-a475e0dd-577fdb5999-dwtjd"] Oct 30 16:41:38.541647 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:38.537540 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-a475e0dd\"" Oct 30 16:41:38.541647 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:38.538258 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-a475e0dd-dockercfg-rh2mv\"" Oct 30 16:41:38.568286 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:38.568242 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fv97\" (UniqueName: \"kubernetes.io/projected/4ee414b4-410f-4506-a81d-26c21b331674-kube-api-access-4fv97\") pod \"payload-pre-processing-e2e-migrate-a475e0dd-577fdb5999-dwtjd\" (UID: \"4ee414b4-410f-4506-a81d-26c21b331674\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-a475e0dd-577fdb5999-dwtjd" Oct 30 16:41:38.568792 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:38.568766 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ee414b4-410f-4506-a81d-26c21b331674-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-a475e0dd-577fdb5999-dwtjd\" (UID: \"4ee414b4-410f-4506-a81d-26c21b331674\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-a475e0dd-577fdb5999-dwtjd" Oct 30 16:41:38.676420 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:38.676376 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ee414b4-410f-4506-a81d-26c21b331674-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-a475e0dd-577fdb5999-dwtjd\" (UID: \"4ee414b4-410f-4506-a81d-26c21b331674\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-a475e0dd-577fdb5999-dwtjd" Oct 30 16:41:38.676854 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:38.676825 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fv97\" (UniqueName: \"kubernetes.io/projected/4ee414b4-410f-4506-a81d-26c21b331674-kube-api-access-4fv97\") pod \"payload-pre-processing-e2e-migrate-a475e0dd-577fdb5999-dwtjd\" (UID: \"4ee414b4-410f-4506-a81d-26c21b331674\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-a475e0dd-577fdb5999-dwtjd" Oct 30 16:41:38.678551 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:38.678528 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ee414b4-410f-4506-a81d-26c21b331674-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-a475e0dd-577fdb5999-dwtjd\" (UID: \"4ee414b4-410f-4506-a81d-26c21b331674\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-a475e0dd-577fdb5999-dwtjd" Oct 30 16:41:38.734195 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:38.718528 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fv97\" (UniqueName: \"kubernetes.io/projected/4ee414b4-410f-4506-a81d-26c21b331674-kube-api-access-4fv97\") pod \"payload-pre-processing-e2e-migrate-a475e0dd-577fdb5999-dwtjd\" (UID: \"4ee414b4-410f-4506-a81d-26c21b331674\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-a475e0dd-577fdb5999-dwtjd" Oct 30 16:41:38.754207 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:38.754159 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a475e0dd-577fdb5999-dwtjd" Oct 30 16:41:39.171758 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:39.171731 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-a475e0dd-577fdb5999-dwtjd"] Oct 30 16:41:39.173847 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:41:39.173813 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ee414b4_410f_4506_a81d_26c21b331674.slice/crio-553131b7c42e93fd33047a3245ab857e6fc77f2831dcb282e4b93f51530bf3e4 WatchSource:0}: Error finding container 553131b7c42e93fd33047a3245ab857e6fc77f2831dcb282e4b93f51530bf3e4: Status 404 returned error can't find the container with id 553131b7c42e93fd33047a3245ab857e6fc77f2831dcb282e4b93f51530bf3e4 Oct 30 16:41:39.912960 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:39.912919 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a475e0dd-577fdb5999-dwtjd" event={"ID":"4ee414b4-410f-4506-a81d-26c21b331674","Type":"ContainerStarted","Data":"b727136dd7927467f7ee6f3624c501770139b6689e65d8ddd11d6c962a30b87f"} Oct 30 16:41:39.913566 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:39.913540 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a475e0dd-577fdb5999-dwtjd" event={"ID":"4ee414b4-410f-4506-a81d-26c21b331674","Type":"ContainerStarted","Data":"553131b7c42e93fd33047a3245ab857e6fc77f2831dcb282e4b93f51530bf3e4"} Oct 30 16:41:39.921873 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:39.921837 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a475e0dd-577fdb5999-dwtjd" Oct 30 16:41:39.967390 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:39.967323 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a475e0dd-577fdb5999-dwtjd" podStartSLOduration=1.967304632 podStartE2EDuration="1.967304632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:41:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:41:39.9649249 +0000 UTC m=+1255.492260122" watchObservedRunningTime="2026-10-30 16:41:39.967304632 +0000 UTC m=+1255.494639853" Oct 30 16:41:41.884729 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:41.884699 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4db2924d-7fdc89f884-6t9w8" Oct 30 16:41:48.369152 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:48.369106 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cdwcj"] Oct 30 16:41:48.849303 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:41:48.849258 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 16:41:49.323560 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:49.323531 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-7xqp2"] Oct 30 16:41:49.849609 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:49.843781 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-7xqp2"] Oct 30 16:41:49.849609 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:49.843946 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-7xqp2" Oct 30 16:41:50.038616 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:50.027800 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f43327c0-fce9-4bdf-ae16-2111377c71ad-istio-token\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-7xqp2\" (UID: \"f43327c0-fce9-4bdf-ae16-2111377c71ad\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-7xqp2" Oct 30 16:41:50.041118 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:50.041081 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f43327c0-fce9-4bdf-ae16-2111377c71ad-credential-socket\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-7xqp2\" (UID: \"f43327c0-fce9-4bdf-ae16-2111377c71ad\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-7xqp2" Oct 30 16:41:50.041426 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:50.041410 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f43327c0-fce9-4bdf-ae16-2111377c71ad-workload-certs\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-7xqp2\" (UID: \"f43327c0-fce9-4bdf-ae16-2111377c71ad\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-7xqp2" Oct 30 16:41:50.082074 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:50.082037 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f43327c0-fce9-4bdf-ae16-2111377c71ad-istiod-ca-cert\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-7xqp2\" (UID: \"f43327c0-fce9-4bdf-ae16-2111377c71ad\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-7xqp2" Oct 30 16:41:50.082394 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:50.082369 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2ggn\" (UniqueName: \"kubernetes.io/projected/f43327c0-fce9-4bdf-ae16-2111377c71ad-kube-api-access-d2ggn\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-7xqp2\" (UID: \"f43327c0-fce9-4bdf-ae16-2111377c71ad\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-7xqp2" Oct 30 16:41:50.082548 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:50.082530 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f43327c0-fce9-4bdf-ae16-2111377c71ad-istio-envoy\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-7xqp2\" (UID: \"f43327c0-fce9-4bdf-ae16-2111377c71ad\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-7xqp2" Oct 30 16:41:50.088307 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:50.088284 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f43327c0-fce9-4bdf-ae16-2111377c71ad-istio-data\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-7xqp2\" (UID: \"f43327c0-fce9-4bdf-ae16-2111377c71ad\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-7xqp2" Oct 30 16:41:50.089159 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:50.089139 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f43327c0-fce9-4bdf-ae16-2111377c71ad-workload-socket\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-7xqp2\" (UID: \"f43327c0-fce9-4bdf-ae16-2111377c71ad\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-7xqp2" Oct 30 16:41:50.089387 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:50.089368 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f43327c0-fce9-4bdf-ae16-2111377c71ad-istio-podinfo\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-7xqp2\" (UID: \"f43327c0-fce9-4bdf-ae16-2111377c71ad\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-7xqp2" Oct 30 16:41:50.217607 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:50.200446 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f43327c0-fce9-4bdf-ae16-2111377c71ad-workload-socket\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-7xqp2\" (UID: \"f43327c0-fce9-4bdf-ae16-2111377c71ad\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-7xqp2" Oct 30 16:41:50.217948 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:50.217909 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f43327c0-fce9-4bdf-ae16-2111377c71ad-istio-podinfo\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-7xqp2\" (UID: \"f43327c0-fce9-4bdf-ae16-2111377c71ad\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-7xqp2" Oct 30 16:41:50.218361 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:50.218337 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f43327c0-fce9-4bdf-ae16-2111377c71ad-istio-token\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-7xqp2\" (UID: \"f43327c0-fce9-4bdf-ae16-2111377c71ad\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-7xqp2" Oct 30 16:41:50.218561 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:50.218543 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f43327c0-fce9-4bdf-ae16-2111377c71ad-credential-socket\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-7xqp2\" (UID: \"f43327c0-fce9-4bdf-ae16-2111377c71ad\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-7xqp2" Oct 30 16:41:50.218811 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:50.218792 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f43327c0-fce9-4bdf-ae16-2111377c71ad-workload-certs\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-7xqp2\" (UID: \"f43327c0-fce9-4bdf-ae16-2111377c71ad\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-7xqp2" Oct 30 16:41:50.219028 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:50.219009 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f43327c0-fce9-4bdf-ae16-2111377c71ad-istiod-ca-cert\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-7xqp2\" (UID: \"f43327c0-fce9-4bdf-ae16-2111377c71ad\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-7xqp2" Oct 30 16:41:50.219199 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:50.219181 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2ggn\" (UniqueName: \"kubernetes.io/projected/f43327c0-fce9-4bdf-ae16-2111377c71ad-kube-api-access-d2ggn\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-7xqp2\" (UID: \"f43327c0-fce9-4bdf-ae16-2111377c71ad\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-7xqp2" Oct 30 16:41:50.219350 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:50.219334 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f43327c0-fce9-4bdf-ae16-2111377c71ad-istio-envoy\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-7xqp2\" (UID: \"f43327c0-fce9-4bdf-ae16-2111377c71ad\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-7xqp2" Oct 30 16:41:50.219500 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:50.219484 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f43327c0-fce9-4bdf-ae16-2111377c71ad-istio-data\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-7xqp2\" (UID: \"f43327c0-fce9-4bdf-ae16-2111377c71ad\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-7xqp2" Oct 30 16:41:50.220246 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:50.220221 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f43327c0-fce9-4bdf-ae16-2111377c71ad-istio-data\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-7xqp2\" (UID: \"f43327c0-fce9-4bdf-ae16-2111377c71ad\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-7xqp2" Oct 30 16:41:50.220382 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:50.209078 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f43327c0-fce9-4bdf-ae16-2111377c71ad-workload-socket\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-7xqp2\" (UID: \"f43327c0-fce9-4bdf-ae16-2111377c71ad\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-7xqp2" Oct 30 16:41:50.229034 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:50.229002 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f43327c0-fce9-4bdf-ae16-2111377c71ad-workload-certs\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-7xqp2\" (UID: \"f43327c0-fce9-4bdf-ae16-2111377c71ad\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-7xqp2" Oct 30 16:41:50.235666 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:50.234252 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f43327c0-fce9-4bdf-ae16-2111377c71ad-istiod-ca-cert\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-7xqp2\" (UID: \"f43327c0-fce9-4bdf-ae16-2111377c71ad\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-7xqp2" Oct 30 16:41:50.285619 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:50.236536 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f43327c0-fce9-4bdf-ae16-2111377c71ad-credential-socket\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-7xqp2\" (UID: \"f43327c0-fce9-4bdf-ae16-2111377c71ad\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-7xqp2" Oct 30 16:41:50.350488 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:50.350449 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2ggn\" (UniqueName: \"kubernetes.io/projected/f43327c0-fce9-4bdf-ae16-2111377c71ad-kube-api-access-d2ggn\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-7xqp2\" (UID: \"f43327c0-fce9-4bdf-ae16-2111377c71ad\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-7xqp2" Oct 30 16:41:50.352287 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:50.352257 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f43327c0-fce9-4bdf-ae16-2111377c71ad-istio-token\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-7xqp2\" (UID: \"f43327c0-fce9-4bdf-ae16-2111377c71ad\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-7xqp2" Oct 30 16:41:50.413712 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:50.408974 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f43327c0-fce9-4bdf-ae16-2111377c71ad-istio-envoy\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-7xqp2\" (UID: \"f43327c0-fce9-4bdf-ae16-2111377c71ad\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-7xqp2" Oct 30 16:41:50.413712 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:50.411669 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f43327c0-fce9-4bdf-ae16-2111377c71ad-istio-podinfo\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-7xqp2\" (UID: \"f43327c0-fce9-4bdf-ae16-2111377c71ad\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-7xqp2" Oct 30 16:41:50.513663 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:50.510732 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/distinct-w5-967fac-kserve-69cbcccd8b-b85bn" podUID="8f6b9354-378f-4451-9ff7-be7f7fec12f2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:50.545607 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:50.542857 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-7xqp2" Oct 30 16:41:50.795877 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:50.795783 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx" podUID="bd662f3a-659f-477c-b8d5-94c64288c0c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:51.439056 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:51.435667 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d693d0/distinct-2-w6-d693d0-kserve-6498fdc474-dpbjj" podUID="4b98b8ff-72cd-4ddb-a4b8-67078162e611" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Oct 30 16:41:51.526607 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:51.510947 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/distinct-w5-967fac-kserve-69cbcccd8b-b85bn" podUID="8f6b9354-378f-4451-9ff7-be7f7fec12f2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:51.526607 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:51.511516 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-967fac/distinct-w5-967fac-kserve-69cbcccd8b-b85bn" Oct 30 16:41:52.008872 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:52.008777 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/distinct-2-w4-263f00-kserve-8464897cf5-q8ck8" podUID="15ed43ef-5cdd-489a-a70e-063239822ff3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:52.038969 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:52.032948 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx" podUID="bd662f3a-659f-477c-b8d5-94c64288c0c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 16:41:52.038969 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:52.034769 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-967fac/distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx" Oct 30 16:41:52.143609 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:52.142703 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a475e0dd-577fdb5999-dwtjd" Oct 30 16:41:52.350662 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:52.326653 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-750dcb/distinct-w2-750dcb-kserve-86bbdfcbb-bpn77" podUID="c45190af-d795-4e13-8f2f-670b2c597658" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:52.479274 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:52.454215 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d693d0/distinct-2-w6-d693d0-kserve-6498fdc474-dpbjj" podUID="4b98b8ff-72cd-4ddb-a4b8-67078162e611" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:52.479274 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:52.455289 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d693d0/distinct-2-w6-d693d0-kserve-6498fdc474-dpbjj" Oct 30 16:41:52.484701 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:41:52.484556 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf43327c0_fce9_4bdf_ae16_2111377c71ad.slice/crio-72375590bd1ed1f62b754af8e457832e830d7d5e5dd823caf636b52e4b9c4f42 WatchSource:0}: Error finding container 72375590bd1ed1f62b754af8e457832e830d7d5e5dd823caf636b52e4b9c4f42: Status 404 returned error can't find the container with id 72375590bd1ed1f62b754af8e457832e830d7d5e5dd823caf636b52e4b9c4f42 Oct 30 16:41:52.522949 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:52.520116 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/distinct-w5-967fac-kserve-69cbcccd8b-b85bn" podUID="8f6b9354-378f-4451-9ff7-be7f7fec12f2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Oct 30 16:41:52.693007 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:52.686250 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:41:52.693007 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:52.686358 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:41:52.693007 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:52.686466 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:41:53.034032 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:53.033839 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx" podUID="bd662f3a-659f-477c-b8d5-94c64288c0c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:53.078239 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:53.074039 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-7xqp2"] Oct 30 16:41:53.078239 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:53.074177 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/distinct-2-w4-263f00-kserve-8464897cf5-q8ck8" podUID="15ed43ef-5cdd-489a-a70e-063239822ff3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:53.078239 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:53.075773 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-263f00/distinct-2-w4-263f00-kserve-8464897cf5-q8ck8" Oct 30 16:41:53.106067 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:53.080642 5052 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-967fac/distinct-w5-967fac-kserve-69cbcccd8b-b85bn" podUID="8f6b9354-378f-4451-9ff7-be7f7fec12f2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": context deadline exceeded" Oct 30 16:41:53.114538 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:53.098332 5052 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-750dcb/distinct-w2-750dcb-kserve-86bbdfcbb-bpn77" podUID="c45190af-d795-4e13-8f2f-670b2c597658" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": context deadline exceeded" Oct 30 16:41:53.335823 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:53.335707 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-750dcb/distinct-w2-750dcb-kserve-86bbdfcbb-bpn77" podUID="c45190af-d795-4e13-8f2f-670b2c597658" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:53.381628 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:53.369781 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-750dcb/distinct-w2-750dcb-kserve-86bbdfcbb-bpn77" Oct 30 16:41:53.485094 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:53.485054 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d693d0/distinct-2-w6-d693d0-kserve-6498fdc474-dpbjj" podUID="4b98b8ff-72cd-4ddb-a4b8-67078162e611" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:53.522700 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:53.522653 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/distinct-w5-967fac-kserve-69cbcccd8b-b85bn" podUID="8f6b9354-378f-4451-9ff7-be7f7fec12f2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Oct 30 16:41:53.576293 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:53.576229 5052 patch_prober.go:29] interesting pod/e2e-worker-w3-0b2348-openshift-default-85679bc47c-46brl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:41:53.576616 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:53.576561 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-46brl" podUID="4b8235a9-5724-4354-ad7b-eaef26b3966f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:54.059975 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:54.059933 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx" podUID="bd662f3a-659f-477c-b8d5-94c64288c0c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:54.173233 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:54.081796 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/distinct-2-w4-263f00-kserve-8464897cf5-q8ck8" podUID="15ed43ef-5cdd-489a-a70e-063239822ff3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 30 16:41:54.305201 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:54.305155 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-7xqp2" event={"ID":"f43327c0-fce9-4bdf-ae16-2111377c71ad","Type":"ContainerStarted","Data":"72375590bd1ed1f62b754af8e457832e830d7d5e5dd823caf636b52e4b9c4f42"} Oct 30 16:41:54.377618 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:54.370332 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-750dcb/distinct-w2-750dcb-kserve-86bbdfcbb-bpn77" podUID="c45190af-d795-4e13-8f2f-670b2c597658" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:54.459734 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:54.459682 5052 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-967fac/distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx" podUID="bd662f3a-659f-477c-b8d5-94c64288c0c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:54.505873 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:54.495710 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d693d0/distinct-2-w6-d693d0-kserve-6498fdc474-dpbjj" podUID="4b98b8ff-72cd-4ddb-a4b8-67078162e611" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Oct 30 16:41:54.523905 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:54.523860 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/distinct-w5-967fac-kserve-69cbcccd8b-b85bn" podUID="8f6b9354-378f-4451-9ff7-be7f7fec12f2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Oct 30 16:41:55.187703 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:55.184763 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/distinct-2-w4-263f00-kserve-8464897cf5-q8ck8" podUID="15ed43ef-5cdd-489a-a70e-063239822ff3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:55.187703 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:55.185107 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx" podUID="bd662f3a-659f-477c-b8d5-94c64288c0c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:55.397599 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:55.374358 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-750dcb/distinct-w2-750dcb-kserve-86bbdfcbb-bpn77" podUID="c45190af-d795-4e13-8f2f-670b2c597658" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:55.576675 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:55.546927 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d693d0/distinct-2-w6-d693d0-kserve-6498fdc474-dpbjj" podUID="4b98b8ff-72cd-4ddb-a4b8-67078162e611" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Oct 30 16:41:55.576675 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:55.550211 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/distinct-w5-967fac-kserve-69cbcccd8b-b85bn" podUID="8f6b9354-378f-4451-9ff7-be7f7fec12f2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Oct 30 16:41:55.740171 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:55.740128 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-7xqp2" event={"ID":"f43327c0-fce9-4bdf-ae16-2111377c71ad","Type":"ContainerStarted","Data":"fb4ec2bde8c8f4328e3878bdcf85cb0e0bb3f2d0735307e9e6413b622b30c2ea"} Oct 30 16:41:55.972814 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:55.972772 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d693d0/distinct-2-w6-d693d0-kserve-6498fdc474-dpbjj" Oct 30 16:41:56.045674 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:56.032551 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-7xqp2" podStartSLOduration=7.032531932 podStartE2EDuration="7.032531932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:41:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:41:55.83204432 +0000 UTC m=+1271.359379542" watchObservedRunningTime="2026-10-30 16:41:56.032531932 +0000 UTC m=+1271.559867152" Oct 30 16:41:56.288682 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:56.240156 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/distinct-2-w4-263f00-kserve-8464897cf5-q8ck8" podUID="15ed43ef-5cdd-489a-a70e-063239822ff3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 30 16:41:56.288682 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:56.240508 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx" podUID="bd662f3a-659f-477c-b8d5-94c64288c0c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 16:41:56.375941 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:56.375894 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-750dcb/distinct-w2-750dcb-kserve-86bbdfcbb-bpn77" podUID="c45190af-d795-4e13-8f2f-670b2c597658" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:56.547613 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:56.547451 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-7xqp2" Oct 30 16:41:56.552607 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:56.551683 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/distinct-w5-967fac-kserve-69cbcccd8b-b85bn" podUID="8f6b9354-378f-4451-9ff7-be7f7fec12f2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Oct 30 16:41:56.559187 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:56.559145 5052 patch_prober.go:29] interesting pod/e2e-worker-w2-750dcb-openshift-default-777f567b9f-7xqp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 16:41:56.559456 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:56.559430 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-7xqp2" podUID="f43327c0-fce9-4bdf-ae16-2111377c71ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 16:41:56.847465 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:56.847380 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-967fac/distinct-w5-967fac-kserve-69cbcccd8b-b85bn" Oct 30 16:41:57.240763 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:57.240720 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx" podUID="bd662f3a-659f-477c-b8d5-94c64288c0c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:57.244617 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:57.241406 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/distinct-2-w4-263f00-kserve-8464897cf5-q8ck8" podUID="15ed43ef-5cdd-489a-a70e-063239822ff3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 30 16:41:57.506673 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:57.474927 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-750dcb/distinct-w2-750dcb-kserve-86bbdfcbb-bpn77" podUID="c45190af-d795-4e13-8f2f-670b2c597658" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:57.624934 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:57.624897 5052 patch_prober.go:29] interesting pod/e2e-worker-w2-750dcb-openshift-default-777f567b9f-7xqp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 16:41:57.626475 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:57.626436 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-7xqp2" podUID="f43327c0-fce9-4bdf-ae16-2111377c71ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 16:41:57.826287 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:57.825314 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-263f00/distinct-2-w4-263f00-kserve-8464897cf5-q8ck8" Oct 30 16:41:58.036962 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:58.036932 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-750dcb/distinct-w2-750dcb-kserve-86bbdfcbb-bpn77" Oct 30 16:41:58.249952 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:58.249905 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx" podUID="bd662f3a-659f-477c-b8d5-94c64288c0c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:58.327242 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:58.327210 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-967fac/distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx" Oct 30 16:41:58.558246 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:58.558155 5052 patch_prober.go:29] interesting pod/e2e-worker-w2-750dcb-openshift-default-777f567b9f-7xqp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 16:41:58.558499 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:58.558473 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-7xqp2" podUID="f43327c0-fce9-4bdf-ae16-2111377c71ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 16:41:59.627043 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:59.626990 5052 patch_prober.go:29] interesting pod/e2e-worker-w2-750dcb-openshift-default-777f567b9f-7xqp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 16:41:59.627631 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:41:59.627572 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-7xqp2" podUID="f43327c0-fce9-4bdf-ae16-2111377c71ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 16:42:00.551338 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:00.551302 5052 patch_prober.go:29] interesting pod/e2e-worker-w2-750dcb-openshift-default-777f567b9f-7xqp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 16:42:00.551704 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:00.551669 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-7xqp2" podUID="f43327c0-fce9-4bdf-ae16-2111377c71ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 16:42:01.550466 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:01.550008 5052 patch_prober.go:29] interesting pod/e2e-worker-w2-750dcb-openshift-default-777f567b9f-7xqp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 16:42:01.550466 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:01.550074 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-7xqp2" podUID="f43327c0-fce9-4bdf-ae16-2111377c71ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 16:42:02.550477 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:02.549998 5052 patch_prober.go:29] interesting pod/e2e-worker-w2-750dcb-openshift-default-777f567b9f-7xqp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 16:42:02.550477 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:02.550066 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-7xqp2" podUID="f43327c0-fce9-4bdf-ae16-2111377c71ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 16:42:03.544419 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:03.543970 5052 patch_prober.go:29] interesting pod/e2e-worker-w2-750dcb-openshift-default-777f567b9f-7xqp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 16:42:03.544419 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:03.544049 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-7xqp2" podUID="f43327c0-fce9-4bdf-ae16-2111377c71ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 16:42:04.658204 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:04.658137 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-7xqp2" Oct 30 16:42:04.659516 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:04.659184 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-7xqp2" Oct 30 16:42:04.660741 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:04.660689 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-7xqp2" Oct 30 16:42:04.690097 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:04.690066 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-92x47"] Oct 30 16:42:04.765881 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:04.765815 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-92x47"] Oct 30 16:42:04.767106 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:04.767081 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-92x47" Oct 30 16:42:04.786382 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:04.786341 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39ebd9d1-3580-45a0-84ca-de3126fe5224-credential-socket\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-92x47\" (UID: \"39ebd9d1-3580-45a0-84ca-de3126fe5224\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-92x47" Oct 30 16:42:04.786619 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:04.786434 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39ebd9d1-3580-45a0-84ca-de3126fe5224-istio-data\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-92x47\" (UID: \"39ebd9d1-3580-45a0-84ca-de3126fe5224\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-92x47" Oct 30 16:42:04.786619 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:04.786463 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39ebd9d1-3580-45a0-84ca-de3126fe5224-istio-token\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-92x47\" (UID: \"39ebd9d1-3580-45a0-84ca-de3126fe5224\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-92x47" Oct 30 16:42:04.786619 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:04.786493 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39ebd9d1-3580-45a0-84ca-de3126fe5224-istio-podinfo\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-92x47\" (UID: \"39ebd9d1-3580-45a0-84ca-de3126fe5224\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-92x47" Oct 30 16:42:04.786619 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:04.786562 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7d5mh\" (UniqueName: \"kubernetes.io/projected/39ebd9d1-3580-45a0-84ca-de3126fe5224-kube-api-access-7d5mh\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-92x47\" (UID: \"39ebd9d1-3580-45a0-84ca-de3126fe5224\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-92x47" Oct 30 16:42:04.786805 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:04.786685 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39ebd9d1-3580-45a0-84ca-de3126fe5224-workload-certs\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-92x47\" (UID: \"39ebd9d1-3580-45a0-84ca-de3126fe5224\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-92x47" Oct 30 16:42:04.786805 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:04.786733 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39ebd9d1-3580-45a0-84ca-de3126fe5224-istio-envoy\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-92x47\" (UID: \"39ebd9d1-3580-45a0-84ca-de3126fe5224\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-92x47" Oct 30 16:42:04.786805 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:04.786770 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39ebd9d1-3580-45a0-84ca-de3126fe5224-istiod-ca-cert\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-92x47\" (UID: \"39ebd9d1-3580-45a0-84ca-de3126fe5224\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-92x47" Oct 30 16:42:04.786914 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:04.786875 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39ebd9d1-3580-45a0-84ca-de3126fe5224-workload-socket\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-92x47\" (UID: \"39ebd9d1-3580-45a0-84ca-de3126fe5224\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-92x47" Oct 30 16:42:04.900377 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:04.899505 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39ebd9d1-3580-45a0-84ca-de3126fe5224-credential-socket\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-92x47\" (UID: \"39ebd9d1-3580-45a0-84ca-de3126fe5224\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-92x47" Oct 30 16:42:04.900377 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:04.899633 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39ebd9d1-3580-45a0-84ca-de3126fe5224-istio-data\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-92x47\" (UID: \"39ebd9d1-3580-45a0-84ca-de3126fe5224\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-92x47" Oct 30 16:42:04.900377 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:04.899661 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39ebd9d1-3580-45a0-84ca-de3126fe5224-istio-token\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-92x47\" (UID: \"39ebd9d1-3580-45a0-84ca-de3126fe5224\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-92x47" Oct 30 16:42:04.900377 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:04.899694 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39ebd9d1-3580-45a0-84ca-de3126fe5224-istio-podinfo\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-92x47\" (UID: \"39ebd9d1-3580-45a0-84ca-de3126fe5224\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-92x47" Oct 30 16:42:04.900377 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:04.899771 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7d5mh\" (UniqueName: \"kubernetes.io/projected/39ebd9d1-3580-45a0-84ca-de3126fe5224-kube-api-access-7d5mh\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-92x47\" (UID: \"39ebd9d1-3580-45a0-84ca-de3126fe5224\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-92x47" Oct 30 16:42:04.900878 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:04.900467 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39ebd9d1-3580-45a0-84ca-de3126fe5224-credential-socket\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-92x47\" (UID: \"39ebd9d1-3580-45a0-84ca-de3126fe5224\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-92x47" Oct 30 16:42:04.901639 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:04.901085 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39ebd9d1-3580-45a0-84ca-de3126fe5224-istio-data\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-92x47\" (UID: \"39ebd9d1-3580-45a0-84ca-de3126fe5224\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-92x47" Oct 30 16:42:04.901639 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:04.901216 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39ebd9d1-3580-45a0-84ca-de3126fe5224-workload-certs\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-92x47\" (UID: \"39ebd9d1-3580-45a0-84ca-de3126fe5224\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-92x47" Oct 30 16:42:04.901639 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:04.901271 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39ebd9d1-3580-45a0-84ca-de3126fe5224-istio-envoy\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-92x47\" (UID: \"39ebd9d1-3580-45a0-84ca-de3126fe5224\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-92x47" Oct 30 16:42:04.901639 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:04.901306 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39ebd9d1-3580-45a0-84ca-de3126fe5224-istiod-ca-cert\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-92x47\" (UID: \"39ebd9d1-3580-45a0-84ca-de3126fe5224\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-92x47" Oct 30 16:42:04.901639 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:04.901388 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39ebd9d1-3580-45a0-84ca-de3126fe5224-workload-socket\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-92x47\" (UID: \"39ebd9d1-3580-45a0-84ca-de3126fe5224\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-92x47" Oct 30 16:42:04.901977 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:04.901751 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39ebd9d1-3580-45a0-84ca-de3126fe5224-workload-socket\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-92x47\" (UID: \"39ebd9d1-3580-45a0-84ca-de3126fe5224\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-92x47" Oct 30 16:42:04.902255 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:04.902206 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39ebd9d1-3580-45a0-84ca-de3126fe5224-workload-certs\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-92x47\" (UID: \"39ebd9d1-3580-45a0-84ca-de3126fe5224\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-92x47" Oct 30 16:42:04.905878 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:04.905772 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39ebd9d1-3580-45a0-84ca-de3126fe5224-istiod-ca-cert\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-92x47\" (UID: \"39ebd9d1-3580-45a0-84ca-de3126fe5224\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-92x47" Oct 30 16:42:04.909374 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:04.909321 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39ebd9d1-3580-45a0-84ca-de3126fe5224-istio-podinfo\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-92x47\" (UID: \"39ebd9d1-3580-45a0-84ca-de3126fe5224\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-92x47" Oct 30 16:42:04.910676 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:04.910652 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39ebd9d1-3580-45a0-84ca-de3126fe5224-istio-envoy\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-92x47\" (UID: \"39ebd9d1-3580-45a0-84ca-de3126fe5224\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-92x47" Oct 30 16:42:04.924546 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:04.924524 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39ebd9d1-3580-45a0-84ca-de3126fe5224-istio-token\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-92x47\" (UID: \"39ebd9d1-3580-45a0-84ca-de3126fe5224\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-92x47" Oct 30 16:42:04.925157 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:04.925136 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7d5mh\" (UniqueName: \"kubernetes.io/projected/39ebd9d1-3580-45a0-84ca-de3126fe5224-kube-api-access-7d5mh\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-92x47\" (UID: \"39ebd9d1-3580-45a0-84ca-de3126fe5224\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-92x47" Oct 30 16:42:05.136857 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:05.136812 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-92x47" Oct 30 16:42:05.567396 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:05.567185 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-92x47"] Oct 30 16:42:05.572952 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:05.572917 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:42:05.573074 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:05.573001 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:42:05.573074 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:05.573050 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:42:06.126853 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:06.126799 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-92x47" event={"ID":"39ebd9d1-3580-45a0-84ca-de3126fe5224","Type":"ContainerStarted","Data":"fa96c67b575da8b32a91902eca0c2fdea9954fec81050b3dc444dec6c5bbc283"} Oct 30 16:42:06.127474 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:06.126874 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-92x47" event={"ID":"39ebd9d1-3580-45a0-84ca-de3126fe5224","Type":"ContainerStarted","Data":"c20cc8bc540fa007e4580e5c42cf64b41a2989a081fa07f301e209ac0d53012e"} Oct 30 16:42:06.138151 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:06.138109 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-92x47" Oct 30 16:42:06.158690 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:06.140410 5052 patch_prober.go:29] interesting pod/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-92x47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 16:42:06.158690 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:06.140464 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-92x47" podUID="39ebd9d1-3580-45a0-84ca-de3126fe5224" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 16:42:06.195016 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:06.194897 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-92x47" podStartSLOduration=2.194877462 podStartE2EDuration="2.194877462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:42:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:42:06.185713764 +0000 UTC m=+1281.713048986" watchObservedRunningTime="2026-10-30 16:42:06.194877462 +0000 UTC m=+1281.722212682" Oct 30 16:42:06.408644 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:06.401709 5052 patch_prober.go:29] interesting pod/e2e-ait-f654570d-openshift-default-76cdfc5fcf-kwdnz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:42:06.408644 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:06.401763 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-f654570d-openshift-default-76cdfc5fcf-kwdnz" podUID="f6092ad4-fa04-4813-aebb-655353769fc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:42:07.196629 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:07.196488 5052 patch_prober.go:29] interesting pod/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-92x47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 16:42:07.196629 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:07.196549 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-92x47" podUID="39ebd9d1-3580-45a0-84ca-de3126fe5224" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 16:42:07.296935 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:07.296902 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f654570d-56d5957948-lh66x"] Oct 30 16:42:07.365265 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:07.365233 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f654570d-7b49ffb9f-2s4q4"] Oct 30 16:42:08.142636 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:08.140997 5052 patch_prober.go:29] interesting pod/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-92x47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 16:42:08.142636 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:08.141079 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-92x47" podUID="39ebd9d1-3580-45a0-84ca-de3126fe5224" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 16:42:08.269908 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:08.269866 5052 generic.go:356] "Generic (PLEG): container finished" podID="62e3ed6d-3cc2-4e79-9738-43a802b6d58b" containerID="38c010033fc2bc3d8d86c019978d27604a98087fef69dd5b1df3144af7f05b6d" exitCode=0 Oct 30 16:42:08.298936 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:08.274439 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-4db2924d-openshift-default-bc6f66585-r4b8b" event={"ID":"62e3ed6d-3cc2-4e79-9738-43a802b6d58b","Type":"ContainerDied","Data":"38c010033fc2bc3d8d86c019978d27604a98087fef69dd5b1df3144af7f05b6d"} Oct 30 16:42:08.299865 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:08.299839 5052 scope.go:122] "RemoveContainer" containerID="38c010033fc2bc3d8d86c019978d27604a98087fef69dd5b1df3144af7f05b6d" Oct 30 16:42:08.334678 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:08.331129 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:42:08.348142 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:08.348092 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:42:08.348441 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:08.348413 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:42:08.363374 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:08.363341 5052 generic.go:356] "Generic (PLEG): container finished" podID="f6092ad4-fa04-4813-aebb-655353769fc1" containerID="6939f75159ba401a04fb22fb3ae142dfd65077ddfe28f96281a7b01d0de70471" exitCode=0 Oct 30 16:42:08.365197 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:08.363600 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f654570d-openshift-default-76cdfc5fcf-kwdnz" event={"ID":"f6092ad4-fa04-4813-aebb-655353769fc1","Type":"ContainerDied","Data":"6939f75159ba401a04fb22fb3ae142dfd65077ddfe28f96281a7b01d0de70471"} Oct 30 16:42:08.366978 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:08.366958 5052 scope.go:122] "RemoveContainer" containerID="6939f75159ba401a04fb22fb3ae142dfd65077ddfe28f96281a7b01d0de70471" Oct 30 16:42:08.396457 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:08.396343 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:42:08.431912 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:08.420202 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:42:08.431912 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:08.420308 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:42:09.142855 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:09.141057 5052 patch_prober.go:29] interesting pod/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-92x47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 16:42:09.142855 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:09.141119 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-92x47" podUID="39ebd9d1-3580-45a0-84ca-de3126fe5224" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 16:42:09.396692 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:09.393944 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f654570d-openshift-default-76cdfc5fcf-kwdnz" event={"ID":"f6092ad4-fa04-4813-aebb-655353769fc1","Type":"ContainerStarted","Data":"ccab10db017561022715c46306d0825207a302d6535fc999d7b0cf4913504374"} Oct 30 16:42:09.417666 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:09.399199 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-4db2924d-openshift-default-bc6f66585-r4b8b" event={"ID":"62e3ed6d-3cc2-4e79-9738-43a802b6d58b","Type":"ContainerStarted","Data":"92156f9a3149336337221add11aa03a36882c2a4ad2b74306bad6bfbb57ab416"} Oct 30 16:42:09.686609 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:09.686522 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-f654570d-openshift-default-76cdfc5fcf-kwdnz" Oct 30 16:42:09.686799 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:09.686680 5052 patch_prober.go:29] interesting pod/e2e-ait-f654570d-openshift-default-76cdfc5fcf-kwdnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 30 16:42:09.686799 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:09.686736 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f654570d-openshift-default-76cdfc5fcf-kwdnz" podUID="f6092ad4-fa04-4813-aebb-655353769fc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 30 16:42:09.811605 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:09.810995 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-4db2924d-openshift-default-bc6f66585-r4b8b" Oct 30 16:42:09.811605 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:09.811127 5052 patch_prober.go:29] interesting pod/e2e-derive-4db2924d-openshift-default-bc6f66585-r4b8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 30 16:42:09.811605 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:09.811169 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-4db2924d-openshift-default-bc6f66585-r4b8b" podUID="62e3ed6d-3cc2-4e79-9738-43a802b6d58b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 30 16:42:10.146674 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:10.145111 5052 patch_prober.go:29] interesting pod/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-92x47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 16:42:10.146674 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:10.145190 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-92x47" podUID="39ebd9d1-3580-45a0-84ca-de3126fe5224" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 16:42:10.662611 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:10.648745 5052 patch_prober.go:29] interesting pod/e2e-ait-f654570d-openshift-default-76cdfc5fcf-kwdnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 30 16:42:10.662611 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:10.648810 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f654570d-openshift-default-76cdfc5fcf-kwdnz" podUID="f6092ad4-fa04-4813-aebb-655353769fc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 30 16:42:10.693605 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:10.692012 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-4db2924d-6ff4bd685d-mkkgw"] Oct 30 16:42:10.957833 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:10.819396 5052 patch_prober.go:29] interesting pod/e2e-derive-4db2924d-openshift-default-bc6f66585-r4b8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 30 16:42:10.988389 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:10.988346 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-4db2924d-openshift-default-bc6f66585-r4b8b" podUID="62e3ed6d-3cc2-4e79-9738-43a802b6d58b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 30 16:42:11.204855 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:11.204815 5052 patch_prober.go:29] interesting pod/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-92x47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 16:42:11.208750 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:11.208661 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-92x47" podUID="39ebd9d1-3580-45a0-84ca-de3126fe5224" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 16:42:11.537665 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:11.524512 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/distinct-w5-967fac-kserve-69cbcccd8b-b85bn" podUID="8f6b9354-378f-4451-9ff7-be7f7fec12f2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:42:11.683502 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:11.683463 5052 patch_prober.go:29] interesting pod/e2e-ait-f654570d-openshift-default-76cdfc5fcf-kwdnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 30 16:42:11.684986 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:11.684945 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f654570d-openshift-default-76cdfc5fcf-kwdnz" podUID="f6092ad4-fa04-4813-aebb-655353769fc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 30 16:42:11.827699 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:11.820816 5052 patch_prober.go:29] interesting pod/e2e-derive-4db2924d-openshift-default-bc6f66585-r4b8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 30 16:42:11.827988 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:11.827954 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-4db2924d-openshift-default-bc6f66585-r4b8b" podUID="62e3ed6d-3cc2-4e79-9738-43a802b6d58b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 30 16:42:12.206840 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:12.161821 5052 patch_prober.go:29] interesting pod/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-92x47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 16:42:12.206840 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:12.161882 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-92x47" podUID="39ebd9d1-3580-45a0-84ca-de3126fe5224" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 16:42:12.255133 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:12.242697 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx" podUID="bd662f3a-659f-477c-b8d5-94c64288c0c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:42:12.415710 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:12.415664 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-f654570d-56d5957948-lh66x" podUID="cf3426ff-3476-4dbe-8fbf-8196cabedfa4" containerName="payload-processing" containerID="cri-o://e84bdf7f621343f796307bed370d4da876f8582157df6f2132fe164ade1eff4a" gracePeriod=25 Oct 30 16:42:12.417251 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:12.416670 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-f654570d-7b49ffb9f-2s4q4" podUID="d7c5678b-63b4-4fc1-89ca-f43a6ca9926a" containerName="payload-pre-processing" containerID="cri-o://ee15b979f6adbc9b8c73abfd0de8a167491e5998f763042c82058b632b83e968" gracePeriod=25 Oct 30 16:42:12.539440 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:12.535256 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d693d0/distinct-2-w6-d693d0-kserve-6498fdc474-dpbjj" podUID="4b98b8ff-72cd-4ddb-a4b8-67078162e611" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:42:12.721613 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:12.720377 5052 patch_prober.go:29] interesting pod/e2e-ait-f654570d-openshift-default-76cdfc5fcf-kwdnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 30 16:42:12.721613 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:12.720439 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f654570d-openshift-default-76cdfc5fcf-kwdnz" podUID="f6092ad4-fa04-4813-aebb-655353769fc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 30 16:42:12.828658 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:12.827699 5052 patch_prober.go:29] interesting pod/e2e-derive-4db2924d-openshift-default-bc6f66585-r4b8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 30 16:42:12.828658 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:12.827763 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-4db2924d-openshift-default-bc6f66585-r4b8b" podUID="62e3ed6d-3cc2-4e79-9738-43a802b6d58b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 30 16:42:12.896780 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:12.896656 5052 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-967fac/distinct-w5-967fac-kserve-69cbcccd8b-b85bn" podUID="8f6b9354-378f-4451-9ff7-be7f7fec12f2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:42:13.176665 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:13.154704 5052 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-750dcb/distinct-w2-750dcb-kserve-86bbdfcbb-bpn77" podUID="c45190af-d795-4e13-8f2f-670b2c597658" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:42:13.196608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:13.195659 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/distinct-2-w4-263f00-kserve-8464897cf5-q8ck8" podUID="15ed43ef-5cdd-489a-a70e-063239822ff3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:42:13.262687 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:13.250086 5052 patch_prober.go:29] interesting pod/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-92x47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 16:42:13.262687 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:13.250154 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-92x47" podUID="39ebd9d1-3580-45a0-84ca-de3126fe5224" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 16:42:13.650073 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:13.646571 5052 patch_prober.go:29] interesting pod/e2e-ait-f654570d-openshift-default-76cdfc5fcf-kwdnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 30 16:42:13.650073 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:13.646763 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f654570d-openshift-default-76cdfc5fcf-kwdnz" podUID="f6092ad4-fa04-4813-aebb-655353769fc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 30 16:42:13.718459 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:13.718430 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f654570d-7b49ffb9f-2s4q4" Oct 30 16:42:13.805634 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:13.805600 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7c5678b-63b4-4fc1-89ca-f43a6ca9926a-kube-api-access-nwc24\" (UniqueName: \"kubernetes.io/projected/d7c5678b-63b4-4fc1-89ca-f43a6ca9926a-kube-api-access-nwc24\") pod \"d7c5678b-63b4-4fc1-89ca-f43a6ca9926a\" (UID: \"d7c5678b-63b4-4fc1-89ca-f43a6ca9926a\") " Oct 30 16:42:13.806345 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:13.806318 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7c5678b-63b4-4fc1-89ca-f43a6ca9926a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7c5678b-63b4-4fc1-89ca-f43a6ca9926a-plugins-config-volume\") pod \"d7c5678b-63b4-4fc1-89ca-f43a6ca9926a\" (UID: \"d7c5678b-63b4-4fc1-89ca-f43a6ca9926a\") " Oct 30 16:42:13.806817 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:13.806672 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7c5678b-63b4-4fc1-89ca-f43a6ca9926a-plugins-config-volume" pod "d7c5678b-63b4-4fc1-89ca-f43a6ca9926a" (UID: "d7c5678b-63b4-4fc1-89ca-f43a6ca9926a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:42:13.807715 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:13.807690 5052 patch_prober.go:29] interesting pod/e2e-derive-4db2924d-openshift-default-bc6f66585-r4b8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 30 16:42:13.807810 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:13.807756 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-4db2924d-openshift-default-bc6f66585-r4b8b" podUID="62e3ed6d-3cc2-4e79-9738-43a802b6d58b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 30 16:42:13.809116 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:13.809097 5052 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7c5678b-63b4-4fc1-89ca-f43a6ca9926a-plugins-config-volume\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:42:13.811876 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:13.811857 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7c5678b-63b4-4fc1-89ca-f43a6ca9926a-kube-api-access-nwc24" pod "d7c5678b-63b4-4fc1-89ca-f43a6ca9926a" (UID: "d7c5678b-63b4-4fc1-89ca-f43a6ca9926a"). InnerVolumeSpecName "kube-api-access-nwc24". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:42:13.898292 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:13.898246 5052 generic.go:356] "Generic (PLEG): container finished" podID="cf3426ff-3476-4dbe-8fbf-8196cabedfa4" containerID="e84bdf7f621343f796307bed370d4da876f8582157df6f2132fe164ade1eff4a" exitCode=0 Oct 30 16:42:13.898737 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:13.898698 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f654570d-56d5957948-lh66x" event={"ID":"cf3426ff-3476-4dbe-8fbf-8196cabedfa4","Type":"ContainerDied","Data":"e84bdf7f621343f796307bed370d4da876f8582157df6f2132fe164ade1eff4a"} Oct 30 16:42:13.898936 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:13.898917 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f654570d-56d5957948-lh66x" event={"ID":"cf3426ff-3476-4dbe-8fbf-8196cabedfa4","Type":"ContainerDied","Data":"bb3c34a7011c41e3d509cfb9bb8b96b83e09e7910b24a54aa100b18a45a9e54e"} Oct 30 16:42:13.899070 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:13.899054 5052 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bb3c34a7011c41e3d509cfb9bb8b96b83e09e7910b24a54aa100b18a45a9e54e" Oct 30 16:42:13.901646 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:13.900898 5052 generic.go:356] "Generic (PLEG): container finished" podID="d7c5678b-63b4-4fc1-89ca-f43a6ca9926a" containerID="ee15b979f6adbc9b8c73abfd0de8a167491e5998f763042c82058b632b83e968" exitCode=0 Oct 30 16:42:13.901646 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:13.901002 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f654570d-7b49ffb9f-2s4q4" event={"ID":"d7c5678b-63b4-4fc1-89ca-f43a6ca9926a","Type":"ContainerDied","Data":"ee15b979f6adbc9b8c73abfd0de8a167491e5998f763042c82058b632b83e968"} Oct 30 16:42:13.901646 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:13.901060 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f654570d-7b49ffb9f-2s4q4" event={"ID":"d7c5678b-63b4-4fc1-89ca-f43a6ca9926a","Type":"ContainerDied","Data":"4a4ed501b6ac50154aecf1e1e2292064239ac041ca81c4f3505a80812e2c1676"} Oct 30 16:42:13.901646 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:13.901093 5052 scope.go:122] "RemoveContainer" containerID="ee15b979f6adbc9b8c73abfd0de8a167491e5998f763042c82058b632b83e968" Oct 30 16:42:13.901646 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:13.901307 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f654570d-7b49ffb9f-2s4q4" Oct 30 16:42:13.902209 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:13.902187 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f654570d-56d5957948-lh66x" Oct 30 16:42:13.910803 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:13.910767 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwc24\" (UniqueName: \"kubernetes.io/projected/d7c5678b-63b4-4fc1-89ca-f43a6ca9926a-kube-api-access-nwc24\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:42:13.923685 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:13.923663 5052 scope.go:122] "RemoveContainer" containerID="ee15b979f6adbc9b8c73abfd0de8a167491e5998f763042c82058b632b83e968" Oct 30 16:42:13.924139 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:42:13.924117 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee15b979f6adbc9b8c73abfd0de8a167491e5998f763042c82058b632b83e968\": container with ID starting with ee15b979f6adbc9b8c73abfd0de8a167491e5998f763042c82058b632b83e968 not found: ID does not exist" containerID="ee15b979f6adbc9b8c73abfd0de8a167491e5998f763042c82058b632b83e968" Oct 30 16:42:13.924222 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:13.924153 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee15b979f6adbc9b8c73abfd0de8a167491e5998f763042c82058b632b83e968"} err="failed to get container status \"ee15b979f6adbc9b8c73abfd0de8a167491e5998f763042c82058b632b83e968\": rpc error: code = NotFound desc = could not find container \"ee15b979f6adbc9b8c73abfd0de8a167491e5998f763042c82058b632b83e968\": container with ID starting with ee15b979f6adbc9b8c73abfd0de8a167491e5998f763042c82058b632b83e968 not found: ID does not exist" Oct 30 16:42:14.008404 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:14.008372 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f654570d-7b49ffb9f-2s4q4"] Oct 30 16:42:14.011652 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:14.011622 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf3426ff-3476-4dbe-8fbf-8196cabedfa4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf3426ff-3476-4dbe-8fbf-8196cabedfa4-plugins-config-volume\") pod \"cf3426ff-3476-4dbe-8fbf-8196cabedfa4\" (UID: \"cf3426ff-3476-4dbe-8fbf-8196cabedfa4\") " Oct 30 16:42:14.011875 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:14.011856 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf3426ff-3476-4dbe-8fbf-8196cabedfa4-kube-api-access-cvkn7\" (UniqueName: \"kubernetes.io/projected/cf3426ff-3476-4dbe-8fbf-8196cabedfa4-kube-api-access-cvkn7\") pod \"cf3426ff-3476-4dbe-8fbf-8196cabedfa4\" (UID: \"cf3426ff-3476-4dbe-8fbf-8196cabedfa4\") " Oct 30 16:42:14.012086 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:14.012059 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf3426ff-3476-4dbe-8fbf-8196cabedfa4-plugins-config-volume" pod "cf3426ff-3476-4dbe-8fbf-8196cabedfa4" (UID: "cf3426ff-3476-4dbe-8fbf-8196cabedfa4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:42:14.012773 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:14.012748 5052 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf3426ff-3476-4dbe-8fbf-8196cabedfa4-plugins-config-volume\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:42:14.013152 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:14.013133 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f654570d-7b49ffb9f-2s4q4"] Oct 30 16:42:14.014122 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:14.014090 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf3426ff-3476-4dbe-8fbf-8196cabedfa4-kube-api-access-cvkn7" pod "cf3426ff-3476-4dbe-8fbf-8196cabedfa4" (UID: "cf3426ff-3476-4dbe-8fbf-8196cabedfa4"). InnerVolumeSpecName "kube-api-access-cvkn7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:42:14.114564 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:14.114489 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvkn7\" (UniqueName: \"kubernetes.io/projected/cf3426ff-3476-4dbe-8fbf-8196cabedfa4-kube-api-access-cvkn7\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:42:14.149606 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:14.143346 5052 patch_prober.go:29] interesting pod/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-92x47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 16:42:14.149606 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:14.143407 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-92x47" podUID="39ebd9d1-3580-45a0-84ca-de3126fe5224" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 16:42:14.613090 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:14.613058 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-4db2924d-openshift-default-bc6f66585-r4b8b" Oct 30 16:42:14.626128 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:14.626051 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7c5678b-63b4-4fc1-89ca-f43a6ca9926a" path="/var/lib/kubelet/pods/d7c5678b-63b4-4fc1-89ca-f43a6ca9926a/volumes" Oct 30 16:42:14.626599 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:14.626558 5052 patch_prober.go:29] interesting pod/e2e-ait-f654570d-openshift-default-76cdfc5fcf-kwdnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 30 16:42:14.626740 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:14.626622 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f654570d-openshift-default-76cdfc5fcf-kwdnz" podUID="f6092ad4-fa04-4813-aebb-655353769fc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 30 16:42:14.807468 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:14.807428 5052 patch_prober.go:29] interesting pod/e2e-derive-4db2924d-openshift-default-bc6f66585-r4b8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 30 16:42:14.807903 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:14.807480 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-4db2924d-openshift-default-bc6f66585-r4b8b" podUID="62e3ed6d-3cc2-4e79-9738-43a802b6d58b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 30 16:42:14.910135 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:14.910097 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f654570d-56d5957948-lh66x" Oct 30 16:42:14.999688 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:14.999645 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f654570d-56d5957948-lh66x"] Oct 30 16:42:15.003495 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:15.003468 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f654570d-56d5957948-lh66x"] Oct 30 16:42:15.625704 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:15.625600 5052 patch_prober.go:29] interesting pod/e2e-ait-f654570d-openshift-default-76cdfc5fcf-kwdnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 30 16:42:15.625704 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:15.625648 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f654570d-openshift-default-76cdfc5fcf-kwdnz" podUID="f6092ad4-fa04-4813-aebb-655353769fc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 30 16:42:15.711616 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:15.711350 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-4db2924d-6ff4bd685d-mkkgw" podUID="d00a8c79-7b28-4ea2-97ea-d76e2df5e441" containerName="payload-processing" containerID="cri-o://dd5c94607bc92e2b64cc06f85b82178f2f5c9655850f826d7c1e92a50b47f269" gracePeriod=25 Oct 30 16:42:15.806937 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:15.806823 5052 patch_prober.go:29] interesting pod/e2e-derive-4db2924d-openshift-default-bc6f66585-r4b8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 30 16:42:15.806937 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:15.806876 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-4db2924d-openshift-default-bc6f66585-r4b8b" podUID="62e3ed6d-3cc2-4e79-9738-43a802b6d58b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 30 16:42:15.811518 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:15.811489 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-92x47" Oct 30 16:42:15.812569 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:15.812543 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-92x47" Oct 30 16:42:15.814709 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:15.814684 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-92x47" Oct 30 16:42:16.009604 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:16.004614 5052 generic.go:356] "Generic (PLEG): container finished" podID="d00a8c79-7b28-4ea2-97ea-d76e2df5e441" containerID="dd5c94607bc92e2b64cc06f85b82178f2f5c9655850f826d7c1e92a50b47f269" exitCode=0 Oct 30 16:42:16.009604 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:16.005897 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-4db2924d-6ff4bd685d-mkkgw" event={"ID":"d00a8c79-7b28-4ea2-97ea-d76e2df5e441","Type":"ContainerDied","Data":"dd5c94607bc92e2b64cc06f85b82178f2f5c9655850f826d7c1e92a50b47f269"} Oct 30 16:42:16.352455 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:16.352423 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-4db2924d-6ff4bd685d-mkkgw" Oct 30 16:42:16.423896 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:16.423243 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d00a8c79-7b28-4ea2-97ea-d76e2df5e441-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d00a8c79-7b28-4ea2-97ea-d76e2df5e441-plugins-config-volume\") pod \"d00a8c79-7b28-4ea2-97ea-d76e2df5e441\" (UID: \"d00a8c79-7b28-4ea2-97ea-d76e2df5e441\") " Oct 30 16:42:16.423896 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:16.423608 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d00a8c79-7b28-4ea2-97ea-d76e2df5e441-kube-api-access-kj5v8\" (UniqueName: \"kubernetes.io/projected/d00a8c79-7b28-4ea2-97ea-d76e2df5e441-kube-api-access-kj5v8\") pod \"d00a8c79-7b28-4ea2-97ea-d76e2df5e441\" (UID: \"d00a8c79-7b28-4ea2-97ea-d76e2df5e441\") " Oct 30 16:42:16.435689 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:16.434236 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d00a8c79-7b28-4ea2-97ea-d76e2df5e441-plugins-config-volume" pod "d00a8c79-7b28-4ea2-97ea-d76e2df5e441" (UID: "d00a8c79-7b28-4ea2-97ea-d76e2df5e441"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:42:16.451104 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:16.448543 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d00a8c79-7b28-4ea2-97ea-d76e2df5e441-kube-api-access-kj5v8" pod "d00a8c79-7b28-4ea2-97ea-d76e2df5e441" (UID: "d00a8c79-7b28-4ea2-97ea-d76e2df5e441"). InnerVolumeSpecName "kube-api-access-kj5v8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:42:16.571603 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:16.553996 5052 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d00a8c79-7b28-4ea2-97ea-d76e2df5e441-plugins-config-volume\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:42:16.571603 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:16.554037 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kj5v8\" (UniqueName: \"kubernetes.io/projected/d00a8c79-7b28-4ea2-97ea-d76e2df5e441-kube-api-access-kj5v8\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:42:16.615428 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:16.611101 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf3426ff-3476-4dbe-8fbf-8196cabedfa4" path="/var/lib/kubelet/pods/cf3426ff-3476-4dbe-8fbf-8196cabedfa4/volumes" Oct 30 16:42:16.674044 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:16.674011 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-f654570d-openshift-default-76cdfc5fcf-kwdnz" Oct 30 16:42:16.675684 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:16.675613 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-f654570d-openshift-default-76cdfc5fcf-kwdnz" Oct 30 16:42:16.679518 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:16.679448 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-f654570d-openshift-default-76cdfc5fcf-kwdnz" Oct 30 16:42:16.706027 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:42:16.705955 5052 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd00a8c79_7b28_4ea2_97ea_d76e2df5e441.slice/crio-fc041e09ce4b5e87f1441bb1e7a6f6da176e19e45668c6522bb8553d2f74502c\": RecentStats: unable to find data in memory cache]" Oct 30 16:42:16.962544 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:16.962466 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-4db2924d-openshift-default-bc6f66585-r4b8b" Oct 30 16:42:16.964072 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:16.964041 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-4db2924d-openshift-default-bc6f66585-r4b8b" Oct 30 16:42:17.013947 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:17.013922 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-4db2924d-6ff4bd685d-mkkgw" Oct 30 16:42:17.014149 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:17.013963 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-4db2924d-6ff4bd685d-mkkgw" event={"ID":"d00a8c79-7b28-4ea2-97ea-d76e2df5e441","Type":"ContainerDied","Data":"fc041e09ce4b5e87f1441bb1e7a6f6da176e19e45668c6522bb8553d2f74502c"} Oct 30 16:42:17.014149 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:17.014021 5052 scope.go:122] "RemoveContainer" containerID="dd5c94607bc92e2b64cc06f85b82178f2f5c9655850f826d7c1e92a50b47f269" Oct 30 16:42:17.095910 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:17.095870 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-4db2924d-6ff4bd685d-mkkgw"] Oct 30 16:42:17.099702 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:17.099626 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-4db2924d-6ff4bd685d-mkkgw"] Oct 30 16:42:17.729370 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:17.729339 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-4db2924d-7fdc89f884-6t9w8"] Oct 30 16:42:17.729615 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:17.729562 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4db2924d-7fdc89f884-6t9w8" podUID="69a28b3b-3990-46f7-b2da-42824fb9af4a" containerName="maas-api" containerID="cri-o://44a5a72becfb9491d652be2e66d73855d2bc8a6f4732a9a3f25b2b4a97c23f52" gracePeriod=30 Oct 30 16:42:18.139653 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:18.135660 5052 generic.go:356] "Generic (PLEG): container finished" podID="69a28b3b-3990-46f7-b2da-42824fb9af4a" containerID="44a5a72becfb9491d652be2e66d73855d2bc8a6f4732a9a3f25b2b4a97c23f52" exitCode=0 Oct 30 16:42:18.139653 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:18.139606 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4db2924d-7fdc89f884-6t9w8" event={"ID":"69a28b3b-3990-46f7-b2da-42824fb9af4a","Type":"ContainerDied","Data":"44a5a72becfb9491d652be2e66d73855d2bc8a6f4732a9a3f25b2b4a97c23f52"} Oct 30 16:42:18.394025 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:18.393934 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4db2924d-7fdc89f884-6t9w8" Oct 30 16:42:18.571377 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:18.492217 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69a28b3b-3990-46f7-b2da-42824fb9af4a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/69a28b3b-3990-46f7-b2da-42824fb9af4a-metrics-certs\") pod \"69a28b3b-3990-46f7-b2da-42824fb9af4a\" (UID: \"69a28b3b-3990-46f7-b2da-42824fb9af4a\") " Oct 30 16:42:18.574030 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:18.573988 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69a28b3b-3990-46f7-b2da-42824fb9af4a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69a28b3b-3990-46f7-b2da-42824fb9af4a-maas-api-tls\") pod \"69a28b3b-3990-46f7-b2da-42824fb9af4a\" (UID: \"69a28b3b-3990-46f7-b2da-42824fb9af4a\") " Oct 30 16:42:18.574291 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:18.574266 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69a28b3b-3990-46f7-b2da-42824fb9af4a-kube-api-access-nljpz\" (UniqueName: \"kubernetes.io/projected/69a28b3b-3990-46f7-b2da-42824fb9af4a-kube-api-access-nljpz\") pod \"69a28b3b-3990-46f7-b2da-42824fb9af4a\" (UID: \"69a28b3b-3990-46f7-b2da-42824fb9af4a\") " Oct 30 16:42:18.574488 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:18.574472 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69a28b3b-3990-46f7-b2da-42824fb9af4a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69a28b3b-3990-46f7-b2da-42824fb9af4a-openshift-service-ca\") pod \"69a28b3b-3990-46f7-b2da-42824fb9af4a\" (UID: \"69a28b3b-3990-46f7-b2da-42824fb9af4a\") " Oct 30 16:42:18.602625 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:18.538742 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69a28b3b-3990-46f7-b2da-42824fb9af4a-metrics-certs" pod "69a28b3b-3990-46f7-b2da-42824fb9af4a" (UID: "69a28b3b-3990-46f7-b2da-42824fb9af4a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:42:18.614721 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:18.608153 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69a28b3b-3990-46f7-b2da-42824fb9af4a-maas-api-tls" pod "69a28b3b-3990-46f7-b2da-42824fb9af4a" (UID: "69a28b3b-3990-46f7-b2da-42824fb9af4a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:42:18.643002 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:18.628764 5052 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69a28b3b-3990-46f7-b2da-42824fb9af4a-metrics-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:42:18.643002 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:18.628795 5052 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69a28b3b-3990-46f7-b2da-42824fb9af4a-maas-api-tls\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:42:18.643002 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:18.635783 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69a28b3b-3990-46f7-b2da-42824fb9af4a-openshift-service-ca" pod "69a28b3b-3990-46f7-b2da-42824fb9af4a" (UID: "69a28b3b-3990-46f7-b2da-42824fb9af4a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:42:18.643225 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:18.643053 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69a28b3b-3990-46f7-b2da-42824fb9af4a-kube-api-access-nljpz" pod "69a28b3b-3990-46f7-b2da-42824fb9af4a" (UID: "69a28b3b-3990-46f7-b2da-42824fb9af4a"). InnerVolumeSpecName "kube-api-access-nljpz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:42:18.648059 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:18.647389 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d00a8c79-7b28-4ea2-97ea-d76e2df5e441" path="/var/lib/kubelet/pods/d00a8c79-7b28-4ea2-97ea-d76e2df5e441/volumes" Oct 30 16:42:18.731336 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:18.731297 5052 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69a28b3b-3990-46f7-b2da-42824fb9af4a-openshift-service-ca\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:42:18.731336 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:18.731343 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nljpz\" (UniqueName: \"kubernetes.io/projected/69a28b3b-3990-46f7-b2da-42824fb9af4a-kube-api-access-nljpz\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:42:18.847522 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:42:18.847471 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 16:42:18.976708 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:18.976627 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-4db2924d-openshift-default-bc6f66585-r4b8b"] Oct 30 16:42:18.976899 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:18.976870 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-4db2924d-openshift-default-bc6f66585-r4b8b" podUID="62e3ed6d-3cc2-4e79-9738-43a802b6d58b" containerName="istio-proxy" containerID="cri-o://92156f9a3149336337221add11aa03a36882c2a4ad2b74306bad6bfbb57ab416" gracePeriod=30 Oct 30 16:42:19.222533 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:19.217773 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4db2924d-7fdc89f884-6t9w8" event={"ID":"69a28b3b-3990-46f7-b2da-42824fb9af4a","Type":"ContainerDied","Data":"a878e945f6c5f98516e3bdbe9402375111e16491e7dbc4d661c3062ddc3b9067"} Oct 30 16:42:19.222533 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:19.217856 5052 scope.go:122] "RemoveContainer" containerID="44a5a72becfb9491d652be2e66d73855d2bc8a6f4732a9a3f25b2b4a97c23f52" Oct 30 16:42:19.222533 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:19.218002 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4db2924d-7fdc89f884-6t9w8" Oct 30 16:42:19.317657 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:19.312671 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-4db2924d-7fdc89f884-6t9w8"] Oct 30 16:42:19.328798 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:19.323312 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-4db2924d-7fdc89f884-6t9w8"] Oct 30 16:42:20.600955 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:20.600918 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69a28b3b-3990-46f7-b2da-42824fb9af4a" path="/var/lib/kubelet/pods/69a28b3b-3990-46f7-b2da-42824fb9af4a/volumes" Oct 30 16:42:24.339418 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:24.339381 5052 generic.go:356] "Generic (PLEG): container finished" podID="62e3ed6d-3cc2-4e79-9738-43a802b6d58b" containerID="92156f9a3149336337221add11aa03a36882c2a4ad2b74306bad6bfbb57ab416" exitCode=0 Oct 30 16:42:24.339904 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:24.339464 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-4db2924d-openshift-default-bc6f66585-r4b8b" event={"ID":"62e3ed6d-3cc2-4e79-9738-43a802b6d58b","Type":"ContainerDied","Data":"92156f9a3149336337221add11aa03a36882c2a4ad2b74306bad6bfbb57ab416"} Oct 30 16:42:24.339904 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:24.339534 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-4db2924d-openshift-default-bc6f66585-r4b8b" event={"ID":"62e3ed6d-3cc2-4e79-9738-43a802b6d58b","Type":"ContainerDied","Data":"ecc7ea57c684fc75a985e62f77715ebcdad2e7d244f244796860a91e43295491"} Oct 30 16:42:24.339904 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:24.339560 5052 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ecc7ea57c684fc75a985e62f77715ebcdad2e7d244f244796860a91e43295491" Oct 30 16:42:24.339904 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:24.339618 5052 scope.go:122] "RemoveContainer" containerID="38c010033fc2bc3d8d86c019978d27604a98087fef69dd5b1df3144af7f05b6d" Oct 30 16:42:24.347085 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:24.347061 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-4db2924d-openshift-default-bc6f66585-r4b8b" Oct 30 16:42:24.477573 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:24.477538 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62e3ed6d-3cc2-4e79-9738-43a802b6d58b-istio-token\" (UniqueName: \"kubernetes.io/projected/62e3ed6d-3cc2-4e79-9738-43a802b6d58b-istio-token\") pod \"62e3ed6d-3cc2-4e79-9738-43a802b6d58b\" (UID: \"62e3ed6d-3cc2-4e79-9738-43a802b6d58b\") " Oct 30 16:42:24.477791 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:24.477600 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62e3ed6d-3cc2-4e79-9738-43a802b6d58b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62e3ed6d-3cc2-4e79-9738-43a802b6d58b-workload-certs\") pod \"62e3ed6d-3cc2-4e79-9738-43a802b6d58b\" (UID: \"62e3ed6d-3cc2-4e79-9738-43a802b6d58b\") " Oct 30 16:42:24.477791 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:24.477654 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62e3ed6d-3cc2-4e79-9738-43a802b6d58b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62e3ed6d-3cc2-4e79-9738-43a802b6d58b-istio-data\") pod \"62e3ed6d-3cc2-4e79-9738-43a802b6d58b\" (UID: \"62e3ed6d-3cc2-4e79-9738-43a802b6d58b\") " Oct 30 16:42:24.477791 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:24.477768 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62e3ed6d-3cc2-4e79-9738-43a802b6d58b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62e3ed6d-3cc2-4e79-9738-43a802b6d58b-workload-socket\") pod \"62e3ed6d-3cc2-4e79-9738-43a802b6d58b\" (UID: \"62e3ed6d-3cc2-4e79-9738-43a802b6d58b\") " Oct 30 16:42:24.477980 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:24.477864 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62e3ed6d-3cc2-4e79-9738-43a802b6d58b-workload-certs" pod "62e3ed6d-3cc2-4e79-9738-43a802b6d58b" (UID: "62e3ed6d-3cc2-4e79-9738-43a802b6d58b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:42:24.477980 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:24.477902 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62e3ed6d-3cc2-4e79-9738-43a802b6d58b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62e3ed6d-3cc2-4e79-9738-43a802b6d58b-credential-socket\") pod \"62e3ed6d-3cc2-4e79-9738-43a802b6d58b\" (UID: \"62e3ed6d-3cc2-4e79-9738-43a802b6d58b\") " Oct 30 16:42:24.478091 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:24.477997 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62e3ed6d-3cc2-4e79-9738-43a802b6d58b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62e3ed6d-3cc2-4e79-9738-43a802b6d58b-istiod-ca-cert\") pod \"62e3ed6d-3cc2-4e79-9738-43a802b6d58b\" (UID: \"62e3ed6d-3cc2-4e79-9738-43a802b6d58b\") " Oct 30 16:42:24.478091 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:24.478067 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62e3ed6d-3cc2-4e79-9738-43a802b6d58b-workload-socket" pod "62e3ed6d-3cc2-4e79-9738-43a802b6d58b" (UID: "62e3ed6d-3cc2-4e79-9738-43a802b6d58b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:42:24.478217 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:24.478078 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/62e3ed6d-3cc2-4e79-9738-43a802b6d58b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62e3ed6d-3cc2-4e79-9738-43a802b6d58b-istio-podinfo\") pod \"62e3ed6d-3cc2-4e79-9738-43a802b6d58b\" (UID: \"62e3ed6d-3cc2-4e79-9738-43a802b6d58b\") " Oct 30 16:42:24.478217 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:24.478205 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62e3ed6d-3cc2-4e79-9738-43a802b6d58b-credential-socket" pod "62e3ed6d-3cc2-4e79-9738-43a802b6d58b" (UID: "62e3ed6d-3cc2-4e79-9738-43a802b6d58b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:42:24.478300 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:24.478252 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62e3ed6d-3cc2-4e79-9738-43a802b6d58b-kube-api-access-qtqfw\" (UniqueName: \"kubernetes.io/projected/62e3ed6d-3cc2-4e79-9738-43a802b6d58b-kube-api-access-qtqfw\") pod \"62e3ed6d-3cc2-4e79-9738-43a802b6d58b\" (UID: \"62e3ed6d-3cc2-4e79-9738-43a802b6d58b\") " Oct 30 16:42:24.478425 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:24.478342 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62e3ed6d-3cc2-4e79-9738-43a802b6d58b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62e3ed6d-3cc2-4e79-9738-43a802b6d58b-istio-envoy\") pod \"62e3ed6d-3cc2-4e79-9738-43a802b6d58b\" (UID: \"62e3ed6d-3cc2-4e79-9738-43a802b6d58b\") " Oct 30 16:42:24.479209 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:24.479185 5052 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62e3ed6d-3cc2-4e79-9738-43a802b6d58b-credential-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:42:24.479294 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:24.479217 5052 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62e3ed6d-3cc2-4e79-9738-43a802b6d58b-workload-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:42:24.479294 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:24.479232 5052 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62e3ed6d-3cc2-4e79-9738-43a802b6d58b-workload-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:42:24.479554 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:24.479525 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62e3ed6d-3cc2-4e79-9738-43a802b6d58b-istiod-ca-cert" pod "62e3ed6d-3cc2-4e79-9738-43a802b6d58b" (UID: "62e3ed6d-3cc2-4e79-9738-43a802b6d58b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:42:24.480247 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:24.480182 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62e3ed6d-3cc2-4e79-9738-43a802b6d58b-istio-data" pod "62e3ed6d-3cc2-4e79-9738-43a802b6d58b" (UID: "62e3ed6d-3cc2-4e79-9738-43a802b6d58b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:42:24.481100 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:24.481074 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62e3ed6d-3cc2-4e79-9738-43a802b6d58b-istio-token" pod "62e3ed6d-3cc2-4e79-9738-43a802b6d58b" (UID: "62e3ed6d-3cc2-4e79-9738-43a802b6d58b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:42:24.481626 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:24.481599 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/62e3ed6d-3cc2-4e79-9738-43a802b6d58b-istio-podinfo" pod "62e3ed6d-3cc2-4e79-9738-43a802b6d58b" (UID: "62e3ed6d-3cc2-4e79-9738-43a802b6d58b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:42:24.482068 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:24.482042 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62e3ed6d-3cc2-4e79-9738-43a802b6d58b-kube-api-access-qtqfw" pod "62e3ed6d-3cc2-4e79-9738-43a802b6d58b" (UID: "62e3ed6d-3cc2-4e79-9738-43a802b6d58b"). InnerVolumeSpecName "kube-api-access-qtqfw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:42:24.482443 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:24.482419 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62e3ed6d-3cc2-4e79-9738-43a802b6d58b-istio-envoy" pod "62e3ed6d-3cc2-4e79-9738-43a802b6d58b" (UID: "62e3ed6d-3cc2-4e79-9738-43a802b6d58b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:42:24.580518 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:24.580450 5052 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62e3ed6d-3cc2-4e79-9738-43a802b6d58b-istiod-ca-cert\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:42:24.580518 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:24.580476 5052 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62e3ed6d-3cc2-4e79-9738-43a802b6d58b-istio-podinfo\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:42:24.580518 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:24.580485 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtqfw\" (UniqueName: \"kubernetes.io/projected/62e3ed6d-3cc2-4e79-9738-43a802b6d58b-kube-api-access-qtqfw\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:42:24.580518 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:24.580494 5052 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62e3ed6d-3cc2-4e79-9738-43a802b6d58b-istio-envoy\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:42:24.580518 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:24.580502 5052 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62e3ed6d-3cc2-4e79-9738-43a802b6d58b-istio-token\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:42:24.580518 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:24.580510 5052 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62e3ed6d-3cc2-4e79-9738-43a802b6d58b-istio-data\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:42:25.349858 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:25.349822 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-4db2924d-openshift-default-bc6f66585-r4b8b" Oct 30 16:42:25.452560 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:25.452519 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-4db2924d-openshift-default-bc6f66585-r4b8b"] Oct 30 16:42:25.460412 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:25.460384 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-4db2924d-openshift-default-bc6f66585-r4b8b"] Oct 30 16:42:26.460912 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:26.460883 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-f654570d-openshift-default-76cdfc5fcf-kwdnz"] Oct 30 16:42:26.461360 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:26.461095 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-f654570d-openshift-default-76cdfc5fcf-kwdnz" podUID="f6092ad4-fa04-4813-aebb-655353769fc1" containerName="istio-proxy" containerID="cri-o://ccab10db017561022715c46306d0825207a302d6535fc999d7b0cf4913504374" gracePeriod=30 Oct 30 16:42:26.870054 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:26.867860 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62e3ed6d-3cc2-4e79-9738-43a802b6d58b" path="/var/lib/kubelet/pods/62e3ed6d-3cc2-4e79-9738-43a802b6d58b/volumes" Oct 30 16:42:31.758741 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:31.758704 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f654570d-openshift-default-76cdfc5fcf-kwdnz" Oct 30 16:42:31.824972 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:31.824933 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6092ad4-fa04-4813-aebb-655353769fc1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6092ad4-fa04-4813-aebb-655353769fc1-istiod-ca-cert\") pod \"f6092ad4-fa04-4813-aebb-655353769fc1\" (UID: \"f6092ad4-fa04-4813-aebb-655353769fc1\") " Oct 30 16:42:31.825192 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:31.825017 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6092ad4-fa04-4813-aebb-655353769fc1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6092ad4-fa04-4813-aebb-655353769fc1-istio-data\") pod \"f6092ad4-fa04-4813-aebb-655353769fc1\" (UID: \"f6092ad4-fa04-4813-aebb-655353769fc1\") " Oct 30 16:42:31.825192 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:31.825078 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6092ad4-fa04-4813-aebb-655353769fc1-istio-token\" (UniqueName: \"kubernetes.io/projected/f6092ad4-fa04-4813-aebb-655353769fc1-istio-token\") pod \"f6092ad4-fa04-4813-aebb-655353769fc1\" (UID: \"f6092ad4-fa04-4813-aebb-655353769fc1\") " Oct 30 16:42:31.825192 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:31.825124 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6092ad4-fa04-4813-aebb-655353769fc1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6092ad4-fa04-4813-aebb-655353769fc1-workload-socket\") pod \"f6092ad4-fa04-4813-aebb-655353769fc1\" (UID: \"f6092ad4-fa04-4813-aebb-655353769fc1\") " Oct 30 16:42:31.825192 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:31.825155 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6092ad4-fa04-4813-aebb-655353769fc1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6092ad4-fa04-4813-aebb-655353769fc1-istio-envoy\") pod \"f6092ad4-fa04-4813-aebb-655353769fc1\" (UID: \"f6092ad4-fa04-4813-aebb-655353769fc1\") " Oct 30 16:42:31.825436 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:31.825334 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6092ad4-fa04-4813-aebb-655353769fc1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6092ad4-fa04-4813-aebb-655353769fc1-credential-socket\") pod \"f6092ad4-fa04-4813-aebb-655353769fc1\" (UID: \"f6092ad4-fa04-4813-aebb-655353769fc1\") " Oct 30 16:42:31.825436 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:31.825376 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6092ad4-fa04-4813-aebb-655353769fc1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6092ad4-fa04-4813-aebb-655353769fc1-workload-certs\") pod \"f6092ad4-fa04-4813-aebb-655353769fc1\" (UID: \"f6092ad4-fa04-4813-aebb-655353769fc1\") " Oct 30 16:42:31.825532 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:31.825448 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6092ad4-fa04-4813-aebb-655353769fc1-kube-api-access-msjqp\" (UniqueName: \"kubernetes.io/projected/f6092ad4-fa04-4813-aebb-655353769fc1-kube-api-access-msjqp\") pod \"f6092ad4-fa04-4813-aebb-655353769fc1\" (UID: \"f6092ad4-fa04-4813-aebb-655353769fc1\") " Oct 30 16:42:31.825532 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:31.825484 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f6092ad4-fa04-4813-aebb-655353769fc1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6092ad4-fa04-4813-aebb-655353769fc1-istio-podinfo\") pod \"f6092ad4-fa04-4813-aebb-655353769fc1\" (UID: \"f6092ad4-fa04-4813-aebb-655353769fc1\") " Oct 30 16:42:31.828685 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:31.827795 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6092ad4-fa04-4813-aebb-655353769fc1-istiod-ca-cert" pod "f6092ad4-fa04-4813-aebb-655353769fc1" (UID: "f6092ad4-fa04-4813-aebb-655353769fc1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:42:31.828685 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:31.828053 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6092ad4-fa04-4813-aebb-655353769fc1-istio-data" pod "f6092ad4-fa04-4813-aebb-655353769fc1" (UID: "f6092ad4-fa04-4813-aebb-655353769fc1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:42:31.828685 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:31.828262 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6092ad4-fa04-4813-aebb-655353769fc1-credential-socket" pod "f6092ad4-fa04-4813-aebb-655353769fc1" (UID: "f6092ad4-fa04-4813-aebb-655353769fc1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:42:31.828685 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:31.828433 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6092ad4-fa04-4813-aebb-655353769fc1-workload-socket" pod "f6092ad4-fa04-4813-aebb-655353769fc1" (UID: "f6092ad4-fa04-4813-aebb-655353769fc1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:42:31.829630 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:31.829592 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6092ad4-fa04-4813-aebb-655353769fc1-workload-certs" pod "f6092ad4-fa04-4813-aebb-655353769fc1" (UID: "f6092ad4-fa04-4813-aebb-655353769fc1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:42:31.830267 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:31.829918 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f6092ad4-fa04-4813-aebb-655353769fc1-istio-podinfo" pod "f6092ad4-fa04-4813-aebb-655353769fc1" (UID: "f6092ad4-fa04-4813-aebb-655353769fc1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:42:31.838640 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:31.837297 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6092ad4-fa04-4813-aebb-655353769fc1-istio-token" pod "f6092ad4-fa04-4813-aebb-655353769fc1" (UID: "f6092ad4-fa04-4813-aebb-655353769fc1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:42:31.838640 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:31.837393 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6092ad4-fa04-4813-aebb-655353769fc1-istio-envoy" pod "f6092ad4-fa04-4813-aebb-655353769fc1" (UID: "f6092ad4-fa04-4813-aebb-655353769fc1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:42:31.838640 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:31.838562 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6092ad4-fa04-4813-aebb-655353769fc1-kube-api-access-msjqp" pod "f6092ad4-fa04-4813-aebb-655353769fc1" (UID: "f6092ad4-fa04-4813-aebb-655353769fc1"). InnerVolumeSpecName "kube-api-access-msjqp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:42:31.927370 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:31.927334 5052 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6092ad4-fa04-4813-aebb-655353769fc1-istiod-ca-cert\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:42:31.927370 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:31.927366 5052 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6092ad4-fa04-4813-aebb-655353769fc1-istio-data\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:42:31.927370 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:31.927378 5052 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6092ad4-fa04-4813-aebb-655353769fc1-istio-token\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:42:31.927727 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:31.927390 5052 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6092ad4-fa04-4813-aebb-655353769fc1-workload-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:42:31.927727 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:31.927404 5052 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6092ad4-fa04-4813-aebb-655353769fc1-istio-envoy\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:42:31.927727 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:31.927416 5052 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6092ad4-fa04-4813-aebb-655353769fc1-credential-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:42:31.927727 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:31.927428 5052 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6092ad4-fa04-4813-aebb-655353769fc1-workload-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:42:31.927727 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:31.927441 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msjqp\" (UniqueName: \"kubernetes.io/projected/f6092ad4-fa04-4813-aebb-655353769fc1-kube-api-access-msjqp\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:42:31.927727 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:31.927453 5052 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6092ad4-fa04-4813-aebb-655353769fc1-istio-podinfo\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:42:32.402538 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:32.402499 5052 generic.go:356] "Generic (PLEG): container finished" podID="f6092ad4-fa04-4813-aebb-655353769fc1" containerID="ccab10db017561022715c46306d0825207a302d6535fc999d7b0cf4913504374" exitCode=0 Oct 30 16:42:32.402766 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:32.402570 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f654570d-openshift-default-76cdfc5fcf-kwdnz" Oct 30 16:42:32.402766 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:32.402614 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f654570d-openshift-default-76cdfc5fcf-kwdnz" event={"ID":"f6092ad4-fa04-4813-aebb-655353769fc1","Type":"ContainerDied","Data":"ccab10db017561022715c46306d0825207a302d6535fc999d7b0cf4913504374"} Oct 30 16:42:32.402766 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:32.402698 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f654570d-openshift-default-76cdfc5fcf-kwdnz" event={"ID":"f6092ad4-fa04-4813-aebb-655353769fc1","Type":"ContainerDied","Data":"85f92d6dd4362f1034f060cde1940b8c9fa66e240c407e2f09570d4beec251ff"} Oct 30 16:42:32.402766 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:32.402722 5052 scope.go:122] "RemoveContainer" containerID="ccab10db017561022715c46306d0825207a302d6535fc999d7b0cf4913504374" Oct 30 16:42:32.415459 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:32.415440 5052 scope.go:122] "RemoveContainer" containerID="6939f75159ba401a04fb22fb3ae142dfd65077ddfe28f96281a7b01d0de70471" Oct 30 16:42:32.431883 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:32.431865 5052 scope.go:122] "RemoveContainer" containerID="ccab10db017561022715c46306d0825207a302d6535fc999d7b0cf4913504374" Oct 30 16:42:32.432176 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:42:32.432153 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccab10db017561022715c46306d0825207a302d6535fc999d7b0cf4913504374\": container with ID starting with ccab10db017561022715c46306d0825207a302d6535fc999d7b0cf4913504374 not found: ID does not exist" containerID="ccab10db017561022715c46306d0825207a302d6535fc999d7b0cf4913504374" Oct 30 16:42:32.432240 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:32.432186 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccab10db017561022715c46306d0825207a302d6535fc999d7b0cf4913504374"} err="failed to get container status \"ccab10db017561022715c46306d0825207a302d6535fc999d7b0cf4913504374\": rpc error: code = NotFound desc = could not find container \"ccab10db017561022715c46306d0825207a302d6535fc999d7b0cf4913504374\": container with ID starting with ccab10db017561022715c46306d0825207a302d6535fc999d7b0cf4913504374 not found: ID does not exist" Oct 30 16:42:32.432240 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:32.432207 5052 scope.go:122] "RemoveContainer" containerID="6939f75159ba401a04fb22fb3ae142dfd65077ddfe28f96281a7b01d0de70471" Oct 30 16:42:32.432453 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:42:32.432438 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6939f75159ba401a04fb22fb3ae142dfd65077ddfe28f96281a7b01d0de70471\": container with ID starting with 6939f75159ba401a04fb22fb3ae142dfd65077ddfe28f96281a7b01d0de70471 not found: ID does not exist" containerID="6939f75159ba401a04fb22fb3ae142dfd65077ddfe28f96281a7b01d0de70471" Oct 30 16:42:32.432507 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:32.432456 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6939f75159ba401a04fb22fb3ae142dfd65077ddfe28f96281a7b01d0de70471"} err="failed to get container status \"6939f75159ba401a04fb22fb3ae142dfd65077ddfe28f96281a7b01d0de70471\": rpc error: code = NotFound desc = could not find container \"6939f75159ba401a04fb22fb3ae142dfd65077ddfe28f96281a7b01d0de70471\": container with ID starting with 6939f75159ba401a04fb22fb3ae142dfd65077ddfe28f96281a7b01d0de70471 not found: ID does not exist" Oct 30 16:42:32.442555 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:32.442523 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a0777a0c-85f5c98494-2896v"] Oct 30 16:42:32.443209 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:32.443195 5052 cpu_manager.go:414] "containerMap: removing container" podUID="cf3426ff-3476-4dbe-8fbf-8196cabedfa4" containerName="payload-processing" Oct 30 16:42:32.443248 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:32.443210 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf3426ff-3476-4dbe-8fbf-8196cabedfa4" containerName="payload-processing" Oct 30 16:42:32.443248 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:32.443227 5052 cpu_manager.go:414] "containerMap: removing container" podUID="62e3ed6d-3cc2-4e79-9738-43a802b6d58b" containerName="istio-proxy" Oct 30 16:42:32.443248 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:32.443232 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62e3ed6d-3cc2-4e79-9738-43a802b6d58b" containerName="istio-proxy" Oct 30 16:42:32.443351 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:32.443250 5052 cpu_manager.go:414] "containerMap: removing container" podUID="d00a8c79-7b28-4ea2-97ea-d76e2df5e441" containerName="payload-processing" Oct 30 16:42:32.443351 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:32.443256 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d00a8c79-7b28-4ea2-97ea-d76e2df5e441" containerName="payload-processing" Oct 30 16:42:32.443351 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:32.443268 5052 cpu_manager.go:414] "containerMap: removing container" podUID="f6092ad4-fa04-4813-aebb-655353769fc1" containerName="istio-proxy" Oct 30 16:42:32.443351 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:32.443273 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6092ad4-fa04-4813-aebb-655353769fc1" containerName="istio-proxy" Oct 30 16:42:32.443351 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:32.443280 5052 cpu_manager.go:414] "containerMap: removing container" podUID="62e3ed6d-3cc2-4e79-9738-43a802b6d58b" containerName="istio-proxy" Oct 30 16:42:32.443351 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:32.443285 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62e3ed6d-3cc2-4e79-9738-43a802b6d58b" containerName="istio-proxy" Oct 30 16:42:32.443351 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:32.443293 5052 cpu_manager.go:414] "containerMap: removing container" podUID="d7c5678b-63b4-4fc1-89ca-f43a6ca9926a" containerName="payload-pre-processing" Oct 30 16:42:32.443351 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:32.443298 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7c5678b-63b4-4fc1-89ca-f43a6ca9926a" containerName="payload-pre-processing" Oct 30 16:42:32.443351 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:32.443310 5052 cpu_manager.go:414] "containerMap: removing container" podUID="69a28b3b-3990-46f7-b2da-42824fb9af4a" containerName="maas-api" Oct 30 16:42:32.443351 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:32.443317 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69a28b3b-3990-46f7-b2da-42824fb9af4a" containerName="maas-api" Oct 30 16:42:32.443637 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:32.443458 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7c5678b-63b4-4fc1-89ca-f43a6ca9926a" containerName="payload-pre-processing" Oct 30 16:42:32.443637 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:32.443469 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="69a28b3b-3990-46f7-b2da-42824fb9af4a" containerName="maas-api" Oct 30 16:42:32.443637 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:32.443479 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6092ad4-fa04-4813-aebb-655353769fc1" containerName="istio-proxy" Oct 30 16:42:32.443637 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:32.443489 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf3426ff-3476-4dbe-8fbf-8196cabedfa4" containerName="payload-processing" Oct 30 16:42:32.443637 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:32.443496 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="62e3ed6d-3cc2-4e79-9738-43a802b6d58b" containerName="istio-proxy" Oct 30 16:42:32.443637 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:32.443504 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="d00a8c79-7b28-4ea2-97ea-d76e2df5e441" containerName="payload-processing" Oct 30 16:42:32.448618 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:32.448598 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a0777a0c-85f5c98494-2896v" Oct 30 16:42:32.462397 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:32.462342 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-f654570d-openshift-default-76cdfc5fcf-kwdnz"] Oct 30 16:42:32.467387 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:32.467293 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a0777a0c-85f5c98494-2896v"] Oct 30 16:42:32.469671 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:32.468761 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-a0777a0c\"" Oct 30 16:42:32.469671 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:32.469511 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-a0777a0c-dockercfg-vcr5r\"" Oct 30 16:42:32.474848 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:32.474827 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-f654570d-openshift-default-76cdfc5fcf-kwdnz"] Oct 30 16:42:32.493115 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:32.493089 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a0777a0c-69644576f5-4t4vr"] Oct 30 16:42:32.493840 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:32.493823 5052 cpu_manager.go:414] "containerMap: removing container" podUID="f6092ad4-fa04-4813-aebb-655353769fc1" containerName="istio-proxy" Oct 30 16:42:32.493887 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:32.493841 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6092ad4-fa04-4813-aebb-655353769fc1" containerName="istio-proxy" Oct 30 16:42:32.493966 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:32.493957 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6092ad4-fa04-4813-aebb-655353769fc1" containerName="istio-proxy" Oct 30 16:42:32.494004 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:32.493971 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="62e3ed6d-3cc2-4e79-9738-43a802b6d58b" containerName="istio-proxy" Oct 30 16:42:32.497542 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:32.497521 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a0777a0c-69644576f5-4t4vr" Oct 30 16:42:32.521711 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:32.521679 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a0777a0c-69644576f5-4t4vr"] Oct 30 16:42:32.536479 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:32.536454 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43d9d192-2f2b-4a26-8ecc-fe143833e3b3-plugins-config-volume\") pod \"payload-processing-e2e-ait-a0777a0c-85f5c98494-2896v\" (UID: \"43d9d192-2f2b-4a26-8ecc-fe143833e3b3\") " pod="openshift-ingress/payload-processing-e2e-ait-a0777a0c-85f5c98494-2896v" Oct 30 16:42:32.536604 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:32.536568 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5xsh\" (UniqueName: \"kubernetes.io/projected/30dbe70e-554d-4f2c-a7e9-80546a168c30-kube-api-access-f5xsh\") pod \"payload-pre-processing-e2e-ait-a0777a0c-69644576f5-4t4vr\" (UID: \"30dbe70e-554d-4f2c-a7e9-80546a168c30\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a0777a0c-69644576f5-4t4vr" Oct 30 16:42:32.536708 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:32.536690 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmjb2\" (UniqueName: \"kubernetes.io/projected/43d9d192-2f2b-4a26-8ecc-fe143833e3b3-kube-api-access-qmjb2\") pod \"payload-processing-e2e-ait-a0777a0c-85f5c98494-2896v\" (UID: \"43d9d192-2f2b-4a26-8ecc-fe143833e3b3\") " pod="openshift-ingress/payload-processing-e2e-ait-a0777a0c-85f5c98494-2896v" Oct 30 16:42:32.536748 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:32.536726 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30dbe70e-554d-4f2c-a7e9-80546a168c30-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a0777a0c-69644576f5-4t4vr\" (UID: \"30dbe70e-554d-4f2c-a7e9-80546a168c30\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a0777a0c-69644576f5-4t4vr" Oct 30 16:42:32.669653 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:32.659655 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43d9d192-2f2b-4a26-8ecc-fe143833e3b3-plugins-config-volume\") pod \"payload-processing-e2e-ait-a0777a0c-85f5c98494-2896v\" (UID: \"43d9d192-2f2b-4a26-8ecc-fe143833e3b3\") " pod="openshift-ingress/payload-processing-e2e-ait-a0777a0c-85f5c98494-2896v" Oct 30 16:42:32.669653 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:32.659812 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5xsh\" (UniqueName: \"kubernetes.io/projected/30dbe70e-554d-4f2c-a7e9-80546a168c30-kube-api-access-f5xsh\") pod \"payload-pre-processing-e2e-ait-a0777a0c-69644576f5-4t4vr\" (UID: \"30dbe70e-554d-4f2c-a7e9-80546a168c30\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a0777a0c-69644576f5-4t4vr" Oct 30 16:42:32.669653 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:32.659920 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmjb2\" (UniqueName: \"kubernetes.io/projected/43d9d192-2f2b-4a26-8ecc-fe143833e3b3-kube-api-access-qmjb2\") pod \"payload-processing-e2e-ait-a0777a0c-85f5c98494-2896v\" (UID: \"43d9d192-2f2b-4a26-8ecc-fe143833e3b3\") " pod="openshift-ingress/payload-processing-e2e-ait-a0777a0c-85f5c98494-2896v" Oct 30 16:42:32.669653 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:32.659958 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30dbe70e-554d-4f2c-a7e9-80546a168c30-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a0777a0c-69644576f5-4t4vr\" (UID: \"30dbe70e-554d-4f2c-a7e9-80546a168c30\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a0777a0c-69644576f5-4t4vr" Oct 30 16:42:32.669653 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:32.662817 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43d9d192-2f2b-4a26-8ecc-fe143833e3b3-plugins-config-volume\") pod \"payload-processing-e2e-ait-a0777a0c-85f5c98494-2896v\" (UID: \"43d9d192-2f2b-4a26-8ecc-fe143833e3b3\") " pod="openshift-ingress/payload-processing-e2e-ait-a0777a0c-85f5c98494-2896v" Oct 30 16:42:32.695986 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:32.694343 5052 patch_prober.go:29] interesting pod/e2e-ait-f654570d-openshift-default-76cdfc5fcf-kwdnz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:42:32.695986 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:32.694411 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-f654570d-openshift-default-76cdfc5fcf-kwdnz" podUID="f6092ad4-fa04-4813-aebb-655353769fc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:42:32.717604 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:32.696557 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30dbe70e-554d-4f2c-a7e9-80546a168c30-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a0777a0c-69644576f5-4t4vr\" (UID: \"30dbe70e-554d-4f2c-a7e9-80546a168c30\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a0777a0c-69644576f5-4t4vr" Oct 30 16:42:32.717604 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:32.698390 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6092ad4-fa04-4813-aebb-655353769fc1" path="/var/lib/kubelet/pods/f6092ad4-fa04-4813-aebb-655353769fc1/volumes" Oct 30 16:42:32.751192 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:32.747317 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmjb2\" (UniqueName: \"kubernetes.io/projected/43d9d192-2f2b-4a26-8ecc-fe143833e3b3-kube-api-access-qmjb2\") pod \"payload-processing-e2e-ait-a0777a0c-85f5c98494-2896v\" (UID: \"43d9d192-2f2b-4a26-8ecc-fe143833e3b3\") " pod="openshift-ingress/payload-processing-e2e-ait-a0777a0c-85f5c98494-2896v" Oct 30 16:42:32.751192 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:32.749268 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5xsh\" (UniqueName: \"kubernetes.io/projected/30dbe70e-554d-4f2c-a7e9-80546a168c30-kube-api-access-f5xsh\") pod \"payload-pre-processing-e2e-ait-a0777a0c-69644576f5-4t4vr\" (UID: \"30dbe70e-554d-4f2c-a7e9-80546a168c30\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a0777a0c-69644576f5-4t4vr" Oct 30 16:42:32.804601 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:32.798944 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a0777a0c-85f5c98494-2896v" Oct 30 16:42:32.819147 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:32.818912 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a0777a0c-69644576f5-4t4vr" Oct 30 16:42:33.279338 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:33.279310 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a0777a0c-85f5c98494-2896v"] Oct 30 16:42:33.281175 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:42:33.281146 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43d9d192_2f2b_4a26_8ecc_fe143833e3b3.slice/crio-a44f88dc9df3a7f3aed8e9f1590aa685bb28f1a4ae4730ad213e82cfc16bb6b2 WatchSource:0}: Error finding container a44f88dc9df3a7f3aed8e9f1590aa685bb28f1a4ae4730ad213e82cfc16bb6b2: Status 404 returned error can't find the container with id a44f88dc9df3a7f3aed8e9f1590aa685bb28f1a4ae4730ad213e82cfc16bb6b2 Oct 30 16:42:33.304179 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:33.304156 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a0777a0c-69644576f5-4t4vr"] Oct 30 16:42:33.306652 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:42:33.306627 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30dbe70e_554d_4f2c_a7e9_80546a168c30.slice/crio-c4b4d51703489d1274e686878fafc5689ab564a276f2e14f3b36ec3412d54ec0 WatchSource:0}: Error finding container c4b4d51703489d1274e686878fafc5689ab564a276f2e14f3b36ec3412d54ec0: Status 404 returned error can't find the container with id c4b4d51703489d1274e686878fafc5689ab564a276f2e14f3b36ec3412d54ec0 Oct 30 16:42:33.412142 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:33.412102 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a0777a0c-85f5c98494-2896v" event={"ID":"43d9d192-2f2b-4a26-8ecc-fe143833e3b3","Type":"ContainerStarted","Data":"fa0f0dd3bc98cd300523227c6c6727d8d63b6860b3dd46ca5190da197b21cb97"} Oct 30 16:42:33.412308 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:33.412172 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a0777a0c-85f5c98494-2896v" event={"ID":"43d9d192-2f2b-4a26-8ecc-fe143833e3b3","Type":"ContainerStarted","Data":"a44f88dc9df3a7f3aed8e9f1590aa685bb28f1a4ae4730ad213e82cfc16bb6b2"} Oct 30 16:42:33.412308 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:33.412203 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-a0777a0c-85f5c98494-2896v" Oct 30 16:42:33.414526 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:33.414501 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a0777a0c-69644576f5-4t4vr" event={"ID":"30dbe70e-554d-4f2c-a7e9-80546a168c30","Type":"ContainerStarted","Data":"c4b4d51703489d1274e686878fafc5689ab564a276f2e14f3b36ec3412d54ec0"} Oct 30 16:42:33.460295 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:33.460230 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-a0777a0c-85f5c98494-2896v" podStartSLOduration=1.460210548 podStartE2EDuration="1.460210548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:42:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:42:33.454077688 +0000 UTC m=+1308.981412910" watchObservedRunningTime="2026-10-30 16:42:33.460210548 +0000 UTC m=+1308.987545768" Oct 30 16:42:34.261604 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:34.258932 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cdwcj"] Oct 30 16:42:34.501016 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:34.500982 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e4aca49c-c46697f47-wnm2p"] Oct 30 16:42:34.509528 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:34.509503 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e4aca49c-c46697f47-wnm2p" Oct 30 16:42:34.558610 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:34.557804 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-e4aca49c-dockercfg-7dds4\"" Oct 30 16:42:34.558610 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:34.557866 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-e4aca49c\"" Oct 30 16:42:34.844175 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:34.731995 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhm5k\" (UniqueName: \"kubernetes.io/projected/6725ae7a-0765-4505-976a-f265e4c026c8-kube-api-access-hhm5k\") pod \"payload-pre-processing-e2e-mt-e4aca49c-c46697f47-wnm2p\" (UID: \"6725ae7a-0765-4505-976a-f265e4c026c8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e4aca49c-c46697f47-wnm2p" Oct 30 16:42:34.844175 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:34.765026 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6725ae7a-0765-4505-976a-f265e4c026c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e4aca49c-c46697f47-wnm2p\" (UID: \"6725ae7a-0765-4505-976a-f265e4c026c8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e4aca49c-c46697f47-wnm2p" Oct 30 16:42:34.907666 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:34.904228 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6725ae7a-0765-4505-976a-f265e4c026c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e4aca49c-c46697f47-wnm2p\" (UID: \"6725ae7a-0765-4505-976a-f265e4c026c8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e4aca49c-c46697f47-wnm2p" Oct 30 16:42:34.907666 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:34.904404 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhm5k\" (UniqueName: \"kubernetes.io/projected/6725ae7a-0765-4505-976a-f265e4c026c8-kube-api-access-hhm5k\") pod \"payload-pre-processing-e2e-mt-e4aca49c-c46697f47-wnm2p\" (UID: \"6725ae7a-0765-4505-976a-f265e4c026c8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e4aca49c-c46697f47-wnm2p" Oct 30 16:42:34.907666 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:34.905692 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6725ae7a-0765-4505-976a-f265e4c026c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e4aca49c-c46697f47-wnm2p\" (UID: \"6725ae7a-0765-4505-976a-f265e4c026c8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e4aca49c-c46697f47-wnm2p" Oct 30 16:42:35.003784 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:35.003747 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhm5k\" (UniqueName: \"kubernetes.io/projected/6725ae7a-0765-4505-976a-f265e4c026c8-kube-api-access-hhm5k\") pod \"payload-pre-processing-e2e-mt-e4aca49c-c46697f47-wnm2p\" (UID: \"6725ae7a-0765-4505-976a-f265e4c026c8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e4aca49c-c46697f47-wnm2p" Oct 30 16:42:35.022203 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:35.022173 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-a0777a0c-69644576f5-4t4vr" Oct 30 16:42:35.022377 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:35.022231 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a0777a0c-69644576f5-4t4vr" event={"ID":"30dbe70e-554d-4f2c-a7e9-80546a168c30","Type":"ContainerStarted","Data":"a495ab0430d649ea406835bf02e5fedf37aa0402e42b741d2ffd673dfe68ad99"} Oct 30 16:42:35.022377 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:35.022267 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e4aca49c-c46697f47-wnm2p"] Oct 30 16:42:35.022377 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:35.022300 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e4aca49c-7795f84896-nql4z"] Oct 30 16:42:35.027416 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:35.027388 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e4aca49c-7795f84896-nql4z"] Oct 30 16:42:35.027548 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:35.027485 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e4aca49c-7795f84896-nql4z" Oct 30 16:42:35.068647 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:35.068557 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-a0777a0c-69644576f5-4t4vr" podStartSLOduration=3.068540625 podStartE2EDuration="3.068540625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:42:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:42:35.066383513 +0000 UTC m=+1310.593718726" watchObservedRunningTime="2026-10-30 16:42:35.068540625 +0000 UTC m=+1310.595875843" Oct 30 16:42:35.117803 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:35.117728 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef0b2556-42cf-4366-a753-68f881b6a02b-plugins-config-volume\") pod \"payload-processing-e2e-mt-e4aca49c-7795f84896-nql4z\" (UID: \"ef0b2556-42cf-4366-a753-68f881b6a02b\") " pod="openshift-ingress/payload-processing-e2e-mt-e4aca49c-7795f84896-nql4z" Oct 30 16:42:35.117979 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:35.117959 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcff9\" (UniqueName: \"kubernetes.io/projected/ef0b2556-42cf-4366-a753-68f881b6a02b-kube-api-access-zcff9\") pod \"payload-processing-e2e-mt-e4aca49c-7795f84896-nql4z\" (UID: \"ef0b2556-42cf-4366-a753-68f881b6a02b\") " pod="openshift-ingress/payload-processing-e2e-mt-e4aca49c-7795f84896-nql4z" Oct 30 16:42:35.132308 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:35.132257 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e4aca49c-c46697f47-wnm2p" Oct 30 16:42:35.239738 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:35.230308 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcff9\" (UniqueName: \"kubernetes.io/projected/ef0b2556-42cf-4366-a753-68f881b6a02b-kube-api-access-zcff9\") pod \"payload-processing-e2e-mt-e4aca49c-7795f84896-nql4z\" (UID: \"ef0b2556-42cf-4366-a753-68f881b6a02b\") " pod="openshift-ingress/payload-processing-e2e-mt-e4aca49c-7795f84896-nql4z" Oct 30 16:42:35.239738 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:35.230492 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef0b2556-42cf-4366-a753-68f881b6a02b-plugins-config-volume\") pod \"payload-processing-e2e-mt-e4aca49c-7795f84896-nql4z\" (UID: \"ef0b2556-42cf-4366-a753-68f881b6a02b\") " pod="openshift-ingress/payload-processing-e2e-mt-e4aca49c-7795f84896-nql4z" Oct 30 16:42:35.239738 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:35.232249 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef0b2556-42cf-4366-a753-68f881b6a02b-plugins-config-volume\") pod \"payload-processing-e2e-mt-e4aca49c-7795f84896-nql4z\" (UID: \"ef0b2556-42cf-4366-a753-68f881b6a02b\") " pod="openshift-ingress/payload-processing-e2e-mt-e4aca49c-7795f84896-nql4z" Oct 30 16:42:35.264656 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:35.264176 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcff9\" (UniqueName: \"kubernetes.io/projected/ef0b2556-42cf-4366-a753-68f881b6a02b-kube-api-access-zcff9\") pod \"payload-processing-e2e-mt-e4aca49c-7795f84896-nql4z\" (UID: \"ef0b2556-42cf-4366-a753-68f881b6a02b\") " pod="openshift-ingress/payload-processing-e2e-mt-e4aca49c-7795f84896-nql4z" Oct 30 16:42:35.353773 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:35.353480 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e4aca49c-7795f84896-nql4z" Oct 30 16:42:35.604571 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:35.604540 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e4aca49c-c46697f47-wnm2p"] Oct 30 16:42:35.835630 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:35.826510 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e4aca49c-7795f84896-nql4z"] Oct 30 16:42:36.053542 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:36.053158 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e4aca49c-7795f84896-nql4z" event={"ID":"ef0b2556-42cf-4366-a753-68f881b6a02b","Type":"ContainerStarted","Data":"57d978b22cf062e66b3e16407487251dba9c23ce8e6280ca5f10b0dc23084992"} Oct 30 16:42:36.069074 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:36.062415 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e4aca49c-c46697f47-wnm2p" event={"ID":"6725ae7a-0765-4505-976a-f265e4c026c8","Type":"ContainerStarted","Data":"8d44c0d91d6642fee75db890f4d51387fbde261d9b9468c32c1d64dd7872e600"} Oct 30 16:42:37.192733 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:37.192689 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/distinct-2-w4-263f00-kserve-8464897cf5-q8ck8" podUID="15ed43ef-5cdd-489a-a70e-063239822ff3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 30 16:42:37.193716 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:37.193684 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx" podUID="bd662f3a-659f-477c-b8d5-94c64288c0c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:42:37.242640 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:37.242354 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e4aca49c-c46697f47-wnm2p" event={"ID":"6725ae7a-0765-4505-976a-f265e4c026c8","Type":"ContainerStarted","Data":"6e8f844d38bf2158d7a567b292f55186ba2d96b21e44aa6967783d2c328fb24e"} Oct 30 16:42:37.252604 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:37.243713 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-e4aca49c-c46697f47-wnm2p" Oct 30 16:42:37.300605 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:37.278284 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e4aca49c-7795f84896-nql4z" event={"ID":"ef0b2556-42cf-4366-a753-68f881b6a02b","Type":"ContainerStarted","Data":"b100bf2b7cb110867b10f5ec44794e7c5be114f7fd85ff978be9c9c0951eaa38"} Oct 30 16:42:37.300605 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:37.279241 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-e4aca49c-7795f84896-nql4z" Oct 30 16:42:37.369311 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:37.369237 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-e4aca49c-7795f84896-nql4z" podStartSLOduration=3.369216534 podStartE2EDuration="3.369216534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:42:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:42:37.36541094 +0000 UTC m=+1312.892746162" watchObservedRunningTime="2026-10-30 16:42:37.369216534 +0000 UTC m=+1312.896551755" Oct 30 16:42:37.370894 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:37.370845 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-e4aca49c-c46697f47-wnm2p" podStartSLOduration=3.370832707 podStartE2EDuration="3.370832707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:42:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:42:37.304851556 +0000 UTC m=+1312.832186776" watchObservedRunningTime="2026-10-30 16:42:37.370832707 +0000 UTC m=+1312.898167942" Oct 30 16:42:37.381702 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:37.377704 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-750dcb/distinct-w2-750dcb-kserve-86bbdfcbb-bpn77" podUID="c45190af-d795-4e13-8f2f-670b2c597658" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:42:37.412769 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:37.412708 5052 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-d693d0/distinct-2-w6-d693d0-kserve-6498fdc474-dpbjj" podUID="4b98b8ff-72cd-4ddb-a4b8-67078162e611" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:42:37.532668 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:37.530995 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/distinct-w5-967fac-kserve-69cbcccd8b-b85bn" podUID="8f6b9354-378f-4451-9ff7-be7f7fec12f2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:42:37.546603 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:37.537818 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d693d0/distinct-2-w6-d693d0-kserve-6498fdc474-dpbjj" podUID="4b98b8ff-72cd-4ddb-a4b8-67078162e611" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:42:38.200605 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:38.195663 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx" podUID="bd662f3a-659f-477c-b8d5-94c64288c0c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:42:38.200605 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:38.196068 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/distinct-2-w4-263f00-kserve-8464897cf5-q8ck8" podUID="15ed43ef-5cdd-489a-a70e-063239822ff3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 30 16:42:38.200605 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:38.196655 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-967fac/distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx" Oct 30 16:42:38.200605 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:38.196715 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-263f00/distinct-2-w4-263f00-kserve-8464897cf5-q8ck8" Oct 30 16:42:38.439819 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:38.394279 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-750dcb/distinct-w2-750dcb-kserve-86bbdfcbb-bpn77" podUID="c45190af-d795-4e13-8f2f-670b2c597658" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded" Oct 30 16:42:38.439819 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:38.395810 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-750dcb/distinct-w2-750dcb-kserve-86bbdfcbb-bpn77" Oct 30 16:42:38.584634 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:38.573411 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d693d0/distinct-2-w6-d693d0-kserve-6498fdc474-dpbjj" podUID="4b98b8ff-72cd-4ddb-a4b8-67078162e611" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Oct 30 16:42:38.665566 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:38.653695 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d693d0/distinct-2-w6-d693d0-kserve-6498fdc474-dpbjj" Oct 30 16:42:38.665566 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:38.654669 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/distinct-w5-967fac-kserve-69cbcccd8b-b85bn" podUID="8f6b9354-378f-4451-9ff7-be7f7fec12f2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:42:39.232460 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:39.231912 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/distinct-2-w4-263f00-kserve-8464897cf5-q8ck8" podUID="15ed43ef-5cdd-489a-a70e-063239822ff3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:42:39.237961 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:39.237921 5052 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-263f00/distinct-2-w4-263f00-kserve-8464897cf5-q8ck8" podUID="15ed43ef-5cdd-489a-a70e-063239822ff3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": context deadline exceeded" Oct 30 16:42:39.258498 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:39.258459 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx" podUID="bd662f3a-659f-477c-b8d5-94c64288c0c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:42:39.339606 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:39.333302 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-967fac/distinct-w5-967fac-kserve-69cbcccd8b-b85bn" Oct 30 16:42:39.397204 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:39.397163 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-750dcb/distinct-w2-750dcb-kserve-86bbdfcbb-bpn77" podUID="c45190af-d795-4e13-8f2f-670b2c597658" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:42:39.636663 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:39.607080 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d693d0/distinct-2-w6-d693d0-kserve-6498fdc474-dpbjj" podUID="4b98b8ff-72cd-4ddb-a4b8-67078162e611" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:42:39.696743 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:39.666546 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/distinct-w5-967fac-kserve-69cbcccd8b-b85bn" podUID="8f6b9354-378f-4451-9ff7-be7f7fec12f2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:42:40.120604 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:40.120199 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d693d0/distinct-2-w6-d693d0-kserve-6498fdc474-dpbjj" Oct 30 16:42:40.197172 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:40.197138 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-750dcb/distinct-w2-750dcb-kserve-86bbdfcbb-bpn77" Oct 30 16:42:40.250290 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:40.250238 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/distinct-2-w4-263f00-kserve-8464897cf5-q8ck8" podUID="15ed43ef-5cdd-489a-a70e-063239822ff3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 30 16:42:40.277693 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:40.268234 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx" podUID="bd662f3a-659f-477c-b8d5-94c64288c0c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:42:40.771881 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:40.771840 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/distinct-w5-967fac-kserve-69cbcccd8b-b85bn" podUID="8f6b9354-378f-4451-9ff7-be7f7fec12f2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 16:42:40.988073 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:40.987110 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-967fac/distinct-w5-967fac-kserve-69cbcccd8b-b85bn" Oct 30 16:42:41.118683 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:41.085006 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-263f00/distinct-2-w4-263f00-kserve-8464897cf5-q8ck8" Oct 30 16:42:41.237609 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:41.230333 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-967fac/distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx" Oct 30 16:42:41.682603 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:41.681780 5052 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-t8b5p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:42:41.682603 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:41.681865 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-t8b5p" podUID="fcc7c9e1-99ea-4b97-9e95-2e23a00ac096" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:42:43.796854 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:43.796819 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-a475e0dd-577fdb5999-dwtjd"] Oct 30 16:42:45.013638 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:45.013592 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-a0777a0c-85f5c98494-2896v" Oct 30 16:42:46.065327 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:46.065295 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-a0777a0c-69644576f5-4t4vr" Oct 30 16:42:48.104155 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:48.104120 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bvq2v"] Oct 30 16:42:48.110544 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:48.110503 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bvq2v" Oct 30 16:42:48.143662 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:48.143629 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bvq2v"] Oct 30 16:42:48.240853 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:48.240819 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cb92868-e7dd-4783-a4c1-2d2cd75b70fb-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-bvq2v\" (UID: \"2cb92868-e7dd-4783-a4c1-2d2cd75b70fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bvq2v" Oct 30 16:42:48.240853 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:48.240854 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwdwh\" (UniqueName: \"kubernetes.io/projected/2cb92868-e7dd-4783-a4c1-2d2cd75b70fb-kube-api-access-gwdwh\") pod \"maas-default-gateway-openshift-default-76f4876667-bvq2v\" (UID: \"2cb92868-e7dd-4783-a4c1-2d2cd75b70fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bvq2v" Oct 30 16:42:48.241076 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:48.240884 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cb92868-e7dd-4783-a4c1-2d2cd75b70fb-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-bvq2v\" (UID: \"2cb92868-e7dd-4783-a4c1-2d2cd75b70fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bvq2v" Oct 30 16:42:48.241076 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:48.240940 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cb92868-e7dd-4783-a4c1-2d2cd75b70fb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-bvq2v\" (UID: \"2cb92868-e7dd-4783-a4c1-2d2cd75b70fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bvq2v" Oct 30 16:42:48.241076 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:48.241011 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cb92868-e7dd-4783-a4c1-2d2cd75b70fb-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-bvq2v\" (UID: \"2cb92868-e7dd-4783-a4c1-2d2cd75b70fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bvq2v" Oct 30 16:42:48.241076 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:48.241034 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cb92868-e7dd-4783-a4c1-2d2cd75b70fb-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-bvq2v\" (UID: \"2cb92868-e7dd-4783-a4c1-2d2cd75b70fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bvq2v" Oct 30 16:42:48.241076 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:48.241065 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cb92868-e7dd-4783-a4c1-2d2cd75b70fb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-bvq2v\" (UID: \"2cb92868-e7dd-4783-a4c1-2d2cd75b70fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bvq2v" Oct 30 16:42:48.241254 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:48.241220 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cb92868-e7dd-4783-a4c1-2d2cd75b70fb-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-bvq2v\" (UID: \"2cb92868-e7dd-4783-a4c1-2d2cd75b70fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bvq2v" Oct 30 16:42:48.241358 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:48.241343 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cb92868-e7dd-4783-a4c1-2d2cd75b70fb-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-bvq2v\" (UID: \"2cb92868-e7dd-4783-a4c1-2d2cd75b70fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bvq2v" Oct 30 16:42:48.342325 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:48.342291 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cb92868-e7dd-4783-a4c1-2d2cd75b70fb-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-bvq2v\" (UID: \"2cb92868-e7dd-4783-a4c1-2d2cd75b70fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bvq2v" Oct 30 16:42:48.342325 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:48.342326 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwdwh\" (UniqueName: \"kubernetes.io/projected/2cb92868-e7dd-4783-a4c1-2d2cd75b70fb-kube-api-access-gwdwh\") pod \"maas-default-gateway-openshift-default-76f4876667-bvq2v\" (UID: \"2cb92868-e7dd-4783-a4c1-2d2cd75b70fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bvq2v" Oct 30 16:42:48.342549 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:48.342356 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cb92868-e7dd-4783-a4c1-2d2cd75b70fb-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-bvq2v\" (UID: \"2cb92868-e7dd-4783-a4c1-2d2cd75b70fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bvq2v" Oct 30 16:42:48.342549 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:48.342381 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cb92868-e7dd-4783-a4c1-2d2cd75b70fb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-bvq2v\" (UID: \"2cb92868-e7dd-4783-a4c1-2d2cd75b70fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bvq2v" Oct 30 16:42:48.342549 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:48.342418 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cb92868-e7dd-4783-a4c1-2d2cd75b70fb-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-bvq2v\" (UID: \"2cb92868-e7dd-4783-a4c1-2d2cd75b70fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bvq2v" Oct 30 16:42:48.342549 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:48.342449 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cb92868-e7dd-4783-a4c1-2d2cd75b70fb-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-bvq2v\" (UID: \"2cb92868-e7dd-4783-a4c1-2d2cd75b70fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bvq2v" Oct 30 16:42:48.342549 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:48.342500 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cb92868-e7dd-4783-a4c1-2d2cd75b70fb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-bvq2v\" (UID: \"2cb92868-e7dd-4783-a4c1-2d2cd75b70fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bvq2v" Oct 30 16:42:48.342852 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:48.342597 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cb92868-e7dd-4783-a4c1-2d2cd75b70fb-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-bvq2v\" (UID: \"2cb92868-e7dd-4783-a4c1-2d2cd75b70fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bvq2v" Oct 30 16:42:48.342852 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:48.342710 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cb92868-e7dd-4783-a4c1-2d2cd75b70fb-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-bvq2v\" (UID: \"2cb92868-e7dd-4783-a4c1-2d2cd75b70fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bvq2v" Oct 30 16:42:48.342852 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:48.342765 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cb92868-e7dd-4783-a4c1-2d2cd75b70fb-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-bvq2v\" (UID: \"2cb92868-e7dd-4783-a4c1-2d2cd75b70fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bvq2v" Oct 30 16:42:48.343008 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:48.342920 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cb92868-e7dd-4783-a4c1-2d2cd75b70fb-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-bvq2v\" (UID: \"2cb92868-e7dd-4783-a4c1-2d2cd75b70fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bvq2v" Oct 30 16:42:48.343064 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:48.343017 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cb92868-e7dd-4783-a4c1-2d2cd75b70fb-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-bvq2v\" (UID: \"2cb92868-e7dd-4783-a4c1-2d2cd75b70fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bvq2v" Oct 30 16:42:48.343064 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:48.343029 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cb92868-e7dd-4783-a4c1-2d2cd75b70fb-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-bvq2v\" (UID: \"2cb92868-e7dd-4783-a4c1-2d2cd75b70fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bvq2v" Oct 30 16:42:48.343349 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:48.343322 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cb92868-e7dd-4783-a4c1-2d2cd75b70fb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-bvq2v\" (UID: \"2cb92868-e7dd-4783-a4c1-2d2cd75b70fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bvq2v" Oct 30 16:42:48.344865 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:48.344844 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cb92868-e7dd-4783-a4c1-2d2cd75b70fb-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-bvq2v\" (UID: \"2cb92868-e7dd-4783-a4c1-2d2cd75b70fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bvq2v" Oct 30 16:42:48.345209 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:48.345188 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cb92868-e7dd-4783-a4c1-2d2cd75b70fb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-bvq2v\" (UID: \"2cb92868-e7dd-4783-a4c1-2d2cd75b70fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bvq2v" Oct 30 16:42:48.364963 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:48.364915 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cb92868-e7dd-4783-a4c1-2d2cd75b70fb-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-bvq2v\" (UID: \"2cb92868-e7dd-4783-a4c1-2d2cd75b70fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bvq2v" Oct 30 16:42:48.365083 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:48.365061 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwdwh\" (UniqueName: \"kubernetes.io/projected/2cb92868-e7dd-4783-a4c1-2d2cd75b70fb-kube-api-access-gwdwh\") pod \"maas-default-gateway-openshift-default-76f4876667-bvq2v\" (UID: \"2cb92868-e7dd-4783-a4c1-2d2cd75b70fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bvq2v" Oct 30 16:42:48.447427 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:48.445896 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bvq2v" Oct 30 16:42:48.797965 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:48.797925 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a475e0dd-577fdb5999-dwtjd" podUID="4ee414b4-410f-4506-a81d-26c21b331674" containerName="payload-pre-processing" containerID="cri-o://b727136dd7927467f7ee6f3624c501770139b6689e65d8ddd11d6c962a30b87f" gracePeriod=25 Oct 30 16:42:48.841711 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:48.841677 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bvq2v"] Oct 30 16:42:48.847398 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:42:48.847367 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 16:42:48.859565 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:42:48.859539 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cb92868_e7dd_4783_a4c1_2d2cd75b70fb.slice/crio-11474e3cd12a86de143f1f711f41aea666bdbd87eb52736228384c1773984673 WatchSource:0}: Error finding container 11474e3cd12a86de143f1f711f41aea666bdbd87eb52736228384c1773984673: Status 404 returned error can't find the container with id 11474e3cd12a86de143f1f711f41aea666bdbd87eb52736228384c1773984673 Oct 30 16:42:48.904114 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:48.904079 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:42:48.904236 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:48.904164 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:42:48.904236 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:48.904211 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:42:49.087264 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:49.087232 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a475e0dd-577fdb5999-dwtjd" Oct 30 16:42:49.161181 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:49.161151 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ee414b4-410f-4506-a81d-26c21b331674-kube-api-access-4fv97\" (UniqueName: \"kubernetes.io/projected/4ee414b4-410f-4506-a81d-26c21b331674-kube-api-access-4fv97\") pod \"4ee414b4-410f-4506-a81d-26c21b331674\" (UID: \"4ee414b4-410f-4506-a81d-26c21b331674\") " Oct 30 16:42:49.161723 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:49.161318 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ee414b4-410f-4506-a81d-26c21b331674-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ee414b4-410f-4506-a81d-26c21b331674-plugins-config-volume\") pod \"4ee414b4-410f-4506-a81d-26c21b331674\" (UID: \"4ee414b4-410f-4506-a81d-26c21b331674\") " Oct 30 16:42:49.161828 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:49.161800 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ee414b4-410f-4506-a81d-26c21b331674-plugins-config-volume" pod "4ee414b4-410f-4506-a81d-26c21b331674" (UID: "4ee414b4-410f-4506-a81d-26c21b331674"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:42:49.162007 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:49.161984 5052 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ee414b4-410f-4506-a81d-26c21b331674-plugins-config-volume\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:42:49.164151 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:49.164129 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ee414b4-410f-4506-a81d-26c21b331674-kube-api-access-4fv97" pod "4ee414b4-410f-4506-a81d-26c21b331674" (UID: "4ee414b4-410f-4506-a81d-26c21b331674"). InnerVolumeSpecName "kube-api-access-4fv97". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:42:49.263852 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:49.263818 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fv97\" (UniqueName: \"kubernetes.io/projected/4ee414b4-410f-4506-a81d-26c21b331674-kube-api-access-4fv97\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:42:49.779125 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:49.779081 5052 generic.go:356] "Generic (PLEG): container finished" podID="4ee414b4-410f-4506-a81d-26c21b331674" containerID="b727136dd7927467f7ee6f3624c501770139b6689e65d8ddd11d6c962a30b87f" exitCode=0 Oct 30 16:42:49.779336 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:49.779141 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a475e0dd-577fdb5999-dwtjd" Oct 30 16:42:49.779336 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:49.779149 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a475e0dd-577fdb5999-dwtjd" event={"ID":"4ee414b4-410f-4506-a81d-26c21b331674","Type":"ContainerDied","Data":"b727136dd7927467f7ee6f3624c501770139b6689e65d8ddd11d6c962a30b87f"} Oct 30 16:42:49.780225 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:49.780198 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a475e0dd-577fdb5999-dwtjd" event={"ID":"4ee414b4-410f-4506-a81d-26c21b331674","Type":"ContainerDied","Data":"553131b7c42e93fd33047a3245ab857e6fc77f2831dcb282e4b93f51530bf3e4"} Oct 30 16:42:49.780371 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:49.780246 5052 scope.go:122] "RemoveContainer" containerID="b727136dd7927467f7ee6f3624c501770139b6689e65d8ddd11d6c962a30b87f" Oct 30 16:42:49.781216 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:49.781195 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bvq2v" event={"ID":"2cb92868-e7dd-4783-a4c1-2d2cd75b70fb","Type":"ContainerStarted","Data":"e8e58497167dbf6f3aadf3c42ec8cbdd05dc813bb74b2b262bc7d08d16c82d43"} Oct 30 16:42:49.781335 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:49.781240 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bvq2v" event={"ID":"2cb92868-e7dd-4783-a4c1-2d2cd75b70fb","Type":"ContainerStarted","Data":"11474e3cd12a86de143f1f711f41aea666bdbd87eb52736228384c1773984673"} Oct 30 16:42:49.793018 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:49.793002 5052 scope.go:122] "RemoveContainer" containerID="b727136dd7927467f7ee6f3624c501770139b6689e65d8ddd11d6c962a30b87f" Oct 30 16:42:49.793265 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:42:49.793244 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b727136dd7927467f7ee6f3624c501770139b6689e65d8ddd11d6c962a30b87f\": container with ID starting with b727136dd7927467f7ee6f3624c501770139b6689e65d8ddd11d6c962a30b87f not found: ID does not exist" containerID="b727136dd7927467f7ee6f3624c501770139b6689e65d8ddd11d6c962a30b87f" Oct 30 16:42:49.793317 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:49.793275 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b727136dd7927467f7ee6f3624c501770139b6689e65d8ddd11d6c962a30b87f"} err="failed to get container status \"b727136dd7927467f7ee6f3624c501770139b6689e65d8ddd11d6c962a30b87f\": rpc error: code = NotFound desc = could not find container \"b727136dd7927467f7ee6f3624c501770139b6689e65d8ddd11d6c962a30b87f\": container with ID starting with b727136dd7927467f7ee6f3624c501770139b6689e65d8ddd11d6c962a30b87f not found: ID does not exist" Oct 30 16:42:49.854618 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:49.853287 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-a475e0dd-577fdb5999-dwtjd"] Oct 30 16:42:49.863562 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:49.863530 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-a475e0dd-577fdb5999-dwtjd"] Oct 30 16:42:49.902521 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:49.902472 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bvq2v" podStartSLOduration=1.902457567 podStartE2EDuration="1.902457567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:42:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:42:49.899029153 +0000 UTC m=+1325.426364375" watchObservedRunningTime="2026-10-30 16:42:49.902457567 +0000 UTC m=+1325.429792789" Oct 30 16:42:49.969653 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:49.969624 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a0777a0c-5d898fd794-dt7pf"] Oct 30 16:42:49.970373 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:49.970359 5052 cpu_manager.go:414] "containerMap: removing container" podUID="4ee414b4-410f-4506-a81d-26c21b331674" containerName="payload-pre-processing" Oct 30 16:42:49.970416 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:49.970374 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ee414b4-410f-4506-a81d-26c21b331674" containerName="payload-pre-processing" Oct 30 16:42:49.970476 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:49.970467 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ee414b4-410f-4506-a81d-26c21b331674" containerName="payload-pre-processing" Oct 30 16:42:49.975433 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:49.975413 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a0777a0c-5d898fd794-dt7pf" Oct 30 16:42:49.994817 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:49.994779 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-a0777a0c-serving-cert\"" Oct 30 16:42:49.995361 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:49.995157 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a0777a0c-5d898fd794-dt7pf"] Oct 30 16:42:50.076475 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:50.076380 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5sz2\" (UniqueName: \"kubernetes.io/projected/536a4ad6-b9d8-40c9-bd0a-ad09c8a2a015-kube-api-access-p5sz2\") pod \"maas-api-e2e-ait-a0777a0c-5d898fd794-dt7pf\" (UID: \"536a4ad6-b9d8-40c9-bd0a-ad09c8a2a015\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a0777a0c-5d898fd794-dt7pf" Oct 30 16:42:50.076671 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:50.076646 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/536a4ad6-b9d8-40c9-bd0a-ad09c8a2a015-metrics-certs\") pod \"maas-api-e2e-ait-a0777a0c-5d898fd794-dt7pf\" (UID: \"536a4ad6-b9d8-40c9-bd0a-ad09c8a2a015\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a0777a0c-5d898fd794-dt7pf" Oct 30 16:42:50.076822 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:50.076800 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/536a4ad6-b9d8-40c9-bd0a-ad09c8a2a015-openshift-service-ca\") pod \"maas-api-e2e-ait-a0777a0c-5d898fd794-dt7pf\" (UID: \"536a4ad6-b9d8-40c9-bd0a-ad09c8a2a015\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a0777a0c-5d898fd794-dt7pf" Oct 30 16:42:50.076967 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:50.076950 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/536a4ad6-b9d8-40c9-bd0a-ad09c8a2a015-maas-api-tls\") pod \"maas-api-e2e-ait-a0777a0c-5d898fd794-dt7pf\" (UID: \"536a4ad6-b9d8-40c9-bd0a-ad09c8a2a015\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a0777a0c-5d898fd794-dt7pf" Oct 30 16:42:50.178381 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:50.178342 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/536a4ad6-b9d8-40c9-bd0a-ad09c8a2a015-maas-api-tls\") pod \"maas-api-e2e-ait-a0777a0c-5d898fd794-dt7pf\" (UID: \"536a4ad6-b9d8-40c9-bd0a-ad09c8a2a015\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a0777a0c-5d898fd794-dt7pf" Oct 30 16:42:50.178851 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:50.178433 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5sz2\" (UniqueName: \"kubernetes.io/projected/536a4ad6-b9d8-40c9-bd0a-ad09c8a2a015-kube-api-access-p5sz2\") pod \"maas-api-e2e-ait-a0777a0c-5d898fd794-dt7pf\" (UID: \"536a4ad6-b9d8-40c9-bd0a-ad09c8a2a015\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a0777a0c-5d898fd794-dt7pf" Oct 30 16:42:50.178851 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:50.178657 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/536a4ad6-b9d8-40c9-bd0a-ad09c8a2a015-metrics-certs\") pod \"maas-api-e2e-ait-a0777a0c-5d898fd794-dt7pf\" (UID: \"536a4ad6-b9d8-40c9-bd0a-ad09c8a2a015\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a0777a0c-5d898fd794-dt7pf" Oct 30 16:42:50.178851 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:50.178778 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/536a4ad6-b9d8-40c9-bd0a-ad09c8a2a015-openshift-service-ca\") pod \"maas-api-e2e-ait-a0777a0c-5d898fd794-dt7pf\" (UID: \"536a4ad6-b9d8-40c9-bd0a-ad09c8a2a015\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a0777a0c-5d898fd794-dt7pf" Oct 30 16:42:50.179355 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:50.179323 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/536a4ad6-b9d8-40c9-bd0a-ad09c8a2a015-openshift-service-ca\") pod \"maas-api-e2e-ait-a0777a0c-5d898fd794-dt7pf\" (UID: \"536a4ad6-b9d8-40c9-bd0a-ad09c8a2a015\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a0777a0c-5d898fd794-dt7pf" Oct 30 16:42:50.180924 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:50.180893 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/536a4ad6-b9d8-40c9-bd0a-ad09c8a2a015-maas-api-tls\") pod \"maas-api-e2e-ait-a0777a0c-5d898fd794-dt7pf\" (UID: \"536a4ad6-b9d8-40c9-bd0a-ad09c8a2a015\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a0777a0c-5d898fd794-dt7pf" Oct 30 16:42:50.181057 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:50.181035 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/536a4ad6-b9d8-40c9-bd0a-ad09c8a2a015-metrics-certs\") pod \"maas-api-e2e-ait-a0777a0c-5d898fd794-dt7pf\" (UID: \"536a4ad6-b9d8-40c9-bd0a-ad09c8a2a015\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a0777a0c-5d898fd794-dt7pf" Oct 30 16:42:50.201461 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:50.201432 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5sz2\" (UniqueName: \"kubernetes.io/projected/536a4ad6-b9d8-40c9-bd0a-ad09c8a2a015-kube-api-access-p5sz2\") pod \"maas-api-e2e-ait-a0777a0c-5d898fd794-dt7pf\" (UID: \"536a4ad6-b9d8-40c9-bd0a-ad09c8a2a015\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a0777a0c-5d898fd794-dt7pf" Oct 30 16:42:50.293147 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:50.291874 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a0777a0c-5d898fd794-dt7pf" Oct 30 16:42:50.475005 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:50.472500 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bvq2v" Oct 30 16:42:50.475005 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:50.472746 5052 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-bvq2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 30 16:42:50.475005 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:50.472795 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bvq2v" podUID="2cb92868-e7dd-4783-a4c1-2d2cd75b70fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 30 16:42:50.562129 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:50.561991 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-e4aca49c-c46697f47-wnm2p" Oct 30 16:42:50.572622 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:50.565784 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-e4aca49c-7795f84896-nql4z" Oct 30 16:42:50.603011 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:50.602979 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ee414b4-410f-4506-a81d-26c21b331674" path="/var/lib/kubelet/pods/4ee414b4-410f-4506-a81d-26c21b331674/volumes" Oct 30 16:42:50.799129 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:50.799099 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a0777a0c-5d898fd794-dt7pf"] Oct 30 16:42:50.800704 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:42:50.800680 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod536a4ad6_b9d8_40c9_bd0a_ad09c8a2a015.slice/crio-91ee4cb331d6d52790daad5502029ab9425870522f0f9bec72a06176373825a4 WatchSource:0}: Error finding container 91ee4cb331d6d52790daad5502029ab9425870522f0f9bec72a06176373825a4: Status 404 returned error can't find the container with id 91ee4cb331d6d52790daad5502029ab9425870522f0f9bec72a06176373825a4 Oct 30 16:42:51.449180 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:51.449151 5052 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-bvq2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 30 16:42:51.449661 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:51.449218 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bvq2v" podUID="2cb92868-e7dd-4783-a4c1-2d2cd75b70fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 30 16:42:51.800196 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:51.800099 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a0777a0c-5d898fd794-dt7pf" event={"ID":"536a4ad6-b9d8-40c9-bd0a-ad09c8a2a015","Type":"ContainerStarted","Data":"65c49ba7027225606927764745921156c6411201a3c796bad02a277e07ef4461"} Oct 30 16:42:51.800366 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:51.800212 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a0777a0c-5d898fd794-dt7pf" event={"ID":"536a4ad6-b9d8-40c9-bd0a-ad09c8a2a015","Type":"ContainerStarted","Data":"91ee4cb331d6d52790daad5502029ab9425870522f0f9bec72a06176373825a4"} Oct 30 16:42:51.800366 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:51.800345 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a0777a0c-5d898fd794-dt7pf" Oct 30 16:42:52.452674 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:52.450954 5052 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-bvq2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 30 16:42:52.452674 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:52.451017 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bvq2v" podUID="2cb92868-e7dd-4783-a4c1-2d2cd75b70fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 30 16:42:53.447972 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:53.447935 5052 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-bvq2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 30 16:42:53.448262 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:53.448235 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bvq2v" podUID="2cb92868-e7dd-4783-a4c1-2d2cd75b70fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 30 16:42:54.449564 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:54.449527 5052 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-bvq2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 30 16:42:54.450195 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:54.450163 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bvq2v" podUID="2cb92868-e7dd-4783-a4c1-2d2cd75b70fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 30 16:42:55.509660 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:55.509563 5052 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-bvq2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 30 16:42:55.510137 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:55.509686 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bvq2v" podUID="2cb92868-e7dd-4783-a4c1-2d2cd75b70fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 30 16:42:56.466604 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:56.453044 5052 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-bvq2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 30 16:42:56.466604 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:56.453107 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bvq2v" podUID="2cb92868-e7dd-4783-a4c1-2d2cd75b70fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 30 16:42:57.448296 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:57.448263 5052 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-bvq2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 30 16:42:57.448794 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:57.448331 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bvq2v" podUID="2cb92868-e7dd-4783-a4c1-2d2cd75b70fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 30 16:42:57.819360 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:57.818972 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a0777a0c-5d898fd794-dt7pf" Oct 30 16:42:57.873674 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:57.873605 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a0777a0c-5d898fd794-dt7pf" podStartSLOduration=8.456672632 podStartE2EDuration="8.873568586s" totalImagesPullingTime="416.895954ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:42:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:42:51.855090402 +0000 UTC m=+1327.382425622" watchObservedRunningTime="2026-10-30 16:42:57.873568586 +0000 UTC m=+1333.400903805" Oct 30 16:42:58.846827 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:58.846788 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bvq2v" Oct 30 16:42:58.848608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:58.848567 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bvq2v" Oct 30 16:42:58.850224 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:58.850194 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bvq2v" Oct 30 16:42:58.974765 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:58.974727 5052 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-l6dw5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:42:58.975042 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:42:58.975009 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-l6dw5" podUID="4b3c4341-826b-489d-81ae-14bf71548dd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:43:12.986149 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:12.986096 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cdwcj"] Oct 30 16:43:18.894261 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:43:18.894227 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 16:43:23.013570 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:23.013534 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a0777a0c-85f5c98494-2896v"] Oct 30 16:43:23.051783 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:23.051754 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a0777a0c-69644576f5-4t4vr"] Oct 30 16:43:26.485610 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:26.485558 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a0777a0c-5d898fd794-dt7pf"] Oct 30 16:43:26.486674 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:26.486637 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a0777a0c-5d898fd794-dt7pf" podUID="536a4ad6-b9d8-40c9-bd0a-ad09c8a2a015" containerName="maas-api" containerID="cri-o://65c49ba7027225606927764745921156c6411201a3c796bad02a277e07ef4461" gracePeriod=30 Oct 30 16:43:27.141766 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:27.141731 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a0777a0c-5d898fd794-dt7pf" Oct 30 16:43:27.345422 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:27.345330 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/536a4ad6-b9d8-40c9-bd0a-ad09c8a2a015-kube-api-access-p5sz2\" (UniqueName: \"kubernetes.io/projected/536a4ad6-b9d8-40c9-bd0a-ad09c8a2a015-kube-api-access-p5sz2\") pod \"536a4ad6-b9d8-40c9-bd0a-ad09c8a2a015\" (UID: \"536a4ad6-b9d8-40c9-bd0a-ad09c8a2a015\") " Oct 30 16:43:27.345422 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:27.345399 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/536a4ad6-b9d8-40c9-bd0a-ad09c8a2a015-metrics-certs\" (UniqueName: \"kubernetes.io/secret/536a4ad6-b9d8-40c9-bd0a-ad09c8a2a015-metrics-certs\") pod \"536a4ad6-b9d8-40c9-bd0a-ad09c8a2a015\" (UID: \"536a4ad6-b9d8-40c9-bd0a-ad09c8a2a015\") " Oct 30 16:43:27.345674 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:27.345451 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/536a4ad6-b9d8-40c9-bd0a-ad09c8a2a015-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/536a4ad6-b9d8-40c9-bd0a-ad09c8a2a015-maas-api-tls\") pod \"536a4ad6-b9d8-40c9-bd0a-ad09c8a2a015\" (UID: \"536a4ad6-b9d8-40c9-bd0a-ad09c8a2a015\") " Oct 30 16:43:27.345674 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:27.345619 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/536a4ad6-b9d8-40c9-bd0a-ad09c8a2a015-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/536a4ad6-b9d8-40c9-bd0a-ad09c8a2a015-openshift-service-ca\") pod \"536a4ad6-b9d8-40c9-bd0a-ad09c8a2a015\" (UID: \"536a4ad6-b9d8-40c9-bd0a-ad09c8a2a015\") " Oct 30 16:43:27.346825 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:27.346795 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/536a4ad6-b9d8-40c9-bd0a-ad09c8a2a015-openshift-service-ca" pod "536a4ad6-b9d8-40c9-bd0a-ad09c8a2a015" (UID: "536a4ad6-b9d8-40c9-bd0a-ad09c8a2a015"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:43:27.349334 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:27.349304 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/536a4ad6-b9d8-40c9-bd0a-ad09c8a2a015-metrics-certs" pod "536a4ad6-b9d8-40c9-bd0a-ad09c8a2a015" (UID: "536a4ad6-b9d8-40c9-bd0a-ad09c8a2a015"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:43:27.349479 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:27.349462 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/536a4ad6-b9d8-40c9-bd0a-ad09c8a2a015-kube-api-access-p5sz2" pod "536a4ad6-b9d8-40c9-bd0a-ad09c8a2a015" (UID: "536a4ad6-b9d8-40c9-bd0a-ad09c8a2a015"). InnerVolumeSpecName "kube-api-access-p5sz2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:43:27.350746 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:27.350725 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/536a4ad6-b9d8-40c9-bd0a-ad09c8a2a015-maas-api-tls" pod "536a4ad6-b9d8-40c9-bd0a-ad09c8a2a015" (UID: "536a4ad6-b9d8-40c9-bd0a-ad09c8a2a015"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:43:27.398963 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:27.398924 5052 generic.go:356] "Generic (PLEG): container finished" podID="536a4ad6-b9d8-40c9-bd0a-ad09c8a2a015" containerID="65c49ba7027225606927764745921156c6411201a3c796bad02a277e07ef4461" exitCode=0 Oct 30 16:43:27.399137 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:27.398975 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a0777a0c-5d898fd794-dt7pf" Oct 30 16:43:27.399137 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:27.399011 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a0777a0c-5d898fd794-dt7pf" event={"ID":"536a4ad6-b9d8-40c9-bd0a-ad09c8a2a015","Type":"ContainerDied","Data":"65c49ba7027225606927764745921156c6411201a3c796bad02a277e07ef4461"} Oct 30 16:43:27.399137 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:27.399062 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a0777a0c-5d898fd794-dt7pf" event={"ID":"536a4ad6-b9d8-40c9-bd0a-ad09c8a2a015","Type":"ContainerDied","Data":"91ee4cb331d6d52790daad5502029ab9425870522f0f9bec72a06176373825a4"} Oct 30 16:43:27.399137 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:27.399084 5052 scope.go:122] "RemoveContainer" containerID="65c49ba7027225606927764745921156c6411201a3c796bad02a277e07ef4461" Oct 30 16:43:27.410551 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:27.410532 5052 scope.go:122] "RemoveContainer" containerID="65c49ba7027225606927764745921156c6411201a3c796bad02a277e07ef4461" Oct 30 16:43:27.410863 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:43:27.410842 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65c49ba7027225606927764745921156c6411201a3c796bad02a277e07ef4461\": container with ID starting with 65c49ba7027225606927764745921156c6411201a3c796bad02a277e07ef4461 not found: ID does not exist" containerID="65c49ba7027225606927764745921156c6411201a3c796bad02a277e07ef4461" Oct 30 16:43:27.410930 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:27.410872 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65c49ba7027225606927764745921156c6411201a3c796bad02a277e07ef4461"} err="failed to get container status \"65c49ba7027225606927764745921156c6411201a3c796bad02a277e07ef4461\": rpc error: code = NotFound desc = could not find container \"65c49ba7027225606927764745921156c6411201a3c796bad02a277e07ef4461\": container with ID starting with 65c49ba7027225606927764745921156c6411201a3c796bad02a277e07ef4461 not found: ID does not exist" Oct 30 16:43:27.447849 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:27.447783 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5sz2\" (UniqueName: \"kubernetes.io/projected/536a4ad6-b9d8-40c9-bd0a-ad09c8a2a015-kube-api-access-p5sz2\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:43:27.448433 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:27.448410 5052 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/536a4ad6-b9d8-40c9-bd0a-ad09c8a2a015-metrics-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:43:27.448433 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:27.448438 5052 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/536a4ad6-b9d8-40c9-bd0a-ad09c8a2a015-maas-api-tls\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:43:27.448571 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:27.448453 5052 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/536a4ad6-b9d8-40c9-bd0a-ad09c8a2a015-openshift-service-ca\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:43:27.454111 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:27.454074 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a0777a0c-5d898fd794-dt7pf"] Oct 30 16:43:27.458223 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:27.458198 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a0777a0c-5d898fd794-dt7pf"] Oct 30 16:43:28.013937 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:28.013894 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-a0777a0c-85f5c98494-2896v" podUID="43d9d192-2f2b-4a26-8ecc-fe143833e3b3" containerName="payload-processing" containerID="cri-o://fa0f0dd3bc98cd300523227c6c6727d8d63b6860b3dd46ca5190da197b21cb97" gracePeriod=25 Oct 30 16:43:28.052650 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:28.052615 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-a0777a0c-69644576f5-4t4vr" podUID="30dbe70e-554d-4f2c-a7e9-80546a168c30" containerName="payload-pre-processing" containerID="cri-o://a495ab0430d649ea406835bf02e5fedf37aa0402e42b741d2ffd673dfe68ad99" gracePeriod=25 Oct 30 16:43:28.340297 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:28.340272 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a0777a0c-85f5c98494-2896v" Oct 30 16:43:28.368985 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:28.368956 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43d9d192-2f2b-4a26-8ecc-fe143833e3b3-kube-api-access-qmjb2\" (UniqueName: \"kubernetes.io/projected/43d9d192-2f2b-4a26-8ecc-fe143833e3b3-kube-api-access-qmjb2\") pod \"43d9d192-2f2b-4a26-8ecc-fe143833e3b3\" (UID: \"43d9d192-2f2b-4a26-8ecc-fe143833e3b3\") " Oct 30 16:43:28.369143 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:28.369129 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43d9d192-2f2b-4a26-8ecc-fe143833e3b3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43d9d192-2f2b-4a26-8ecc-fe143833e3b3-plugins-config-volume\") pod \"43d9d192-2f2b-4a26-8ecc-fe143833e3b3\" (UID: \"43d9d192-2f2b-4a26-8ecc-fe143833e3b3\") " Oct 30 16:43:28.369552 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:28.369524 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43d9d192-2f2b-4a26-8ecc-fe143833e3b3-plugins-config-volume" pod "43d9d192-2f2b-4a26-8ecc-fe143833e3b3" (UID: "43d9d192-2f2b-4a26-8ecc-fe143833e3b3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:43:28.369909 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:28.369890 5052 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43d9d192-2f2b-4a26-8ecc-fe143833e3b3-plugins-config-volume\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:43:28.371024 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:28.370998 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43d9d192-2f2b-4a26-8ecc-fe143833e3b3-kube-api-access-qmjb2" pod "43d9d192-2f2b-4a26-8ecc-fe143833e3b3" (UID: "43d9d192-2f2b-4a26-8ecc-fe143833e3b3"). InnerVolumeSpecName "kube-api-access-qmjb2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:43:28.392302 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:28.392275 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a0777a0c-69644576f5-4t4vr" Oct 30 16:43:28.410779 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:28.410746 5052 generic.go:356] "Generic (PLEG): container finished" podID="43d9d192-2f2b-4a26-8ecc-fe143833e3b3" containerID="fa0f0dd3bc98cd300523227c6c6727d8d63b6860b3dd46ca5190da197b21cb97" exitCode=0 Oct 30 16:43:28.410945 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:28.410795 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a0777a0c-85f5c98494-2896v" Oct 30 16:43:28.410945 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:28.410835 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a0777a0c-85f5c98494-2896v" event={"ID":"43d9d192-2f2b-4a26-8ecc-fe143833e3b3","Type":"ContainerDied","Data":"fa0f0dd3bc98cd300523227c6c6727d8d63b6860b3dd46ca5190da197b21cb97"} Oct 30 16:43:28.410945 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:28.410931 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a0777a0c-85f5c98494-2896v" event={"ID":"43d9d192-2f2b-4a26-8ecc-fe143833e3b3","Type":"ContainerDied","Data":"a44f88dc9df3a7f3aed8e9f1590aa685bb28f1a4ae4730ad213e82cfc16bb6b2"} Oct 30 16:43:28.411119 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:28.410962 5052 scope.go:122] "RemoveContainer" containerID="fa0f0dd3bc98cd300523227c6c6727d8d63b6860b3dd46ca5190da197b21cb97" Oct 30 16:43:28.413541 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:28.413521 5052 generic.go:356] "Generic (PLEG): container finished" podID="30dbe70e-554d-4f2c-a7e9-80546a168c30" containerID="a495ab0430d649ea406835bf02e5fedf37aa0402e42b741d2ffd673dfe68ad99" exitCode=0 Oct 30 16:43:28.413710 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:28.413550 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a0777a0c-69644576f5-4t4vr" event={"ID":"30dbe70e-554d-4f2c-a7e9-80546a168c30","Type":"ContainerDied","Data":"a495ab0430d649ea406835bf02e5fedf37aa0402e42b741d2ffd673dfe68ad99"} Oct 30 16:43:28.413710 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:28.413564 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a0777a0c-69644576f5-4t4vr" Oct 30 16:43:28.413710 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:28.413595 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a0777a0c-69644576f5-4t4vr" event={"ID":"30dbe70e-554d-4f2c-a7e9-80546a168c30","Type":"ContainerDied","Data":"c4b4d51703489d1274e686878fafc5689ab564a276f2e14f3b36ec3412d54ec0"} Oct 30 16:43:28.426059 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:28.426043 5052 scope.go:122] "RemoveContainer" containerID="fa0f0dd3bc98cd300523227c6c6727d8d63b6860b3dd46ca5190da197b21cb97" Oct 30 16:43:28.426332 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:43:28.426314 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa0f0dd3bc98cd300523227c6c6727d8d63b6860b3dd46ca5190da197b21cb97\": container with ID starting with fa0f0dd3bc98cd300523227c6c6727d8d63b6860b3dd46ca5190da197b21cb97 not found: ID does not exist" containerID="fa0f0dd3bc98cd300523227c6c6727d8d63b6860b3dd46ca5190da197b21cb97" Oct 30 16:43:28.426417 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:28.426340 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa0f0dd3bc98cd300523227c6c6727d8d63b6860b3dd46ca5190da197b21cb97"} err="failed to get container status \"fa0f0dd3bc98cd300523227c6c6727d8d63b6860b3dd46ca5190da197b21cb97\": rpc error: code = NotFound desc = could not find container \"fa0f0dd3bc98cd300523227c6c6727d8d63b6860b3dd46ca5190da197b21cb97\": container with ID starting with fa0f0dd3bc98cd300523227c6c6727d8d63b6860b3dd46ca5190da197b21cb97 not found: ID does not exist" Oct 30 16:43:28.426417 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:28.426357 5052 scope.go:122] "RemoveContainer" containerID="a495ab0430d649ea406835bf02e5fedf37aa0402e42b741d2ffd673dfe68ad99" Oct 30 16:43:28.439370 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:28.439344 5052 scope.go:122] "RemoveContainer" containerID="a495ab0430d649ea406835bf02e5fedf37aa0402e42b741d2ffd673dfe68ad99" Oct 30 16:43:28.439699 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:43:28.439681 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a495ab0430d649ea406835bf02e5fedf37aa0402e42b741d2ffd673dfe68ad99\": container with ID starting with a495ab0430d649ea406835bf02e5fedf37aa0402e42b741d2ffd673dfe68ad99 not found: ID does not exist" containerID="a495ab0430d649ea406835bf02e5fedf37aa0402e42b741d2ffd673dfe68ad99" Oct 30 16:43:28.439788 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:28.439717 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a495ab0430d649ea406835bf02e5fedf37aa0402e42b741d2ffd673dfe68ad99"} err="failed to get container status \"a495ab0430d649ea406835bf02e5fedf37aa0402e42b741d2ffd673dfe68ad99\": rpc error: code = NotFound desc = could not find container \"a495ab0430d649ea406835bf02e5fedf37aa0402e42b741d2ffd673dfe68ad99\": container with ID starting with a495ab0430d649ea406835bf02e5fedf37aa0402e42b741d2ffd673dfe68ad99 not found: ID does not exist" Oct 30 16:43:28.471421 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:28.471388 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30dbe70e-554d-4f2c-a7e9-80546a168c30-kube-api-access-f5xsh\" (UniqueName: \"kubernetes.io/projected/30dbe70e-554d-4f2c-a7e9-80546a168c30-kube-api-access-f5xsh\") pod \"30dbe70e-554d-4f2c-a7e9-80546a168c30\" (UID: \"30dbe70e-554d-4f2c-a7e9-80546a168c30\") " Oct 30 16:43:28.471570 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:28.471429 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30dbe70e-554d-4f2c-a7e9-80546a168c30-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30dbe70e-554d-4f2c-a7e9-80546a168c30-plugins-config-volume\") pod \"30dbe70e-554d-4f2c-a7e9-80546a168c30\" (UID: \"30dbe70e-554d-4f2c-a7e9-80546a168c30\") " Oct 30 16:43:28.471920 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:28.471887 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30dbe70e-554d-4f2c-a7e9-80546a168c30-plugins-config-volume" pod "30dbe70e-554d-4f2c-a7e9-80546a168c30" (UID: "30dbe70e-554d-4f2c-a7e9-80546a168c30"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:43:28.472047 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:28.471962 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmjb2\" (UniqueName: \"kubernetes.io/projected/43d9d192-2f2b-4a26-8ecc-fe143833e3b3-kube-api-access-qmjb2\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:43:28.472047 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:28.472041 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a0777a0c-85f5c98494-2896v"] Oct 30 16:43:28.473882 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:28.473860 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30dbe70e-554d-4f2c-a7e9-80546a168c30-kube-api-access-f5xsh" pod "30dbe70e-554d-4f2c-a7e9-80546a168c30" (UID: "30dbe70e-554d-4f2c-a7e9-80546a168c30"). InnerVolumeSpecName "kube-api-access-f5xsh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:43:28.476479 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:28.476456 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a0777a0c-85f5c98494-2896v"] Oct 30 16:43:28.573708 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:28.573637 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5xsh\" (UniqueName: \"kubernetes.io/projected/30dbe70e-554d-4f2c-a7e9-80546a168c30-kube-api-access-f5xsh\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:43:28.573708 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:28.573666 5052 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30dbe70e-554d-4f2c-a7e9-80546a168c30-plugins-config-volume\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:43:28.599728 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:28.599696 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43d9d192-2f2b-4a26-8ecc-fe143833e3b3" path="/var/lib/kubelet/pods/43d9d192-2f2b-4a26-8ecc-fe143833e3b3/volumes" Oct 30 16:43:28.600197 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:28.600183 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="536a4ad6-b9d8-40c9-bd0a-ad09c8a2a015" path="/var/lib/kubelet/pods/536a4ad6-b9d8-40c9-bd0a-ad09c8a2a015/volumes" Oct 30 16:43:28.755135 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:28.755103 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a0777a0c-69644576f5-4t4vr"] Oct 30 16:43:28.760053 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:28.760024 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a0777a0c-69644576f5-4t4vr"] Oct 30 16:43:30.600303 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:30.600273 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30dbe70e-554d-4f2c-a7e9-80546a168c30" path="/var/lib/kubelet/pods/30dbe70e-554d-4f2c-a7e9-80546a168c30/volumes" Oct 30 16:43:46.980238 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:46.980196 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w5-e43400-openshift-default-7cf9477785-5sxxd"] Oct 30 16:43:46.981575 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:46.981295 5052 cpu_manager.go:414] "containerMap: removing container" podUID="536a4ad6-b9d8-40c9-bd0a-ad09c8a2a015" containerName="maas-api" Oct 30 16:43:46.981575 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:46.981319 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="536a4ad6-b9d8-40c9-bd0a-ad09c8a2a015" containerName="maas-api" Oct 30 16:43:46.981575 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:46.981343 5052 cpu_manager.go:414] "containerMap: removing container" podUID="43d9d192-2f2b-4a26-8ecc-fe143833e3b3" containerName="payload-processing" Oct 30 16:43:46.981575 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:46.981351 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43d9d192-2f2b-4a26-8ecc-fe143833e3b3" containerName="payload-processing" Oct 30 16:43:46.981575 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:46.981365 5052 cpu_manager.go:414] "containerMap: removing container" podUID="30dbe70e-554d-4f2c-a7e9-80546a168c30" containerName="payload-pre-processing" Oct 30 16:43:46.981575 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:46.981374 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30dbe70e-554d-4f2c-a7e9-80546a168c30" containerName="payload-pre-processing" Oct 30 16:43:46.981575 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:46.981545 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="536a4ad6-b9d8-40c9-bd0a-ad09c8a2a015" containerName="maas-api" Oct 30 16:43:46.981575 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:46.981561 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="43d9d192-2f2b-4a26-8ecc-fe143833e3b3" containerName="payload-processing" Oct 30 16:43:46.981575 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:46.981570 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="30dbe70e-554d-4f2c-a7e9-80546a168c30" containerName="payload-pre-processing" Oct 30 16:43:46.987704 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:46.987680 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-e43400-openshift-default-7cf9477785-5sxxd" Oct 30 16:43:47.024700 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:47.024666 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w5-e43400-openshift-default-dockercfg-2n8nk\"" Oct 30 16:43:47.032705 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:47.032676 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-e43400-openshift-default-7cf9477785-5sxxd"] Oct 30 16:43:47.055379 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:47.055338 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/050171fd-aa7a-47db-8cea-e7ed38828148-workload-socket\") pod \"e2e-shared-a-w5-e43400-openshift-default-7cf9477785-5sxxd\" (UID: \"050171fd-aa7a-47db-8cea-e7ed38828148\") " pod="openshift-ingress/e2e-shared-a-w5-e43400-openshift-default-7cf9477785-5sxxd" Oct 30 16:43:47.055656 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:47.055438 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/050171fd-aa7a-47db-8cea-e7ed38828148-workload-certs\") pod \"e2e-shared-a-w5-e43400-openshift-default-7cf9477785-5sxxd\" (UID: \"050171fd-aa7a-47db-8cea-e7ed38828148\") " pod="openshift-ingress/e2e-shared-a-w5-e43400-openshift-default-7cf9477785-5sxxd" Oct 30 16:43:47.055656 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:47.055523 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/050171fd-aa7a-47db-8cea-e7ed38828148-istio-token\") pod \"e2e-shared-a-w5-e43400-openshift-default-7cf9477785-5sxxd\" (UID: \"050171fd-aa7a-47db-8cea-e7ed38828148\") " pod="openshift-ingress/e2e-shared-a-w5-e43400-openshift-default-7cf9477785-5sxxd" Oct 30 16:43:47.055656 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:47.055619 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/050171fd-aa7a-47db-8cea-e7ed38828148-istio-podinfo\") pod \"e2e-shared-a-w5-e43400-openshift-default-7cf9477785-5sxxd\" (UID: \"050171fd-aa7a-47db-8cea-e7ed38828148\") " pod="openshift-ingress/e2e-shared-a-w5-e43400-openshift-default-7cf9477785-5sxxd" Oct 30 16:43:47.055876 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:47.055686 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/050171fd-aa7a-47db-8cea-e7ed38828148-istiod-ca-cert\") pod \"e2e-shared-a-w5-e43400-openshift-default-7cf9477785-5sxxd\" (UID: \"050171fd-aa7a-47db-8cea-e7ed38828148\") " pod="openshift-ingress/e2e-shared-a-w5-e43400-openshift-default-7cf9477785-5sxxd" Oct 30 16:43:47.055876 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:47.055758 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75f49\" (UniqueName: \"kubernetes.io/projected/050171fd-aa7a-47db-8cea-e7ed38828148-kube-api-access-75f49\") pod \"e2e-shared-a-w5-e43400-openshift-default-7cf9477785-5sxxd\" (UID: \"050171fd-aa7a-47db-8cea-e7ed38828148\") " pod="openshift-ingress/e2e-shared-a-w5-e43400-openshift-default-7cf9477785-5sxxd" Oct 30 16:43:47.055876 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:47.055856 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/050171fd-aa7a-47db-8cea-e7ed38828148-credential-socket\") pod \"e2e-shared-a-w5-e43400-openshift-default-7cf9477785-5sxxd\" (UID: \"050171fd-aa7a-47db-8cea-e7ed38828148\") " pod="openshift-ingress/e2e-shared-a-w5-e43400-openshift-default-7cf9477785-5sxxd" Oct 30 16:43:47.056032 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:47.055885 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/050171fd-aa7a-47db-8cea-e7ed38828148-istio-data\") pod \"e2e-shared-a-w5-e43400-openshift-default-7cf9477785-5sxxd\" (UID: \"050171fd-aa7a-47db-8cea-e7ed38828148\") " pod="openshift-ingress/e2e-shared-a-w5-e43400-openshift-default-7cf9477785-5sxxd" Oct 30 16:43:47.056032 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:47.055935 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/050171fd-aa7a-47db-8cea-e7ed38828148-istio-envoy\") pod \"e2e-shared-a-w5-e43400-openshift-default-7cf9477785-5sxxd\" (UID: \"050171fd-aa7a-47db-8cea-e7ed38828148\") " pod="openshift-ingress/e2e-shared-a-w5-e43400-openshift-default-7cf9477785-5sxxd" Oct 30 16:43:47.232653 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:47.207278 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/050171fd-aa7a-47db-8cea-e7ed38828148-workload-socket\") pod \"e2e-shared-a-w5-e43400-openshift-default-7cf9477785-5sxxd\" (UID: \"050171fd-aa7a-47db-8cea-e7ed38828148\") " pod="openshift-ingress/e2e-shared-a-w5-e43400-openshift-default-7cf9477785-5sxxd" Oct 30 16:43:47.245606 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:47.236634 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/050171fd-aa7a-47db-8cea-e7ed38828148-workload-certs\") pod \"e2e-shared-a-w5-e43400-openshift-default-7cf9477785-5sxxd\" (UID: \"050171fd-aa7a-47db-8cea-e7ed38828148\") " pod="openshift-ingress/e2e-shared-a-w5-e43400-openshift-default-7cf9477785-5sxxd" Oct 30 16:43:47.245606 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:47.236727 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/050171fd-aa7a-47db-8cea-e7ed38828148-istio-token\") pod \"e2e-shared-a-w5-e43400-openshift-default-7cf9477785-5sxxd\" (UID: \"050171fd-aa7a-47db-8cea-e7ed38828148\") " pod="openshift-ingress/e2e-shared-a-w5-e43400-openshift-default-7cf9477785-5sxxd" Oct 30 16:43:47.245606 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:47.236800 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/050171fd-aa7a-47db-8cea-e7ed38828148-istio-podinfo\") pod \"e2e-shared-a-w5-e43400-openshift-default-7cf9477785-5sxxd\" (UID: \"050171fd-aa7a-47db-8cea-e7ed38828148\") " pod="openshift-ingress/e2e-shared-a-w5-e43400-openshift-default-7cf9477785-5sxxd" Oct 30 16:43:47.245606 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:47.236877 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/050171fd-aa7a-47db-8cea-e7ed38828148-istiod-ca-cert\") pod \"e2e-shared-a-w5-e43400-openshift-default-7cf9477785-5sxxd\" (UID: \"050171fd-aa7a-47db-8cea-e7ed38828148\") " pod="openshift-ingress/e2e-shared-a-w5-e43400-openshift-default-7cf9477785-5sxxd" Oct 30 16:43:47.245606 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:47.236962 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75f49\" (UniqueName: \"kubernetes.io/projected/050171fd-aa7a-47db-8cea-e7ed38828148-kube-api-access-75f49\") pod \"e2e-shared-a-w5-e43400-openshift-default-7cf9477785-5sxxd\" (UID: \"050171fd-aa7a-47db-8cea-e7ed38828148\") " pod="openshift-ingress/e2e-shared-a-w5-e43400-openshift-default-7cf9477785-5sxxd" Oct 30 16:43:47.245606 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:47.237059 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/050171fd-aa7a-47db-8cea-e7ed38828148-credential-socket\") pod \"e2e-shared-a-w5-e43400-openshift-default-7cf9477785-5sxxd\" (UID: \"050171fd-aa7a-47db-8cea-e7ed38828148\") " pod="openshift-ingress/e2e-shared-a-w5-e43400-openshift-default-7cf9477785-5sxxd" Oct 30 16:43:47.245606 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:47.237093 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/050171fd-aa7a-47db-8cea-e7ed38828148-istio-data\") pod \"e2e-shared-a-w5-e43400-openshift-default-7cf9477785-5sxxd\" (UID: \"050171fd-aa7a-47db-8cea-e7ed38828148\") " pod="openshift-ingress/e2e-shared-a-w5-e43400-openshift-default-7cf9477785-5sxxd" Oct 30 16:43:47.245606 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:47.237148 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/050171fd-aa7a-47db-8cea-e7ed38828148-istio-envoy\") pod \"e2e-shared-a-w5-e43400-openshift-default-7cf9477785-5sxxd\" (UID: \"050171fd-aa7a-47db-8cea-e7ed38828148\") " pod="openshift-ingress/e2e-shared-a-w5-e43400-openshift-default-7cf9477785-5sxxd" Oct 30 16:43:47.245606 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:47.239322 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/050171fd-aa7a-47db-8cea-e7ed38828148-workload-certs\") pod \"e2e-shared-a-w5-e43400-openshift-default-7cf9477785-5sxxd\" (UID: \"050171fd-aa7a-47db-8cea-e7ed38828148\") " pod="openshift-ingress/e2e-shared-a-w5-e43400-openshift-default-7cf9477785-5sxxd" Oct 30 16:43:47.245606 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:47.210033 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/050171fd-aa7a-47db-8cea-e7ed38828148-workload-socket\") pod \"e2e-shared-a-w5-e43400-openshift-default-7cf9477785-5sxxd\" (UID: \"050171fd-aa7a-47db-8cea-e7ed38828148\") " pod="openshift-ingress/e2e-shared-a-w5-e43400-openshift-default-7cf9477785-5sxxd" Oct 30 16:43:47.245606 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:47.240471 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/050171fd-aa7a-47db-8cea-e7ed38828148-istiod-ca-cert\") pod \"e2e-shared-a-w5-e43400-openshift-default-7cf9477785-5sxxd\" (UID: \"050171fd-aa7a-47db-8cea-e7ed38828148\") " pod="openshift-ingress/e2e-shared-a-w5-e43400-openshift-default-7cf9477785-5sxxd" Oct 30 16:43:47.245606 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:47.240750 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/050171fd-aa7a-47db-8cea-e7ed38828148-credential-socket\") pod \"e2e-shared-a-w5-e43400-openshift-default-7cf9477785-5sxxd\" (UID: \"050171fd-aa7a-47db-8cea-e7ed38828148\") " pod="openshift-ingress/e2e-shared-a-w5-e43400-openshift-default-7cf9477785-5sxxd" Oct 30 16:43:47.245606 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:47.240980 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/050171fd-aa7a-47db-8cea-e7ed38828148-istio-data\") pod \"e2e-shared-a-w5-e43400-openshift-default-7cf9477785-5sxxd\" (UID: \"050171fd-aa7a-47db-8cea-e7ed38828148\") " pod="openshift-ingress/e2e-shared-a-w5-e43400-openshift-default-7cf9477785-5sxxd" Oct 30 16:43:47.317605 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:47.316202 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/050171fd-aa7a-47db-8cea-e7ed38828148-istio-podinfo\") pod \"e2e-shared-a-w5-e43400-openshift-default-7cf9477785-5sxxd\" (UID: \"050171fd-aa7a-47db-8cea-e7ed38828148\") " pod="openshift-ingress/e2e-shared-a-w5-e43400-openshift-default-7cf9477785-5sxxd" Oct 30 16:43:47.367879 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:47.363605 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75f49\" (UniqueName: \"kubernetes.io/projected/050171fd-aa7a-47db-8cea-e7ed38828148-kube-api-access-75f49\") pod \"e2e-shared-a-w5-e43400-openshift-default-7cf9477785-5sxxd\" (UID: \"050171fd-aa7a-47db-8cea-e7ed38828148\") " pod="openshift-ingress/e2e-shared-a-w5-e43400-openshift-default-7cf9477785-5sxxd" Oct 30 16:43:47.367879 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:47.364547 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/050171fd-aa7a-47db-8cea-e7ed38828148-istio-token\") pod \"e2e-shared-a-w5-e43400-openshift-default-7cf9477785-5sxxd\" (UID: \"050171fd-aa7a-47db-8cea-e7ed38828148\") " pod="openshift-ingress/e2e-shared-a-w5-e43400-openshift-default-7cf9477785-5sxxd" Oct 30 16:43:47.504636 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:47.504534 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/050171fd-aa7a-47db-8cea-e7ed38828148-istio-envoy\") pod \"e2e-shared-a-w5-e43400-openshift-default-7cf9477785-5sxxd\" (UID: \"050171fd-aa7a-47db-8cea-e7ed38828148\") " pod="openshift-ingress/e2e-shared-a-w5-e43400-openshift-default-7cf9477785-5sxxd" Oct 30 16:43:47.607615 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:47.607570 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-e43400-openshift-default-7cf9477785-5sxxd" Oct 30 16:43:47.933986 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:47.933954 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-e43400-openshift-default-7cf9477785-5sxxd"] Oct 30 16:43:47.935892 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:43:47.935865 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod050171fd_aa7a_47db_8cea_e7ed38828148.slice/crio-490c45889a25474d2392f470f2c6922d37b118f68b374320c181306d773bbd86 WatchSource:0}: Error finding container 490c45889a25474d2392f470f2c6922d37b118f68b374320c181306d773bbd86: Status 404 returned error can't find the container with id 490c45889a25474d2392f470f2c6922d37b118f68b374320c181306d773bbd86 Oct 30 16:43:47.938554 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:47.938520 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:43:47.938681 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:47.938625 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:43:47.938681 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:47.938662 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:43:48.684194 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:48.684156 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-e43400-openshift-default-7cf9477785-5sxxd" event={"ID":"050171fd-aa7a-47db-8cea-e7ed38828148","Type":"ContainerStarted","Data":"789a1f1638487db7213badfc4c9b74afaf0324538c0e4019bfb4aefc2af45387"} Oct 30 16:43:48.684662 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:48.684210 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-e43400-openshift-default-7cf9477785-5sxxd" event={"ID":"050171fd-aa7a-47db-8cea-e7ed38828148","Type":"ContainerStarted","Data":"490c45889a25474d2392f470f2c6922d37b118f68b374320c181306d773bbd86"} Oct 30 16:43:48.755290 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:48.755241 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w5-e43400-openshift-default-7cf9477785-5sxxd" podStartSLOduration=2.7552249939999998 podStartE2EDuration="2.755224994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:43:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:43:48.739187194 +0000 UTC m=+1384.266522417" watchObservedRunningTime="2026-10-30 16:43:48.755224994 +0000 UTC m=+1384.282560212" Oct 30 16:43:48.842476 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:43:48.842437 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 16:43:49.608501 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:49.608470 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-e43400-openshift-default-7cf9477785-5sxxd" Oct 30 16:43:49.614639 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:49.614614 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-e43400-openshift-default-7cf9477785-5sxxd" Oct 30 16:43:49.730067 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:49.717177 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-e43400-openshift-default-7cf9477785-5sxxd" Oct 30 16:43:49.730067 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:49.718712 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-e43400-openshift-default-7cf9477785-5sxxd" Oct 30 16:43:53.050288 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:53.050258 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-e43400-5d9d988d86-hvrpq"] Oct 30 16:43:53.054673 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:53.054654 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-e43400-5d9d988d86-hvrpq" Oct 30 16:43:53.073632 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:53.073608 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-e43400-5d9d988d86-hvrpq"] Oct 30 16:43:53.081024 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:53.080987 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w5-e43400\"" Oct 30 16:43:53.081157 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:53.081022 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w5-e43400-dockercfg-kg5gv\"" Oct 30 16:43:53.108907 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:53.108877 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd4ea853-d161-4a29-b561-cad7e8fe2951-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-e43400-5d9d988d86-hvrpq\" (UID: \"bd4ea853-d161-4a29-b561-cad7e8fe2951\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-e43400-5d9d988d86-hvrpq" Oct 30 16:43:53.109037 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:53.108924 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cd8v5\" (UniqueName: \"kubernetes.io/projected/bd4ea853-d161-4a29-b561-cad7e8fe2951-kube-api-access-cd8v5\") pod \"payload-processing-e2e-shared-a-w5-e43400-5d9d988d86-hvrpq\" (UID: \"bd4ea853-d161-4a29-b561-cad7e8fe2951\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-e43400-5d9d988d86-hvrpq" Oct 30 16:43:53.277605 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:53.253797 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd4ea853-d161-4a29-b561-cad7e8fe2951-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-e43400-5d9d988d86-hvrpq\" (UID: \"bd4ea853-d161-4a29-b561-cad7e8fe2951\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-e43400-5d9d988d86-hvrpq" Oct 30 16:43:53.277605 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:53.254040 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cd8v5\" (UniqueName: \"kubernetes.io/projected/bd4ea853-d161-4a29-b561-cad7e8fe2951-kube-api-access-cd8v5\") pod \"payload-processing-e2e-shared-a-w5-e43400-5d9d988d86-hvrpq\" (UID: \"bd4ea853-d161-4a29-b561-cad7e8fe2951\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-e43400-5d9d988d86-hvrpq" Oct 30 16:43:53.277605 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:53.262811 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd4ea853-d161-4a29-b561-cad7e8fe2951-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-e43400-5d9d988d86-hvrpq\" (UID: \"bd4ea853-d161-4a29-b561-cad7e8fe2951\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-e43400-5d9d988d86-hvrpq" Oct 30 16:43:53.300674 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:53.291270 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cd8v5\" (UniqueName: \"kubernetes.io/projected/bd4ea853-d161-4a29-b561-cad7e8fe2951-kube-api-access-cd8v5\") pod \"payload-processing-e2e-shared-a-w5-e43400-5d9d988d86-hvrpq\" (UID: \"bd4ea853-d161-4a29-b561-cad7e8fe2951\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-e43400-5d9d988d86-hvrpq" Oct 30 16:43:53.373165 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:53.372864 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-e43400-5d9d988d86-hvrpq" Oct 30 16:43:53.863920 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:53.863892 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-e43400-5d9d988d86-hvrpq"] Oct 30 16:43:53.866226 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:43:53.866198 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd4ea853_d161_4a29_b561_cad7e8fe2951.slice/crio-831f62f0faf77e7d2b691c7f67be02a7bd8c5a05ae52f23e7fd151fbe5173890 WatchSource:0}: Error finding container 831f62f0faf77e7d2b691c7f67be02a7bd8c5a05ae52f23e7fd151fbe5173890: Status 404 returned error can't find the container with id 831f62f0faf77e7d2b691c7f67be02a7bd8c5a05ae52f23e7fd151fbe5173890 Oct 30 16:43:54.785541 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:54.785501 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-e43400-5d9d988d86-hvrpq" event={"ID":"bd4ea853-d161-4a29-b561-cad7e8fe2951","Type":"ContainerStarted","Data":"a145edb3fb03611ca73689d0fe871829ea76d234b167004b71a5a7c24f806c7c"} Oct 30 16:43:54.786052 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:54.785567 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-e43400-5d9d988d86-hvrpq" Oct 30 16:43:54.786052 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:54.785605 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-e43400-5d9d988d86-hvrpq" event={"ID":"bd4ea853-d161-4a29-b561-cad7e8fe2951","Type":"ContainerStarted","Data":"831f62f0faf77e7d2b691c7f67be02a7bd8c5a05ae52f23e7fd151fbe5173890"} Oct 30 16:43:54.831520 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:43:54.831467 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-e43400-5d9d988d86-hvrpq" podStartSLOduration=1.831453121 podStartE2EDuration="1.831453121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:43:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:43:54.828783721 +0000 UTC m=+1390.356118969" watchObservedRunningTime="2026-10-30 16:43:54.831453121 +0000 UTC m=+1390.358788340" Oct 30 16:44:00.093606 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:00.083659 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-830ba9-5995465bff-gb57g"] Oct 30 16:44:00.171603 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:00.154207 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-830ba9-5995465bff-gb57g" Oct 30 16:44:00.249993 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:00.249940 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-830ba9-5995465bff-gb57g"] Oct 30 16:44:00.253409 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:00.253371 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnjrj\" (UniqueName: \"kubernetes.io/projected/5f25133c-8cae-4395-807d-8b77508a9285-kube-api-access-vnjrj\") pod \"payload-pre-processing-e2e-shared-a-w6-830ba9-5995465bff-gb57g\" (UID: \"5f25133c-8cae-4395-807d-8b77508a9285\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-830ba9-5995465bff-gb57g" Oct 30 16:44:00.253820 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:00.253796 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f25133c-8cae-4395-807d-8b77508a9285-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-830ba9-5995465bff-gb57g\" (UID: \"5f25133c-8cae-4395-807d-8b77508a9285\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-830ba9-5995465bff-gb57g" Oct 30 16:44:00.269636 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:00.262876 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w6-830ba9\"" Oct 30 16:44:00.269636 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:00.263596 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w6-830ba9-dockercfg-pwnfz\"" Oct 30 16:44:00.361061 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:00.360957 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f25133c-8cae-4395-807d-8b77508a9285-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-830ba9-5995465bff-gb57g\" (UID: \"5f25133c-8cae-4395-807d-8b77508a9285\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-830ba9-5995465bff-gb57g" Oct 30 16:44:00.362212 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:00.362044 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f25133c-8cae-4395-807d-8b77508a9285-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-830ba9-5995465bff-gb57g\" (UID: \"5f25133c-8cae-4395-807d-8b77508a9285\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-830ba9-5995465bff-gb57g" Oct 30 16:44:00.362392 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:00.362247 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnjrj\" (UniqueName: \"kubernetes.io/projected/5f25133c-8cae-4395-807d-8b77508a9285-kube-api-access-vnjrj\") pod \"payload-pre-processing-e2e-shared-a-w6-830ba9-5995465bff-gb57g\" (UID: \"5f25133c-8cae-4395-807d-8b77508a9285\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-830ba9-5995465bff-gb57g" Oct 30 16:44:00.384935 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:00.384908 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnjrj\" (UniqueName: \"kubernetes.io/projected/5f25133c-8cae-4395-807d-8b77508a9285-kube-api-access-vnjrj\") pod \"payload-pre-processing-e2e-shared-a-w6-830ba9-5995465bff-gb57g\" (UID: \"5f25133c-8cae-4395-807d-8b77508a9285\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-830ba9-5995465bff-gb57g" Oct 30 16:44:00.672677 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:00.672647 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-830ba9-5995465bff-gb57g" Oct 30 16:44:00.983425 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:00.983401 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-830ba9-5995465bff-gb57g"] Oct 30 16:44:00.985283 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:44:00.985252 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f25133c_8cae_4395_807d_8b77508a9285.slice/crio-6d598f77ab9200db7fcf058ae00606bbad7cc5a52367fa3d7e47e28565e3dee7 WatchSource:0}: Error finding container 6d598f77ab9200db7fcf058ae00606bbad7cc5a52367fa3d7e47e28565e3dee7: Status 404 returned error can't find the container with id 6d598f77ab9200db7fcf058ae00606bbad7cc5a52367fa3d7e47e28565e3dee7 Oct 30 16:44:01.879837 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:01.879796 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-830ba9-5995465bff-gb57g" event={"ID":"5f25133c-8cae-4395-807d-8b77508a9285","Type":"ContainerStarted","Data":"c988b975c2c4e83f36d5f5bcdd16c6fa65f8d932ba95d2ee3ab129807bec5d1f"} Oct 30 16:44:01.880306 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:01.879860 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-830ba9-5995465bff-gb57g" event={"ID":"5f25133c-8cae-4395-807d-8b77508a9285","Type":"ContainerStarted","Data":"6d598f77ab9200db7fcf058ae00606bbad7cc5a52367fa3d7e47e28565e3dee7"} Oct 30 16:44:01.880306 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:01.879888 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-830ba9-5995465bff-gb57g" Oct 30 16:44:01.924761 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:01.924705 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-830ba9-5995465bff-gb57g" podStartSLOduration=2.924690826 podStartE2EDuration="2.924690826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:43:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:44:01.921680223 +0000 UTC m=+1397.449015444" watchObservedRunningTime="2026-10-30 16:44:01.924690826 +0000 UTC m=+1397.452026044" Oct 30 16:44:02.629859 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:02.629820 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cdwcj"] Oct 30 16:44:05.832073 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:05.832040 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-e43400-5d9d988d86-hvrpq" Oct 30 16:44:12.912795 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:12.912761 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-830ba9-5995465bff-gb57g" Oct 30 16:44:14.249613 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:14.249565 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-830ba9-546f46dd46-vm797"] Oct 30 16:44:14.254350 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:14.254329 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-830ba9-546f46dd46-vm797" Oct 30 16:44:14.274619 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:14.273304 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w6-830ba9-serving-cert\"" Oct 30 16:44:14.287095 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:14.287067 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-830ba9-546f46dd46-vm797"] Oct 30 16:44:14.304863 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:14.304823 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37471512-e011-4938-9a3d-296d60601d0b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-830ba9-546f46dd46-vm797\" (UID: \"37471512-e011-4938-9a3d-296d60601d0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-830ba9-546f46dd46-vm797" Oct 30 16:44:14.305004 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:14.304934 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37471512-e011-4938-9a3d-296d60601d0b-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-830ba9-546f46dd46-vm797\" (UID: \"37471512-e011-4938-9a3d-296d60601d0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-830ba9-546f46dd46-vm797" Oct 30 16:44:14.305147 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:14.305123 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhvsw\" (UniqueName: \"kubernetes.io/projected/37471512-e011-4938-9a3d-296d60601d0b-kube-api-access-mhvsw\") pod \"maas-api-e2e-shared-a-w6-830ba9-546f46dd46-vm797\" (UID: \"37471512-e011-4938-9a3d-296d60601d0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-830ba9-546f46dd46-vm797" Oct 30 16:44:14.305203 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:14.305179 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37471512-e011-4938-9a3d-296d60601d0b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-830ba9-546f46dd46-vm797\" (UID: \"37471512-e011-4938-9a3d-296d60601d0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-830ba9-546f46dd46-vm797" Oct 30 16:44:14.406724 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:14.406693 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37471512-e011-4938-9a3d-296d60601d0b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-830ba9-546f46dd46-vm797\" (UID: \"37471512-e011-4938-9a3d-296d60601d0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-830ba9-546f46dd46-vm797" Oct 30 16:44:14.406870 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:14.406754 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37471512-e011-4938-9a3d-296d60601d0b-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-830ba9-546f46dd46-vm797\" (UID: \"37471512-e011-4938-9a3d-296d60601d0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-830ba9-546f46dd46-vm797" Oct 30 16:44:14.406870 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:14.406854 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhvsw\" (UniqueName: \"kubernetes.io/projected/37471512-e011-4938-9a3d-296d60601d0b-kube-api-access-mhvsw\") pod \"maas-api-e2e-shared-a-w6-830ba9-546f46dd46-vm797\" (UID: \"37471512-e011-4938-9a3d-296d60601d0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-830ba9-546f46dd46-vm797" Oct 30 16:44:14.406968 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:14.406899 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37471512-e011-4938-9a3d-296d60601d0b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-830ba9-546f46dd46-vm797\" (UID: \"37471512-e011-4938-9a3d-296d60601d0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-830ba9-546f46dd46-vm797" Oct 30 16:44:14.406968 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:44:14.406857 5052 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-830ba9-serving-cert: secret "maas-api-e2e-shared-a-w6-830ba9-serving-cert" not found Oct 30 16:44:14.407073 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:44:14.407018 5052 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37471512-e011-4938-9a3d-296d60601d0b-maas-api-tls podName:37471512-e011-4938-9a3d-296d60601d0b nodeName:}" failed. No retries permitted until 2026-10-30 16:44:14.906993812 +0000 UTC m=+1410.434329026 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/37471512-e011-4938-9a3d-296d60601d0b-maas-api-tls") pod "maas-api-e2e-shared-a-w6-830ba9-546f46dd46-vm797" (UID: "37471512-e011-4938-9a3d-296d60601d0b") : secret "maas-api-e2e-shared-a-w6-830ba9-serving-cert" not found Oct 30 16:44:14.407627 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:14.407575 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37471512-e011-4938-9a3d-296d60601d0b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-830ba9-546f46dd46-vm797\" (UID: \"37471512-e011-4938-9a3d-296d60601d0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-830ba9-546f46dd46-vm797" Oct 30 16:44:14.409601 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:14.409565 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37471512-e011-4938-9a3d-296d60601d0b-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-830ba9-546f46dd46-vm797\" (UID: \"37471512-e011-4938-9a3d-296d60601d0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-830ba9-546f46dd46-vm797" Oct 30 16:44:14.448499 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:14.448459 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhvsw\" (UniqueName: \"kubernetes.io/projected/37471512-e011-4938-9a3d-296d60601d0b-kube-api-access-mhvsw\") pod \"maas-api-e2e-shared-a-w6-830ba9-546f46dd46-vm797\" (UID: \"37471512-e011-4938-9a3d-296d60601d0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-830ba9-546f46dd46-vm797" Oct 30 16:44:14.919669 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:14.919626 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37471512-e011-4938-9a3d-296d60601d0b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-830ba9-546f46dd46-vm797\" (UID: \"37471512-e011-4938-9a3d-296d60601d0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-830ba9-546f46dd46-vm797" Oct 30 16:44:14.922126 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:14.922091 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37471512-e011-4938-9a3d-296d60601d0b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-830ba9-546f46dd46-vm797\" (UID: \"37471512-e011-4938-9a3d-296d60601d0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-830ba9-546f46dd46-vm797" Oct 30 16:44:15.172705 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:15.171797 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-830ba9-546f46dd46-vm797" Oct 30 16:44:15.481246 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:15.481213 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-830ba9-546f46dd46-vm797"] Oct 30 16:44:15.483081 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:44:15.483053 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37471512_e011_4938_9a3d_296d60601d0b.slice/crio-d07b90e64fc9cfb09cff005b40f5564b05d5d4066c8439fc1b02747adbabd92d WatchSource:0}: Error finding container d07b90e64fc9cfb09cff005b40f5564b05d5d4066c8439fc1b02747adbabd92d: Status 404 returned error can't find the container with id d07b90e64fc9cfb09cff005b40f5564b05d5d4066c8439fc1b02747adbabd92d Oct 30 16:44:16.024320 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:16.024283 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-830ba9-546f46dd46-vm797" event={"ID":"37471512-e011-4938-9a3d-296d60601d0b","Type":"ContainerStarted","Data":"c7915883e8b1da982bbf64f21749e70602a5f50ad1c81e116bbb1b286e0af39e"} Oct 30 16:44:16.024575 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:16.024336 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-830ba9-546f46dd46-vm797" event={"ID":"37471512-e011-4938-9a3d-296d60601d0b","Type":"ContainerStarted","Data":"d07b90e64fc9cfb09cff005b40f5564b05d5d4066c8439fc1b02747adbabd92d"} Oct 30 16:44:16.024575 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:16.024364 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-830ba9-546f46dd46-vm797" Oct 30 16:44:16.067556 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:16.067458 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-830ba9-546f46dd46-vm797" podStartSLOduration=1.745444402 podStartE2EDuration="2.067439322s" totalImagesPullingTime="321.99492ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:44:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:44:16.064948873 +0000 UTC m=+1411.592284094" watchObservedRunningTime="2026-10-30 16:44:16.067439322 +0000 UTC m=+1411.594774543" Oct 30 16:44:18.843244 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:44:18.843199 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 16:44:22.083788 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:22.083754 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-830ba9-546f46dd46-vm797" Oct 30 16:44:24.246606 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:24.234741 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/distinct-2-w4-263f00-kserve-8464897cf5-q8ck8" podUID="15ed43ef-5cdd-489a-a70e-063239822ff3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:44:24.277745 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:24.277668 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx" podUID="bd662f3a-659f-477c-b8d5-94c64288c0c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:44:24.453602 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:24.453016 5052 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-967fac/distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx" podUID="bd662f3a-659f-477c-b8d5-94c64288c0c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:44:24.790605 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:24.784812 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/distinct-w5-967fac-kserve-69cbcccd8b-b85bn" podUID="8f6b9354-378f-4451-9ff7-be7f7fec12f2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:44:25.125739 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:25.125616 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d693d0/distinct-2-w6-d693d0-kserve-6498fdc474-dpbjj" podUID="4b98b8ff-72cd-4ddb-a4b8-67078162e611" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:44:32.362637 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:32.362604 5052 generic.go:356] "Generic (PLEG): container finished" podID="050171fd-aa7a-47db-8cea-e7ed38828148" containerID="789a1f1638487db7213badfc4c9b74afaf0324538c0e4019bfb4aefc2af45387" exitCode=0 Oct 30 16:44:32.363182 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:32.362683 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-e43400-openshift-default-7cf9477785-5sxxd" event={"ID":"050171fd-aa7a-47db-8cea-e7ed38828148","Type":"ContainerDied","Data":"789a1f1638487db7213badfc4c9b74afaf0324538c0e4019bfb4aefc2af45387"} Oct 30 16:44:32.363474 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:32.363459 5052 scope.go:122] "RemoveContainer" containerID="789a1f1638487db7213badfc4c9b74afaf0324538c0e4019bfb4aefc2af45387" Oct 30 16:44:32.364954 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:32.364929 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:44:32.365038 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:32.365002 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:44:32.365091 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:32.365038 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:44:33.375325 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:33.375274 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-e43400-openshift-default-7cf9477785-5sxxd" event={"ID":"050171fd-aa7a-47db-8cea-e7ed38828148","Type":"ContainerStarted","Data":"40a1f368d1a55fda00ebc57f9df1f081aaa213295279362ea35926bbc1ea789d"} Oct 30 16:44:33.608281 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:33.608244 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-e43400-openshift-default-7cf9477785-5sxxd" Oct 30 16:44:33.609251 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:33.609227 5052 patch_prober.go:29] interesting pod/e2e-shared-a-w5-e43400-openshift-default-7cf9477785-5sxxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Oct 30 16:44:33.609317 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:33.609274 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-e43400-openshift-default-7cf9477785-5sxxd" podUID="050171fd-aa7a-47db-8cea-e7ed38828148" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Oct 30 16:44:34.608286 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:34.608253 5052 patch_prober.go:29] interesting pod/e2e-shared-a-w5-e43400-openshift-default-7cf9477785-5sxxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Oct 30 16:44:34.608690 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:34.608322 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-e43400-openshift-default-7cf9477785-5sxxd" podUID="050171fd-aa7a-47db-8cea-e7ed38828148" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Oct 30 16:44:34.717196 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:34.717158 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-e43400-openshift-default-7cf9477785-5sxxd" Oct 30 16:44:35.608910 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:35.608874 5052 patch_prober.go:29] interesting pod/e2e-shared-a-w5-e43400-openshift-default-7cf9477785-5sxxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Oct 30 16:44:35.609423 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:35.608927 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-e43400-openshift-default-7cf9477785-5sxxd" podUID="050171fd-aa7a-47db-8cea-e7ed38828148" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Oct 30 16:44:36.608192 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:36.608159 5052 patch_prober.go:29] interesting pod/e2e-shared-a-w5-e43400-openshift-default-7cf9477785-5sxxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Oct 30 16:44:36.608374 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:36.608220 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-e43400-openshift-default-7cf9477785-5sxxd" podUID="050171fd-aa7a-47db-8cea-e7ed38828148" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Oct 30 16:44:36.615986 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:36.615957 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b8af660d-787d8764b-h9rfs"] Oct 30 16:44:36.620922 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:36.620900 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-b8af660d-787d8764b-h9rfs" Oct 30 16:44:36.663899 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:36.663870 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-b8af660d-dockercfg-64sg8\"" Oct 30 16:44:36.664081 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:36.663989 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-b8af660d\"" Oct 30 16:44:36.666647 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:36.666618 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b8af660d-787d8764b-h9rfs"] Oct 30 16:44:36.673338 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:36.673315 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c983762-43df-4410-9001-d5d25c7a45f3-plugins-config-volume\") pod \"payload-processing-e2e-mt-b8af660d-787d8764b-h9rfs\" (UID: \"2c983762-43df-4410-9001-d5d25c7a45f3\") " pod="openshift-ingress/payload-processing-e2e-mt-b8af660d-787d8764b-h9rfs" Oct 30 16:44:36.673427 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:36.673366 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvvb8\" (UniqueName: \"kubernetes.io/projected/2c983762-43df-4410-9001-d5d25c7a45f3-kube-api-access-hvvb8\") pod \"payload-processing-e2e-mt-b8af660d-787d8764b-h9rfs\" (UID: \"2c983762-43df-4410-9001-d5d25c7a45f3\") " pod="openshift-ingress/payload-processing-e2e-mt-b8af660d-787d8764b-h9rfs" Oct 30 16:44:36.824607 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:36.785570 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c983762-43df-4410-9001-d5d25c7a45f3-plugins-config-volume\") pod \"payload-processing-e2e-mt-b8af660d-787d8764b-h9rfs\" (UID: \"2c983762-43df-4410-9001-d5d25c7a45f3\") " pod="openshift-ingress/payload-processing-e2e-mt-b8af660d-787d8764b-h9rfs" Oct 30 16:44:36.870659 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:36.853856 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvvb8\" (UniqueName: \"kubernetes.io/projected/2c983762-43df-4410-9001-d5d25c7a45f3-kube-api-access-hvvb8\") pod \"payload-processing-e2e-mt-b8af660d-787d8764b-h9rfs\" (UID: \"2c983762-43df-4410-9001-d5d25c7a45f3\") " pod="openshift-ingress/payload-processing-e2e-mt-b8af660d-787d8764b-h9rfs" Oct 30 16:44:36.870659 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:36.810529 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c983762-43df-4410-9001-d5d25c7a45f3-plugins-config-volume\") pod \"payload-processing-e2e-mt-b8af660d-787d8764b-h9rfs\" (UID: \"2c983762-43df-4410-9001-d5d25c7a45f3\") " pod="openshift-ingress/payload-processing-e2e-mt-b8af660d-787d8764b-h9rfs" Oct 30 16:44:37.116336 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:37.116301 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvvb8\" (UniqueName: \"kubernetes.io/projected/2c983762-43df-4410-9001-d5d25c7a45f3-kube-api-access-hvvb8\") pod \"payload-processing-e2e-mt-b8af660d-787d8764b-h9rfs\" (UID: \"2c983762-43df-4410-9001-d5d25c7a45f3\") " pod="openshift-ingress/payload-processing-e2e-mt-b8af660d-787d8764b-h9rfs" Oct 30 16:44:37.268615 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:37.268547 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-b8af660d-787d8764b-h9rfs" Oct 30 16:44:37.719870 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:37.719838 5052 patch_prober.go:29] interesting pod/e2e-shared-a-w5-e43400-openshift-default-7cf9477785-5sxxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Oct 30 16:44:37.720652 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:37.720620 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-e43400-openshift-default-7cf9477785-5sxxd" podUID="050171fd-aa7a-47db-8cea-e7ed38828148" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Oct 30 16:44:37.994032 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:37.993998 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b8af660d-787d8764b-h9rfs"] Oct 30 16:44:37.995609 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:44:37.995557 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c983762_43df_4410_9001_d5d25c7a45f3.slice/crio-4bcdcea078d035126fbeab46ed7b1f13d3e66e176376a2d4bf6de315664ecb9f WatchSource:0}: Error finding container 4bcdcea078d035126fbeab46ed7b1f13d3e66e176376a2d4bf6de315664ecb9f: Status 404 returned error can't find the container with id 4bcdcea078d035126fbeab46ed7b1f13d3e66e176376a2d4bf6de315664ecb9f Oct 30 16:44:38.576658 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:38.567655 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-b8af660d-787d8764b-h9rfs" event={"ID":"2c983762-43df-4410-9001-d5d25c7a45f3","Type":"ContainerStarted","Data":"1cea34d2c5be1ab53a7ea1aff002b811b72bf2029f09335db701ce6f718c1587"} Oct 30 16:44:38.576658 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:38.567812 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-b8af660d-787d8764b-h9rfs" event={"ID":"2c983762-43df-4410-9001-d5d25c7a45f3","Type":"ContainerStarted","Data":"4bcdcea078d035126fbeab46ed7b1f13d3e66e176376a2d4bf6de315664ecb9f"} Oct 30 16:44:38.576658 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:38.568766 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-b8af660d-787d8764b-h9rfs" Oct 30 16:44:38.621506 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:38.621468 5052 patch_prober.go:29] interesting pod/e2e-shared-a-w5-e43400-openshift-default-7cf9477785-5sxxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Oct 30 16:44:38.621860 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:38.621824 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-e43400-openshift-default-7cf9477785-5sxxd" podUID="050171fd-aa7a-47db-8cea-e7ed38828148" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Oct 30 16:44:38.647488 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:38.628692 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-b8af660d-787d8764b-h9rfs" podStartSLOduration=2.628672714 podStartE2EDuration="2.628672714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:44:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:44:38.62247428 +0000 UTC m=+1434.149809500" watchObservedRunningTime="2026-10-30 16:44:38.628672714 +0000 UTC m=+1434.156007934" Oct 30 16:44:39.613886 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:39.613856 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-e43400-openshift-default-7cf9477785-5sxxd" Oct 30 16:44:39.615041 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:39.615016 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-e43400-openshift-default-7cf9477785-5sxxd" Oct 30 16:44:47.341861 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:47.330501 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cdwcj"] Oct 30 16:44:48.844926 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:44:48.842462 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 16:44:49.580953 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:49.579087 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-b8af660d-787d8764b-h9rfs" Oct 30 16:44:56.107900 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:56.107859 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-05810b-67f9c6857b-7jz5n"] Oct 30 16:44:56.235613 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:56.224874 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-05810b-67f9c6857b-7jz5n" Oct 30 16:44:56.308684 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:56.225559 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-05810b-67f9c6857b-7jz5n"] Oct 30 16:44:56.309070 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:56.273421 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-05810b\"" Oct 30 16:44:56.309254 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:56.273510 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-05810b-dockercfg-vm497\"" Oct 30 16:44:56.309384 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:56.284575 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6136d90c-0253-4761-843b-f8a3922c93cf-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-05810b-67f9c6857b-7jz5n\" (UID: \"6136d90c-0253-4761-843b-f8a3922c93cf\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-05810b-67f9c6857b-7jz5n" Oct 30 16:44:56.309724 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:56.309699 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sfwk\" (UniqueName: \"kubernetes.io/projected/6136d90c-0253-4761-843b-f8a3922c93cf-kube-api-access-9sfwk\") pod \"payload-processing-e2e-shared-a-w2-05810b-67f9c6857b-7jz5n\" (UID: \"6136d90c-0253-4761-843b-f8a3922c93cf\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-05810b-67f9c6857b-7jz5n" Oct 30 16:44:56.419607 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:56.412675 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6136d90c-0253-4761-843b-f8a3922c93cf-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-05810b-67f9c6857b-7jz5n\" (UID: \"6136d90c-0253-4761-843b-f8a3922c93cf\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-05810b-67f9c6857b-7jz5n" Oct 30 16:44:56.431921 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:56.431880 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sfwk\" (UniqueName: \"kubernetes.io/projected/6136d90c-0253-4761-843b-f8a3922c93cf-kube-api-access-9sfwk\") pod \"payload-processing-e2e-shared-a-w2-05810b-67f9c6857b-7jz5n\" (UID: \"6136d90c-0253-4761-843b-f8a3922c93cf\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-05810b-67f9c6857b-7jz5n" Oct 30 16:44:56.435726 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:56.420914 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6136d90c-0253-4761-843b-f8a3922c93cf-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-05810b-67f9c6857b-7jz5n\" (UID: \"6136d90c-0253-4761-843b-f8a3922c93cf\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-05810b-67f9c6857b-7jz5n" Oct 30 16:44:56.548438 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:56.524985 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sfwk\" (UniqueName: \"kubernetes.io/projected/6136d90c-0253-4761-843b-f8a3922c93cf-kube-api-access-9sfwk\") pod \"payload-processing-e2e-shared-a-w2-05810b-67f9c6857b-7jz5n\" (UID: \"6136d90c-0253-4761-843b-f8a3922c93cf\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-05810b-67f9c6857b-7jz5n" Oct 30 16:44:56.701668 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:56.690474 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-05810b-67f9c6857b-7jz5n" Oct 30 16:44:57.236031 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:44:57.235996 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6136d90c_0253_4761_843b_f8a3922c93cf.slice/crio-976f7c568909d2daf7838efbdd580d5dd2280311901f18e59593d5d92cd24411 WatchSource:0}: Error finding container 976f7c568909d2daf7838efbdd580d5dd2280311901f18e59593d5d92cd24411: Status 404 returned error can't find the container with id 976f7c568909d2daf7838efbdd580d5dd2280311901f18e59593d5d92cd24411 Oct 30 16:44:57.236702 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:57.236647 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-05810b-67f9c6857b-7jz5n"] Oct 30 16:44:57.945484 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:57.945435 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-05810b-67f9c6857b-7jz5n" event={"ID":"6136d90c-0253-4761-843b-f8a3922c93cf","Type":"ContainerStarted","Data":"cb0f06ad0af10ba3297ecb873778a0595554e2ae8d98f8c2e7f76ec0b9039d98"} Oct 30 16:44:57.945665 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:57.945534 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-05810b-67f9c6857b-7jz5n" event={"ID":"6136d90c-0253-4761-843b-f8a3922c93cf","Type":"ContainerStarted","Data":"976f7c568909d2daf7838efbdd580d5dd2280311901f18e59593d5d92cd24411"} Oct 30 16:44:57.945712 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:57.945664 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-05810b-67f9c6857b-7jz5n" Oct 30 16:44:57.995892 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:57.995840 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-05810b-67f9c6857b-7jz5n" podStartSLOduration=1.9958264300000002 podStartE2EDuration="1.99582643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:44:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:44:57.994750051 +0000 UTC m=+1453.522085269" watchObservedRunningTime="2026-10-30 16:44:57.99582643 +0000 UTC m=+1453.523161648" Oct 30 16:44:59.453768 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:59.453737 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-1b7bb3-7777f75787-27lnc"] Oct 30 16:44:59.458734 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:59.458715 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-1b7bb3-7777f75787-27lnc" Oct 30 16:44:59.481056 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:59.481028 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-1b7bb3-7777f75787-27lnc"] Oct 30 16:44:59.494684 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:59.494658 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84028f94-3e19-4900-a29d-31b2d2576c8d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-1b7bb3-7777f75787-27lnc\" (UID: \"84028f94-3e19-4900-a29d-31b2d2576c8d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-1b7bb3-7777f75787-27lnc" Oct 30 16:44:59.494842 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:59.494830 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htg8f\" (UniqueName: \"kubernetes.io/projected/84028f94-3e19-4900-a29d-31b2d2576c8d-kube-api-access-htg8f\") pod \"payload-pre-processing-e2e-shared-b-w5-1b7bb3-7777f75787-27lnc\" (UID: \"84028f94-3e19-4900-a29d-31b2d2576c8d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-1b7bb3-7777f75787-27lnc" Oct 30 16:44:59.546529 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:59.546492 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w5-1b7bb3-dockercfg-s7vqs\"" Oct 30 16:44:59.546783 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:59.546758 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w5-1b7bb3\"" Oct 30 16:44:59.600606 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:59.598897 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htg8f\" (UniqueName: \"kubernetes.io/projected/84028f94-3e19-4900-a29d-31b2d2576c8d-kube-api-access-htg8f\") pod \"payload-pre-processing-e2e-shared-b-w5-1b7bb3-7777f75787-27lnc\" (UID: \"84028f94-3e19-4900-a29d-31b2d2576c8d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-1b7bb3-7777f75787-27lnc" Oct 30 16:44:59.600606 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:59.599195 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84028f94-3e19-4900-a29d-31b2d2576c8d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-1b7bb3-7777f75787-27lnc\" (UID: \"84028f94-3e19-4900-a29d-31b2d2576c8d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-1b7bb3-7777f75787-27lnc" Oct 30 16:44:59.600606 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:59.600310 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84028f94-3e19-4900-a29d-31b2d2576c8d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-1b7bb3-7777f75787-27lnc\" (UID: \"84028f94-3e19-4900-a29d-31b2d2576c8d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-1b7bb3-7777f75787-27lnc" Oct 30 16:44:59.659617 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:59.656009 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htg8f\" (UniqueName: \"kubernetes.io/projected/84028f94-3e19-4900-a29d-31b2d2576c8d-kube-api-access-htg8f\") pod \"payload-pre-processing-e2e-shared-b-w5-1b7bb3-7777f75787-27lnc\" (UID: \"84028f94-3e19-4900-a29d-31b2d2576c8d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-1b7bb3-7777f75787-27lnc" Oct 30 16:44:59.778080 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:44:59.777993 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-1b7bb3-7777f75787-27lnc" Oct 30 16:45:00.249778 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:00.249744 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-1b7bb3-7777f75787-27lnc"] Oct 30 16:45:00.256207 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:45:00.256141 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84028f94_3e19_4900_a29d_31b2d2576c8d.slice/crio-c67cc0efa912d5bab2e7c2602ad171e37bb0b54863ab914f55c7be20d2021a60 WatchSource:0}: Error finding container c67cc0efa912d5bab2e7c2602ad171e37bb0b54863ab914f55c7be20d2021a60: Status 404 returned error can't find the container with id c67cc0efa912d5bab2e7c2602ad171e37bb0b54863ab914f55c7be20d2021a60 Oct 30 16:45:00.611807 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:00.611715 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e4aca49c-7795f84896-nql4z"] Oct 30 16:45:00.632798 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:00.632769 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e4aca49c-c46697f47-wnm2p"] Oct 30 16:45:01.116859 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:01.116809 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-1b7bb3-7777f75787-27lnc" event={"ID":"84028f94-3e19-4900-a29d-31b2d2576c8d","Type":"ContainerStarted","Data":"94e14c6fe7fd0c28e55867508aaf1ef31cc3be74d22c261cd53b03c5c67a9092"} Oct 30 16:45:01.118456 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:01.118425 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-1b7bb3-7777f75787-27lnc" event={"ID":"84028f94-3e19-4900-a29d-31b2d2576c8d","Type":"ContainerStarted","Data":"c67cc0efa912d5bab2e7c2602ad171e37bb0b54863ab914f55c7be20d2021a60"} Oct 30 16:45:01.118778 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:01.118759 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-1b7bb3-7777f75787-27lnc" Oct 30 16:45:01.183602 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:01.175792 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-1b7bb3-7777f75787-27lnc" podStartSLOduration=2.175769464 podStartE2EDuration="2.175769464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:44:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:45:01.15880183 +0000 UTC m=+1456.686137050" watchObservedRunningTime="2026-10-30 16:45:01.175769464 +0000 UTC m=+1456.703104684" Oct 30 16:45:05.613044 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:05.612999 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-e4aca49c-7795f84896-nql4z" podUID="ef0b2556-42cf-4366-a753-68f881b6a02b" containerName="payload-processing" containerID="cri-o://b100bf2b7cb110867b10f5ec44794e7c5be114f7fd85ff978be9c9c0951eaa38" gracePeriod=25 Oct 30 16:45:05.633631 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:05.633552 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-e4aca49c-c46697f47-wnm2p" podUID="6725ae7a-0765-4505-976a-f265e4c026c8" containerName="payload-pre-processing" containerID="cri-o://6e8f844d38bf2158d7a567b292f55186ba2d96b21e44aa6967783d2c328fb24e" gracePeriod=25 Oct 30 16:45:06.078411 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:06.078359 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e4aca49c-7795f84896-nql4z" Oct 30 16:45:06.109784 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:06.109762 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e4aca49c-c46697f47-wnm2p" Oct 30 16:45:06.177657 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:06.177622 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef0b2556-42cf-4366-a753-68f881b6a02b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef0b2556-42cf-4366-a753-68f881b6a02b-plugins-config-volume\") pod \"ef0b2556-42cf-4366-a753-68f881b6a02b\" (UID: \"ef0b2556-42cf-4366-a753-68f881b6a02b\") " Oct 30 16:45:06.178004 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:06.177983 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef0b2556-42cf-4366-a753-68f881b6a02b-kube-api-access-zcff9\" (UniqueName: \"kubernetes.io/projected/ef0b2556-42cf-4366-a753-68f881b6a02b-kube-api-access-zcff9\") pod \"ef0b2556-42cf-4366-a753-68f881b6a02b\" (UID: \"ef0b2556-42cf-4366-a753-68f881b6a02b\") " Oct 30 16:45:06.178320 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:06.178299 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6725ae7a-0765-4505-976a-f265e4c026c8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6725ae7a-0765-4505-976a-f265e4c026c8-plugins-config-volume\") pod \"6725ae7a-0765-4505-976a-f265e4c026c8\" (UID: \"6725ae7a-0765-4505-976a-f265e4c026c8\") " Oct 30 16:45:06.178859 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:06.178832 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6725ae7a-0765-4505-976a-f265e4c026c8-kube-api-access-hhm5k\" (UniqueName: \"kubernetes.io/projected/6725ae7a-0765-4505-976a-f265e4c026c8-kube-api-access-hhm5k\") pod \"6725ae7a-0765-4505-976a-f265e4c026c8\" (UID: \"6725ae7a-0765-4505-976a-f265e4c026c8\") " Oct 30 16:45:06.179712 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:06.178497 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef0b2556-42cf-4366-a753-68f881b6a02b-plugins-config-volume" pod "ef0b2556-42cf-4366-a753-68f881b6a02b" (UID: "ef0b2556-42cf-4366-a753-68f881b6a02b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:45:06.179843 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:06.178819 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6725ae7a-0765-4505-976a-f265e4c026c8-plugins-config-volume" pod "6725ae7a-0765-4505-976a-f265e4c026c8" (UID: "6725ae7a-0765-4505-976a-f265e4c026c8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:45:06.180095 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:06.180066 5052 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef0b2556-42cf-4366-a753-68f881b6a02b-plugins-config-volume\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:45:06.186785 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:06.186760 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6725ae7a-0765-4505-976a-f265e4c026c8-kube-api-access-hhm5k" pod "6725ae7a-0765-4505-976a-f265e4c026c8" (UID: "6725ae7a-0765-4505-976a-f265e4c026c8"). InnerVolumeSpecName "kube-api-access-hhm5k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:45:06.188130 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:06.188104 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef0b2556-42cf-4366-a753-68f881b6a02b-kube-api-access-zcff9" pod "ef0b2556-42cf-4366-a753-68f881b6a02b" (UID: "ef0b2556-42cf-4366-a753-68f881b6a02b"). InnerVolumeSpecName "kube-api-access-zcff9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:45:06.191407 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:06.191376 5052 generic.go:356] "Generic (PLEG): container finished" podID="ef0b2556-42cf-4366-a753-68f881b6a02b" containerID="b100bf2b7cb110867b10f5ec44794e7c5be114f7fd85ff978be9c9c0951eaa38" exitCode=0 Oct 30 16:45:06.191493 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:06.191435 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e4aca49c-7795f84896-nql4z" Oct 30 16:45:06.191493 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:06.191458 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e4aca49c-7795f84896-nql4z" event={"ID":"ef0b2556-42cf-4366-a753-68f881b6a02b","Type":"ContainerDied","Data":"b100bf2b7cb110867b10f5ec44794e7c5be114f7fd85ff978be9c9c0951eaa38"} Oct 30 16:45:06.191655 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:06.191554 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e4aca49c-7795f84896-nql4z" event={"ID":"ef0b2556-42cf-4366-a753-68f881b6a02b","Type":"ContainerDied","Data":"57d978b22cf062e66b3e16407487251dba9c23ce8e6280ca5f10b0dc23084992"} Oct 30 16:45:06.191655 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:06.191604 5052 scope.go:122] "RemoveContainer" containerID="b100bf2b7cb110867b10f5ec44794e7c5be114f7fd85ff978be9c9c0951eaa38" Oct 30 16:45:06.194084 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:06.194064 5052 generic.go:356] "Generic (PLEG): container finished" podID="6725ae7a-0765-4505-976a-f265e4c026c8" containerID="6e8f844d38bf2158d7a567b292f55186ba2d96b21e44aa6967783d2c328fb24e" exitCode=0 Oct 30 16:45:06.194253 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:06.194232 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e4aca49c-c46697f47-wnm2p" Oct 30 16:45:06.194350 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:06.194205 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e4aca49c-c46697f47-wnm2p" event={"ID":"6725ae7a-0765-4505-976a-f265e4c026c8","Type":"ContainerDied","Data":"6e8f844d38bf2158d7a567b292f55186ba2d96b21e44aa6967783d2c328fb24e"} Oct 30 16:45:06.194419 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:06.194379 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e4aca49c-c46697f47-wnm2p" event={"ID":"6725ae7a-0765-4505-976a-f265e4c026c8","Type":"ContainerDied","Data":"8d44c0d91d6642fee75db890f4d51387fbde261d9b9468c32c1d64dd7872e600"} Oct 30 16:45:06.212270 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:06.212253 5052 scope.go:122] "RemoveContainer" containerID="b100bf2b7cb110867b10f5ec44794e7c5be114f7fd85ff978be9c9c0951eaa38" Oct 30 16:45:06.212565 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:45:06.212543 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b100bf2b7cb110867b10f5ec44794e7c5be114f7fd85ff978be9c9c0951eaa38\": container with ID starting with b100bf2b7cb110867b10f5ec44794e7c5be114f7fd85ff978be9c9c0951eaa38 not found: ID does not exist" containerID="b100bf2b7cb110867b10f5ec44794e7c5be114f7fd85ff978be9c9c0951eaa38" Oct 30 16:45:06.212715 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:06.212574 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b100bf2b7cb110867b10f5ec44794e7c5be114f7fd85ff978be9c9c0951eaa38"} err="failed to get container status \"b100bf2b7cb110867b10f5ec44794e7c5be114f7fd85ff978be9c9c0951eaa38\": rpc error: code = NotFound desc = could not find container \"b100bf2b7cb110867b10f5ec44794e7c5be114f7fd85ff978be9c9c0951eaa38\": container with ID starting with b100bf2b7cb110867b10f5ec44794e7c5be114f7fd85ff978be9c9c0951eaa38 not found: ID does not exist" Oct 30 16:45:06.212715 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:06.212639 5052 scope.go:122] "RemoveContainer" containerID="6e8f844d38bf2158d7a567b292f55186ba2d96b21e44aa6967783d2c328fb24e" Oct 30 16:45:06.224241 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:06.224220 5052 scope.go:122] "RemoveContainer" containerID="6e8f844d38bf2158d7a567b292f55186ba2d96b21e44aa6967783d2c328fb24e" Oct 30 16:45:06.224546 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:45:06.224526 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e8f844d38bf2158d7a567b292f55186ba2d96b21e44aa6967783d2c328fb24e\": container with ID starting with 6e8f844d38bf2158d7a567b292f55186ba2d96b21e44aa6967783d2c328fb24e not found: ID does not exist" containerID="6e8f844d38bf2158d7a567b292f55186ba2d96b21e44aa6967783d2c328fb24e" Oct 30 16:45:06.224710 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:06.224555 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e8f844d38bf2158d7a567b292f55186ba2d96b21e44aa6967783d2c328fb24e"} err="failed to get container status \"6e8f844d38bf2158d7a567b292f55186ba2d96b21e44aa6967783d2c328fb24e\": rpc error: code = NotFound desc = could not find container \"6e8f844d38bf2158d7a567b292f55186ba2d96b21e44aa6967783d2c328fb24e\": container with ID starting with 6e8f844d38bf2158d7a567b292f55186ba2d96b21e44aa6967783d2c328fb24e not found: ID does not exist" Oct 30 16:45:06.255312 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:06.255292 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e4aca49c-c46697f47-wnm2p"] Oct 30 16:45:06.263214 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:06.263192 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e4aca49c-c46697f47-wnm2p"] Oct 30 16:45:06.281724 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:06.281700 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcff9\" (UniqueName: \"kubernetes.io/projected/ef0b2556-42cf-4366-a753-68f881b6a02b-kube-api-access-zcff9\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:45:06.281821 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:06.281725 5052 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6725ae7a-0765-4505-976a-f265e4c026c8-plugins-config-volume\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:45:06.281821 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:06.281737 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhm5k\" (UniqueName: \"kubernetes.io/projected/6725ae7a-0765-4505-976a-f265e4c026c8-kube-api-access-hhm5k\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:45:06.296290 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:06.296269 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e4aca49c-7795f84896-nql4z"] Oct 30 16:45:06.300890 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:06.300873 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e4aca49c-7795f84896-nql4z"] Oct 30 16:45:06.599489 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:06.599414 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6725ae7a-0765-4505-976a-f265e4c026c8" path="/var/lib/kubelet/pods/6725ae7a-0765-4505-976a-f265e4c026c8/volumes" Oct 30 16:45:06.599891 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:06.599875 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef0b2556-42cf-4366-a753-68f881b6a02b" path="/var/lib/kubelet/pods/ef0b2556-42cf-4366-a753-68f881b6a02b/volumes" Oct 30 16:45:08.961245 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:08.961212 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-05810b-67f9c6857b-7jz5n" Oct 30 16:45:13.138373 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:13.138278 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-1b7bb3-7777f75787-27lnc" Oct 30 16:45:14.244012 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:14.243975 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-e5bfb0-6f84945dc8-s9w5w"] Oct 30 16:45:14.244961 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:14.244941 5052 cpu_manager.go:414] "containerMap: removing container" podUID="ef0b2556-42cf-4366-a753-68f881b6a02b" containerName="payload-processing" Oct 30 16:45:14.245030 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:14.244964 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef0b2556-42cf-4366-a753-68f881b6a02b" containerName="payload-processing" Oct 30 16:45:14.245030 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:14.244989 5052 cpu_manager.go:414] "containerMap: removing container" podUID="6725ae7a-0765-4505-976a-f265e4c026c8" containerName="payload-pre-processing" Oct 30 16:45:14.245030 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:14.244998 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6725ae7a-0765-4505-976a-f265e4c026c8" containerName="payload-pre-processing" Oct 30 16:45:14.245156 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:14.245145 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef0b2556-42cf-4366-a753-68f881b6a02b" containerName="payload-processing" Oct 30 16:45:14.245197 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:14.245159 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="6725ae7a-0765-4505-976a-f265e4c026c8" containerName="payload-pre-processing" Oct 30 16:45:14.249287 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:14.249268 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-e5bfb0-6f84945dc8-s9w5w" Oct 30 16:45:14.267990 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:14.267960 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-e5bfb0-6f84945dc8-s9w5w"] Oct 30 16:45:14.280177 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:14.280149 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w6-e5bfb0\"" Oct 30 16:45:14.308306 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:14.308255 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/731de2b2-ab6b-499b-a12e-21b7a550ea86-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-e5bfb0-6f84945dc8-s9w5w\" (UID: \"731de2b2-ab6b-499b-a12e-21b7a550ea86\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-e5bfb0-6f84945dc8-s9w5w" Oct 30 16:45:14.308414 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:14.308363 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rr9sf\" (UniqueName: \"kubernetes.io/projected/731de2b2-ab6b-499b-a12e-21b7a550ea86-kube-api-access-rr9sf\") pod \"payload-processing-e2e-shared-b-w6-e5bfb0-6f84945dc8-s9w5w\" (UID: \"731de2b2-ab6b-499b-a12e-21b7a550ea86\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-e5bfb0-6f84945dc8-s9w5w" Oct 30 16:45:14.344345 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:14.344318 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-e5bfb0-86457d6567-v7zt5"] Oct 30 16:45:14.348794 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:14.348776 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-e5bfb0-86457d6567-v7zt5" Oct 30 16:45:14.413063 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:14.413028 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w6-e5bfb0-dockercfg-28wgd\"" Oct 30 16:45:14.477285 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:14.443277 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-e5bfb0-86457d6567-v7zt5"] Oct 30 16:45:14.477566 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:14.445361 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rr9sf\" (UniqueName: \"kubernetes.io/projected/731de2b2-ab6b-499b-a12e-21b7a550ea86-kube-api-access-rr9sf\") pod \"payload-processing-e2e-shared-b-w6-e5bfb0-6f84945dc8-s9w5w\" (UID: \"731de2b2-ab6b-499b-a12e-21b7a550ea86\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-e5bfb0-6f84945dc8-s9w5w" Oct 30 16:45:14.478186 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:14.478160 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/731de2b2-ab6b-499b-a12e-21b7a550ea86-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-e5bfb0-6f84945dc8-s9w5w\" (UID: \"731de2b2-ab6b-499b-a12e-21b7a550ea86\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-e5bfb0-6f84945dc8-s9w5w" Oct 30 16:45:14.482808 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:14.479132 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/731de2b2-ab6b-499b-a12e-21b7a550ea86-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-e5bfb0-6f84945dc8-s9w5w\" (UID: \"731de2b2-ab6b-499b-a12e-21b7a550ea86\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-e5bfb0-6f84945dc8-s9w5w" Oct 30 16:45:14.611665 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:14.571366 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rr9sf\" (UniqueName: \"kubernetes.io/projected/731de2b2-ab6b-499b-a12e-21b7a550ea86-kube-api-access-rr9sf\") pod \"payload-processing-e2e-shared-b-w6-e5bfb0-6f84945dc8-s9w5w\" (UID: \"731de2b2-ab6b-499b-a12e-21b7a550ea86\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-e5bfb0-6f84945dc8-s9w5w" Oct 30 16:45:14.695360 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:14.695260 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4tnc\" (UniqueName: \"kubernetes.io/projected/72b342d0-d3cc-43c3-8013-ef235a934f3b-kube-api-access-k4tnc\") pod \"payload-pre-processing-e2e-shared-b-w6-e5bfb0-86457d6567-v7zt5\" (UID: \"72b342d0-d3cc-43c3-8013-ef235a934f3b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-e5bfb0-86457d6567-v7zt5" Oct 30 16:45:14.696051 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:14.696009 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72b342d0-d3cc-43c3-8013-ef235a934f3b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-e5bfb0-86457d6567-v7zt5\" (UID: \"72b342d0-d3cc-43c3-8013-ef235a934f3b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-e5bfb0-86457d6567-v7zt5" Oct 30 16:45:14.698572 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:14.698546 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-e5bfb0-6f84945dc8-s9w5w" Oct 30 16:45:14.819818 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:14.819306 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4tnc\" (UniqueName: \"kubernetes.io/projected/72b342d0-d3cc-43c3-8013-ef235a934f3b-kube-api-access-k4tnc\") pod \"payload-pre-processing-e2e-shared-b-w6-e5bfb0-86457d6567-v7zt5\" (UID: \"72b342d0-d3cc-43c3-8013-ef235a934f3b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-e5bfb0-86457d6567-v7zt5" Oct 30 16:45:14.819818 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:14.819496 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72b342d0-d3cc-43c3-8013-ef235a934f3b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-e5bfb0-86457d6567-v7zt5\" (UID: \"72b342d0-d3cc-43c3-8013-ef235a934f3b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-e5bfb0-86457d6567-v7zt5" Oct 30 16:45:14.824802 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:14.824725 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72b342d0-d3cc-43c3-8013-ef235a934f3b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-e5bfb0-86457d6567-v7zt5\" (UID: \"72b342d0-d3cc-43c3-8013-ef235a934f3b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-e5bfb0-86457d6567-v7zt5" Oct 30 16:45:14.843730 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:14.843692 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4tnc\" (UniqueName: \"kubernetes.io/projected/72b342d0-d3cc-43c3-8013-ef235a934f3b-kube-api-access-k4tnc\") pod \"payload-pre-processing-e2e-shared-b-w6-e5bfb0-86457d6567-v7zt5\" (UID: \"72b342d0-d3cc-43c3-8013-ef235a934f3b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-e5bfb0-86457d6567-v7zt5" Oct 30 16:45:14.963686 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:14.963647 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-e5bfb0-86457d6567-v7zt5" Oct 30 16:45:15.124298 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:15.124201 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-e5bfb0-6f84945dc8-s9w5w"] Oct 30 16:45:15.274704 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:15.274672 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-e5bfb0-86457d6567-v7zt5"] Oct 30 16:45:15.276619 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:45:15.276589 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72b342d0_d3cc_43c3_8013_ef235a934f3b.slice/crio-4cc5fc714bab705bc0b60bd087b07c73c9903fdf94cab9bbe80f67bb40b15ba2 WatchSource:0}: Error finding container 4cc5fc714bab705bc0b60bd087b07c73c9903fdf94cab9bbe80f67bb40b15ba2: Status 404 returned error can't find the container with id 4cc5fc714bab705bc0b60bd087b07c73c9903fdf94cab9bbe80f67bb40b15ba2 Oct 30 16:45:15.342649 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:15.342621 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-05810b-dc6664f5c-dtrsg"] Oct 30 16:45:15.349697 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:15.349678 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-05810b-dc6664f5c-dtrsg" Oct 30 16:45:15.368099 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:15.368076 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-05810b-dc6664f5c-dtrsg"] Oct 30 16:45:15.399791 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:15.399761 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-05810b-serving-cert\"" Oct 30 16:45:15.576600 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:15.549095 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7z9z\" (UniqueName: \"kubernetes.io/projected/e9258d14-705c-40d3-8898-48d08dcf7846-kube-api-access-p7z9z\") pod \"maas-api-e2e-shared-a-w2-05810b-dc6664f5c-dtrsg\" (UID: \"e9258d14-705c-40d3-8898-48d08dcf7846\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-05810b-dc6664f5c-dtrsg" Oct 30 16:45:15.576940 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:15.576906 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9258d14-705c-40d3-8898-48d08dcf7846-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-05810b-dc6664f5c-dtrsg\" (UID: \"e9258d14-705c-40d3-8898-48d08dcf7846\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-05810b-dc6664f5c-dtrsg" Oct 30 16:45:15.577184 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:15.577157 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9258d14-705c-40d3-8898-48d08dcf7846-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-05810b-dc6664f5c-dtrsg\" (UID: \"e9258d14-705c-40d3-8898-48d08dcf7846\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-05810b-dc6664f5c-dtrsg" Oct 30 16:45:15.577549 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:15.577517 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9258d14-705c-40d3-8898-48d08dcf7846-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-05810b-dc6664f5c-dtrsg\" (UID: \"e9258d14-705c-40d3-8898-48d08dcf7846\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-05810b-dc6664f5c-dtrsg" Oct 30 16:45:15.680664 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:15.680458 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7z9z\" (UniqueName: \"kubernetes.io/projected/e9258d14-705c-40d3-8898-48d08dcf7846-kube-api-access-p7z9z\") pod \"maas-api-e2e-shared-a-w2-05810b-dc6664f5c-dtrsg\" (UID: \"e9258d14-705c-40d3-8898-48d08dcf7846\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-05810b-dc6664f5c-dtrsg" Oct 30 16:45:15.680664 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:15.680517 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9258d14-705c-40d3-8898-48d08dcf7846-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-05810b-dc6664f5c-dtrsg\" (UID: \"e9258d14-705c-40d3-8898-48d08dcf7846\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-05810b-dc6664f5c-dtrsg" Oct 30 16:45:15.680664 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:15.680561 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9258d14-705c-40d3-8898-48d08dcf7846-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-05810b-dc6664f5c-dtrsg\" (UID: \"e9258d14-705c-40d3-8898-48d08dcf7846\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-05810b-dc6664f5c-dtrsg" Oct 30 16:45:15.685598 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:15.681522 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9258d14-705c-40d3-8898-48d08dcf7846-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-05810b-dc6664f5c-dtrsg\" (UID: \"e9258d14-705c-40d3-8898-48d08dcf7846\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-05810b-dc6664f5c-dtrsg" Oct 30 16:45:15.685598 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:15.681803 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9258d14-705c-40d3-8898-48d08dcf7846-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-05810b-dc6664f5c-dtrsg\" (UID: \"e9258d14-705c-40d3-8898-48d08dcf7846\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-05810b-dc6664f5c-dtrsg" Oct 30 16:45:15.685598 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:45:15.682058 5052 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-05810b-serving-cert: secret "maas-api-e2e-shared-a-w2-05810b-serving-cert" not found Oct 30 16:45:15.685598 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:45:15.682129 5052 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9258d14-705c-40d3-8898-48d08dcf7846-maas-api-tls podName:e9258d14-705c-40d3-8898-48d08dcf7846 nodeName:}" failed. No retries permitted until 2026-10-30 16:45:16.182099999 +0000 UTC m=+1471.709435210 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e9258d14-705c-40d3-8898-48d08dcf7846-maas-api-tls") pod "maas-api-e2e-shared-a-w2-05810b-dc6664f5c-dtrsg" (UID: "e9258d14-705c-40d3-8898-48d08dcf7846") : secret "maas-api-e2e-shared-a-w2-05810b-serving-cert" not found Oct 30 16:45:15.702640 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:15.696146 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9258d14-705c-40d3-8898-48d08dcf7846-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-05810b-dc6664f5c-dtrsg\" (UID: \"e9258d14-705c-40d3-8898-48d08dcf7846\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-05810b-dc6664f5c-dtrsg" Oct 30 16:45:15.710526 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:15.710502 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7z9z\" (UniqueName: \"kubernetes.io/projected/e9258d14-705c-40d3-8898-48d08dcf7846-kube-api-access-p7z9z\") pod \"maas-api-e2e-shared-a-w2-05810b-dc6664f5c-dtrsg\" (UID: \"e9258d14-705c-40d3-8898-48d08dcf7846\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-05810b-dc6664f5c-dtrsg" Oct 30 16:45:15.837040 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:15.837005 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-e5bfb0-6f84945dc8-s9w5w" event={"ID":"731de2b2-ab6b-499b-a12e-21b7a550ea86","Type":"ContainerStarted","Data":"9b128d66989128941673582c034464e89983bf1d1394b7b2a189515dc22f85b0"} Oct 30 16:45:15.837222 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:15.837063 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-e5bfb0-6f84945dc8-s9w5w" event={"ID":"731de2b2-ab6b-499b-a12e-21b7a550ea86","Type":"ContainerStarted","Data":"18925aa9090b8a36a10cc74e4c8d647f5447b6ad9a2f30e9c35cd0bc9624f736"} Oct 30 16:45:15.837222 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:15.837089 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-e5bfb0-6f84945dc8-s9w5w" Oct 30 16:45:15.838646 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:15.838620 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-e5bfb0-86457d6567-v7zt5" event={"ID":"72b342d0-d3cc-43c3-8013-ef235a934f3b","Type":"ContainerStarted","Data":"6f8a74af7c300932373f5973ace89ca43d6806bc45da9f4a264beeffecd593be"} Oct 30 16:45:15.838774 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:15.838662 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-e5bfb0-86457d6567-v7zt5" event={"ID":"72b342d0-d3cc-43c3-8013-ef235a934f3b","Type":"ContainerStarted","Data":"4cc5fc714bab705bc0b60bd087b07c73c9903fdf94cab9bbe80f67bb40b15ba2"} Oct 30 16:45:15.838774 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:15.838712 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-e5bfb0-86457d6567-v7zt5" Oct 30 16:45:15.880430 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:15.880385 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-e5bfb0-6f84945dc8-s9w5w" podStartSLOduration=1.880372342 podStartE2EDuration="1.880372342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:45:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:45:15.876010807 +0000 UTC m=+1471.403346026" watchObservedRunningTime="2026-10-30 16:45:15.880372342 +0000 UTC m=+1471.407707561" Oct 30 16:45:15.918326 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:15.918282 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-e5bfb0-86457d6567-v7zt5" podStartSLOduration=1.918269091 podStartE2EDuration="1.918269091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:45:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:45:15.914984007 +0000 UTC m=+1471.442319227" watchObservedRunningTime="2026-10-30 16:45:15.918269091 +0000 UTC m=+1471.445604309" Oct 30 16:45:16.194233 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:16.194190 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9258d14-705c-40d3-8898-48d08dcf7846-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-05810b-dc6664f5c-dtrsg\" (UID: \"e9258d14-705c-40d3-8898-48d08dcf7846\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-05810b-dc6664f5c-dtrsg" Oct 30 16:45:16.196881 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:16.196858 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9258d14-705c-40d3-8898-48d08dcf7846-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-05810b-dc6664f5c-dtrsg\" (UID: \"e9258d14-705c-40d3-8898-48d08dcf7846\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-05810b-dc6664f5c-dtrsg" Oct 30 16:45:16.277700 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:16.267944 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-05810b-dc6664f5c-dtrsg" Oct 30 16:45:16.634549 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:16.634524 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-05810b-dc6664f5c-dtrsg"] Oct 30 16:45:16.636509 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:45:16.636480 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9258d14_705c_40d3_8898_48d08dcf7846.slice/crio-21b5240c7bf721922b4a7bc75f925334d30cc125b592ecfb258e351cb68fa111 WatchSource:0}: Error finding container 21b5240c7bf721922b4a7bc75f925334d30cc125b592ecfb258e351cb68fa111: Status 404 returned error can't find the container with id 21b5240c7bf721922b4a7bc75f925334d30cc125b592ecfb258e351cb68fa111 Oct 30 16:45:16.856301 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:16.856262 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-05810b-dc6664f5c-dtrsg" event={"ID":"e9258d14-705c-40d3-8898-48d08dcf7846","Type":"ContainerStarted","Data":"21b5240c7bf721922b4a7bc75f925334d30cc125b592ecfb258e351cb68fa111"} Oct 30 16:45:17.864890 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:17.864857 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-05810b-dc6664f5c-dtrsg" event={"ID":"e9258d14-705c-40d3-8898-48d08dcf7846","Type":"ContainerStarted","Data":"f0ae2c0f6d1986a6a9cd3e99fe10268b2a79f28548d710ae98ab06f9fa5d90ae"} Oct 30 16:45:17.865273 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:17.865031 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-05810b-dc6664f5c-dtrsg" Oct 30 16:45:17.910087 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:17.910038 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-05810b-dc6664f5c-dtrsg" podStartSLOduration=2.508346565 podStartE2EDuration="2.910024905s" totalImagesPullingTime="401.67834ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:45:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:45:17.907244901 +0000 UTC m=+1473.434580122" watchObservedRunningTime="2026-10-30 16:45:17.910024905 +0000 UTC m=+1473.437360124" Oct 30 16:45:18.841566 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:45:18.841524 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 16:45:18.979035 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:18.978999 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-dc74d5-5d4d946fcf-hnnsn"] Oct 30 16:45:18.984221 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:18.984202 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-dc74d5-5d4d946fcf-hnnsn" Oct 30 16:45:19.008417 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:19.008389 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-dc74d5-5d4d946fcf-hnnsn"] Oct 30 16:45:19.009059 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:19.009033 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-dc74d5-serving-cert\"" Oct 30 16:45:19.112509 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:19.112422 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/552a6bdf-c418-47bd-b72b-20e311760c33-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-dc74d5-5d4d946fcf-hnnsn\" (UID: \"552a6bdf-c418-47bd-b72b-20e311760c33\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-dc74d5-5d4d946fcf-hnnsn" Oct 30 16:45:19.112785 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:19.112762 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gjjv\" (UniqueName: \"kubernetes.io/projected/552a6bdf-c418-47bd-b72b-20e311760c33-kube-api-access-2gjjv\") pod \"maas-api-e2e-shared-a-w3-dc74d5-5d4d946fcf-hnnsn\" (UID: \"552a6bdf-c418-47bd-b72b-20e311760c33\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-dc74d5-5d4d946fcf-hnnsn" Oct 30 16:45:19.112864 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:19.112837 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/552a6bdf-c418-47bd-b72b-20e311760c33-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-dc74d5-5d4d946fcf-hnnsn\" (UID: \"552a6bdf-c418-47bd-b72b-20e311760c33\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-dc74d5-5d4d946fcf-hnnsn" Oct 30 16:45:19.112930 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:19.112872 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/552a6bdf-c418-47bd-b72b-20e311760c33-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-dc74d5-5d4d946fcf-hnnsn\" (UID: \"552a6bdf-c418-47bd-b72b-20e311760c33\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-dc74d5-5d4d946fcf-hnnsn" Oct 30 16:45:19.214215 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:19.214171 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gjjv\" (UniqueName: \"kubernetes.io/projected/552a6bdf-c418-47bd-b72b-20e311760c33-kube-api-access-2gjjv\") pod \"maas-api-e2e-shared-a-w3-dc74d5-5d4d946fcf-hnnsn\" (UID: \"552a6bdf-c418-47bd-b72b-20e311760c33\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-dc74d5-5d4d946fcf-hnnsn" Oct 30 16:45:19.214401 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:19.214257 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/552a6bdf-c418-47bd-b72b-20e311760c33-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-dc74d5-5d4d946fcf-hnnsn\" (UID: \"552a6bdf-c418-47bd-b72b-20e311760c33\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-dc74d5-5d4d946fcf-hnnsn" Oct 30 16:45:19.214401 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:19.214287 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/552a6bdf-c418-47bd-b72b-20e311760c33-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-dc74d5-5d4d946fcf-hnnsn\" (UID: \"552a6bdf-c418-47bd-b72b-20e311760c33\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-dc74d5-5d4d946fcf-hnnsn" Oct 30 16:45:19.214484 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:19.214410 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/552a6bdf-c418-47bd-b72b-20e311760c33-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-dc74d5-5d4d946fcf-hnnsn\" (UID: \"552a6bdf-c418-47bd-b72b-20e311760c33\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-dc74d5-5d4d946fcf-hnnsn" Oct 30 16:45:19.214938 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:19.214915 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/552a6bdf-c418-47bd-b72b-20e311760c33-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-dc74d5-5d4d946fcf-hnnsn\" (UID: \"552a6bdf-c418-47bd-b72b-20e311760c33\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-dc74d5-5d4d946fcf-hnnsn" Oct 30 16:45:19.217139 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:19.217097 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/552a6bdf-c418-47bd-b72b-20e311760c33-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-dc74d5-5d4d946fcf-hnnsn\" (UID: \"552a6bdf-c418-47bd-b72b-20e311760c33\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-dc74d5-5d4d946fcf-hnnsn" Oct 30 16:45:19.217699 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:19.217570 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/552a6bdf-c418-47bd-b72b-20e311760c33-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-dc74d5-5d4d946fcf-hnnsn\" (UID: \"552a6bdf-c418-47bd-b72b-20e311760c33\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-dc74d5-5d4d946fcf-hnnsn" Oct 30 16:45:19.242040 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:19.242007 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gjjv\" (UniqueName: \"kubernetes.io/projected/552a6bdf-c418-47bd-b72b-20e311760c33-kube-api-access-2gjjv\") pod \"maas-api-e2e-shared-a-w3-dc74d5-5d4d946fcf-hnnsn\" (UID: \"552a6bdf-c418-47bd-b72b-20e311760c33\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-dc74d5-5d4d946fcf-hnnsn" Oct 30 16:45:19.300928 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:19.300886 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-dc74d5-5d4d946fcf-hnnsn" Oct 30 16:45:19.717464 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:19.717437 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-dc74d5-5d4d946fcf-hnnsn"] Oct 30 16:45:19.719803 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:45:19.719780 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod552a6bdf_c418_47bd_b72b_20e311760c33.slice/crio-efd05df02fd826768d6271fe66b4acf5f3ebadde31d5d13b66ca4e1d0a3f10e2 WatchSource:0}: Error finding container efd05df02fd826768d6271fe66b4acf5f3ebadde31d5d13b66ca4e1d0a3f10e2: Status 404 returned error can't find the container with id efd05df02fd826768d6271fe66b4acf5f3ebadde31d5d13b66ca4e1d0a3f10e2 Oct 30 16:45:19.882210 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:19.882172 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-dc74d5-5d4d946fcf-hnnsn" event={"ID":"552a6bdf-c418-47bd-b72b-20e311760c33","Type":"ContainerStarted","Data":"efd05df02fd826768d6271fe66b4acf5f3ebadde31d5d13b66ca4e1d0a3f10e2"} Oct 30 16:45:21.043427 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:21.043389 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-dc74d5-5d4d946fcf-hnnsn" event={"ID":"552a6bdf-c418-47bd-b72b-20e311760c33","Type":"ContainerStarted","Data":"14301e35732ec99a59f5f5454559d03c59d1a96eaf16724b574b719757cdd933"} Oct 30 16:45:21.043975 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:21.043958 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-dc74d5-5d4d946fcf-hnnsn" Oct 30 16:45:21.087971 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:21.087919 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-dc74d5-5d4d946fcf-hnnsn" podStartSLOduration=2.7313385610000003 podStartE2EDuration="3.087905086s" totalImagesPullingTime="356.566525ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:45:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:45:21.083121544 +0000 UTC m=+1476.610456763" watchObservedRunningTime="2026-10-30 16:45:21.087905086 +0000 UTC m=+1476.615240304" Oct 30 16:45:23.180494 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:23.180458 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-1b7bb3-7c468f545b-nlwlv"] Oct 30 16:45:23.185497 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:23.185477 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-1b7bb3-7c468f545b-nlwlv" Oct 30 16:45:23.205188 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:23.205160 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-1b7bb3-7c468f545b-nlwlv"] Oct 30 16:45:23.228312 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:23.228286 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w5-1b7bb3-serving-cert\"" Oct 30 16:45:23.280365 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:23.280334 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7700cd37-52fa-4210-afa9-7771f758a8ba-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-1b7bb3-7c468f545b-nlwlv\" (UID: \"7700cd37-52fa-4210-afa9-7771f758a8ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-1b7bb3-7c468f545b-nlwlv" Oct 30 16:45:23.280365 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:23.280364 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7700cd37-52fa-4210-afa9-7771f758a8ba-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-1b7bb3-7c468f545b-nlwlv\" (UID: \"7700cd37-52fa-4210-afa9-7771f758a8ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-1b7bb3-7c468f545b-nlwlv" Oct 30 16:45:23.280677 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:23.280487 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7700cd37-52fa-4210-afa9-7771f758a8ba-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-1b7bb3-7c468f545b-nlwlv\" (UID: \"7700cd37-52fa-4210-afa9-7771f758a8ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-1b7bb3-7c468f545b-nlwlv" Oct 30 16:45:23.280677 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:23.280665 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crkc6\" (UniqueName: \"kubernetes.io/projected/7700cd37-52fa-4210-afa9-7771f758a8ba-kube-api-access-crkc6\") pod \"maas-api-e2e-shared-b-w5-1b7bb3-7c468f545b-nlwlv\" (UID: \"7700cd37-52fa-4210-afa9-7771f758a8ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-1b7bb3-7c468f545b-nlwlv" Oct 30 16:45:23.382393 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:23.382364 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7700cd37-52fa-4210-afa9-7771f758a8ba-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-1b7bb3-7c468f545b-nlwlv\" (UID: \"7700cd37-52fa-4210-afa9-7771f758a8ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-1b7bb3-7c468f545b-nlwlv" Oct 30 16:45:23.382393 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:23.382395 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7700cd37-52fa-4210-afa9-7771f758a8ba-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-1b7bb3-7c468f545b-nlwlv\" (UID: \"7700cd37-52fa-4210-afa9-7771f758a8ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-1b7bb3-7c468f545b-nlwlv" Oct 30 16:45:23.382676 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:23.382445 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7700cd37-52fa-4210-afa9-7771f758a8ba-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-1b7bb3-7c468f545b-nlwlv\" (UID: \"7700cd37-52fa-4210-afa9-7771f758a8ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-1b7bb3-7c468f545b-nlwlv" Oct 30 16:45:23.382676 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:45:23.382526 5052 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-1b7bb3-serving-cert: secret "maas-api-e2e-shared-b-w5-1b7bb3-serving-cert" not found Oct 30 16:45:23.382676 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:45:23.382625 5052 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7700cd37-52fa-4210-afa9-7771f758a8ba-maas-api-tls podName:7700cd37-52fa-4210-afa9-7771f758a8ba nodeName:}" failed. No retries permitted until 2026-10-30 16:45:23.882574701 +0000 UTC m=+1479.409909899 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7700cd37-52fa-4210-afa9-7771f758a8ba-maas-api-tls") pod "maas-api-e2e-shared-b-w5-1b7bb3-7c468f545b-nlwlv" (UID: "7700cd37-52fa-4210-afa9-7771f758a8ba") : secret "maas-api-e2e-shared-b-w5-1b7bb3-serving-cert" not found Oct 30 16:45:23.382676 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:23.382645 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crkc6\" (UniqueName: \"kubernetes.io/projected/7700cd37-52fa-4210-afa9-7771f758a8ba-kube-api-access-crkc6\") pod \"maas-api-e2e-shared-b-w5-1b7bb3-7c468f545b-nlwlv\" (UID: \"7700cd37-52fa-4210-afa9-7771f758a8ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-1b7bb3-7c468f545b-nlwlv" Oct 30 16:45:23.383357 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:23.383328 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7700cd37-52fa-4210-afa9-7771f758a8ba-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-1b7bb3-7c468f545b-nlwlv\" (UID: \"7700cd37-52fa-4210-afa9-7771f758a8ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-1b7bb3-7c468f545b-nlwlv" Oct 30 16:45:23.385175 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:23.385151 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7700cd37-52fa-4210-afa9-7771f758a8ba-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-1b7bb3-7c468f545b-nlwlv\" (UID: \"7700cd37-52fa-4210-afa9-7771f758a8ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-1b7bb3-7c468f545b-nlwlv" Oct 30 16:45:23.469655 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:23.445516 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crkc6\" (UniqueName: \"kubernetes.io/projected/7700cd37-52fa-4210-afa9-7771f758a8ba-kube-api-access-crkc6\") pod \"maas-api-e2e-shared-b-w5-1b7bb3-7c468f545b-nlwlv\" (UID: \"7700cd37-52fa-4210-afa9-7771f758a8ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-1b7bb3-7c468f545b-nlwlv" Oct 30 16:45:23.877683 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:23.877570 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-05810b-dc6664f5c-dtrsg" Oct 30 16:45:23.904292 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:23.904254 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7700cd37-52fa-4210-afa9-7771f758a8ba-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-1b7bb3-7c468f545b-nlwlv\" (UID: \"7700cd37-52fa-4210-afa9-7771f758a8ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-1b7bb3-7c468f545b-nlwlv" Oct 30 16:45:23.906718 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:23.906694 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7700cd37-52fa-4210-afa9-7771f758a8ba-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-1b7bb3-7c468f545b-nlwlv\" (UID: \"7700cd37-52fa-4210-afa9-7771f758a8ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-1b7bb3-7c468f545b-nlwlv" Oct 30 16:45:24.109614 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:24.108760 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-1b7bb3-7c468f545b-nlwlv" Oct 30 16:45:24.504064 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:24.504033 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-1b7bb3-7c468f545b-nlwlv"] Oct 30 16:45:24.505367 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:45:24.505329 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7700cd37_52fa_4210_afa9_7771f758a8ba.slice/crio-4efabf9fd01572b6c6a721c60106e12a1282beae3c5a851cb7a737f991376a60 WatchSource:0}: Error finding container 4efabf9fd01572b6c6a721c60106e12a1282beae3c5a851cb7a737f991376a60: Status 404 returned error can't find the container with id 4efabf9fd01572b6c6a721c60106e12a1282beae3c5a851cb7a737f991376a60 Oct 30 16:45:25.221680 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:25.221643 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-1b7bb3-7c468f545b-nlwlv" event={"ID":"7700cd37-52fa-4210-afa9-7771f758a8ba","Type":"ContainerStarted","Data":"913bb27ba70458c0c723c4fc44d50cc7add23bd167caca6c4d2cd5bb28f1240b"} Oct 30 16:45:25.221848 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:25.221707 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-1b7bb3-7c468f545b-nlwlv" event={"ID":"7700cd37-52fa-4210-afa9-7771f758a8ba","Type":"ContainerStarted","Data":"4efabf9fd01572b6c6a721c60106e12a1282beae3c5a851cb7a737f991376a60"} Oct 30 16:45:25.221848 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:25.221747 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-1b7bb3-7c468f545b-nlwlv" Oct 30 16:45:26.860050 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:26.860020 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-e5bfb0-86457d6567-v7zt5" Oct 30 16:45:26.860757 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:26.860737 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-e5bfb0-6f84945dc8-s9w5w" Oct 30 16:45:26.904488 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:26.904438 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-1b7bb3-7c468f545b-nlwlv" podStartSLOduration=3.490882722 podStartE2EDuration="3.904425547s" totalImagesPullingTime="413.542825ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:45:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:45:25.284772701 +0000 UTC m=+1480.812107917" watchObservedRunningTime="2026-10-30 16:45:26.904425547 +0000 UTC m=+1482.431760765" Oct 30 16:45:27.072758 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:27.072728 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-dc74d5-5d4d946fcf-hnnsn" Oct 30 16:45:31.234444 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:31.234413 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-1b7bb3-7c468f545b-nlwlv" Oct 30 16:45:38.955510 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:38.955470 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e5bfb0-8486ccf4c4-kqwsv"] Oct 30 16:45:38.964869 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:38.964841 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e5bfb0-8486ccf4c4-kqwsv" Oct 30 16:45:38.984377 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:38.984349 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w6-e5bfb0-serving-cert\"" Oct 30 16:45:38.989901 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:38.989875 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e5bfb0-8486ccf4c4-kqwsv"] Oct 30 16:45:39.067704 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:39.067651 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0562b1aa-d0da-4e5e-ba02-2042715e7b35-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-e5bfb0-8486ccf4c4-kqwsv\" (UID: \"0562b1aa-d0da-4e5e-ba02-2042715e7b35\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e5bfb0-8486ccf4c4-kqwsv" Oct 30 16:45:39.067908 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:39.067741 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0562b1aa-d0da-4e5e-ba02-2042715e7b35-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-e5bfb0-8486ccf4c4-kqwsv\" (UID: \"0562b1aa-d0da-4e5e-ba02-2042715e7b35\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e5bfb0-8486ccf4c4-kqwsv" Oct 30 16:45:39.068062 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:39.068036 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0562b1aa-d0da-4e5e-ba02-2042715e7b35-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-e5bfb0-8486ccf4c4-kqwsv\" (UID: \"0562b1aa-d0da-4e5e-ba02-2042715e7b35\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e5bfb0-8486ccf4c4-kqwsv" Oct 30 16:45:39.068200 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:39.068185 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7dtt\" (UniqueName: \"kubernetes.io/projected/0562b1aa-d0da-4e5e-ba02-2042715e7b35-kube-api-access-j7dtt\") pod \"maas-api-e2e-shared-b-w6-e5bfb0-8486ccf4c4-kqwsv\" (UID: \"0562b1aa-d0da-4e5e-ba02-2042715e7b35\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e5bfb0-8486ccf4c4-kqwsv" Oct 30 16:45:39.070291 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:39.070264 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cdwcj"] Oct 30 16:45:39.169698 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:39.169656 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7dtt\" (UniqueName: \"kubernetes.io/projected/0562b1aa-d0da-4e5e-ba02-2042715e7b35-kube-api-access-j7dtt\") pod \"maas-api-e2e-shared-b-w6-e5bfb0-8486ccf4c4-kqwsv\" (UID: \"0562b1aa-d0da-4e5e-ba02-2042715e7b35\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e5bfb0-8486ccf4c4-kqwsv" Oct 30 16:45:39.170429 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:39.170398 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0562b1aa-d0da-4e5e-ba02-2042715e7b35-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-e5bfb0-8486ccf4c4-kqwsv\" (UID: \"0562b1aa-d0da-4e5e-ba02-2042715e7b35\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e5bfb0-8486ccf4c4-kqwsv" Oct 30 16:45:39.171538 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:39.171424 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0562b1aa-d0da-4e5e-ba02-2042715e7b35-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-e5bfb0-8486ccf4c4-kqwsv\" (UID: \"0562b1aa-d0da-4e5e-ba02-2042715e7b35\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e5bfb0-8486ccf4c4-kqwsv" Oct 30 16:45:39.173792 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:39.171334 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0562b1aa-d0da-4e5e-ba02-2042715e7b35-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-e5bfb0-8486ccf4c4-kqwsv\" (UID: \"0562b1aa-d0da-4e5e-ba02-2042715e7b35\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e5bfb0-8486ccf4c4-kqwsv" Oct 30 16:45:39.174031 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:39.174007 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0562b1aa-d0da-4e5e-ba02-2042715e7b35-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-e5bfb0-8486ccf4c4-kqwsv\" (UID: \"0562b1aa-d0da-4e5e-ba02-2042715e7b35\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e5bfb0-8486ccf4c4-kqwsv" Oct 30 16:45:39.175212 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:39.175186 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0562b1aa-d0da-4e5e-ba02-2042715e7b35-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-e5bfb0-8486ccf4c4-kqwsv\" (UID: \"0562b1aa-d0da-4e5e-ba02-2042715e7b35\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e5bfb0-8486ccf4c4-kqwsv" Oct 30 16:45:39.193260 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:39.193235 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7dtt\" (UniqueName: \"kubernetes.io/projected/0562b1aa-d0da-4e5e-ba02-2042715e7b35-kube-api-access-j7dtt\") pod \"maas-api-e2e-shared-b-w6-e5bfb0-8486ccf4c4-kqwsv\" (UID: \"0562b1aa-d0da-4e5e-ba02-2042715e7b35\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e5bfb0-8486ccf4c4-kqwsv" Oct 30 16:45:39.194476 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:39.194453 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0562b1aa-d0da-4e5e-ba02-2042715e7b35-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-e5bfb0-8486ccf4c4-kqwsv\" (UID: \"0562b1aa-d0da-4e5e-ba02-2042715e7b35\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e5bfb0-8486ccf4c4-kqwsv" Oct 30 16:45:39.413450 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:39.413413 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e5bfb0-8486ccf4c4-kqwsv" Oct 30 16:45:40.011612 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:40.007702 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e5bfb0-8486ccf4c4-kqwsv"] Oct 30 16:45:40.580239 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:40.580181 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e5bfb0-8486ccf4c4-kqwsv" event={"ID":"0562b1aa-d0da-4e5e-ba02-2042715e7b35","Type":"ContainerStarted","Data":"ee78c96aca15c2398ac3c007d73f8fdfb9fa319e10f0b82a1c214defd2d870bd"} Oct 30 16:45:41.624765 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:41.621800 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e5bfb0-8486ccf4c4-kqwsv" event={"ID":"0562b1aa-d0da-4e5e-ba02-2042715e7b35","Type":"ContainerStarted","Data":"35196dbf053d6615411723a65ad229bd92c2de53f9603b2988032fd58936133a"} Oct 30 16:45:41.624765 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:41.622924 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e5bfb0-8486ccf4c4-kqwsv" Oct 30 16:45:41.676117 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:41.676049 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e5bfb0-8486ccf4c4-kqwsv" podStartSLOduration=3.14944785 podStartE2EDuration="3.676030357s" totalImagesPullingTime="526.582507ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:45:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:45:41.674902109 +0000 UTC m=+1497.202237334" watchObservedRunningTime="2026-10-30 16:45:41.676030357 +0000 UTC m=+1497.203365576" Oct 30 16:45:45.347930 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:45.347902 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4wrf8_4f0f46f4-5cda-4d67-b6f3-47df06eb844c/console-operator/2.log" Oct 30 16:45:45.355295 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:45.355272 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4wrf8_4f0f46f4-5cda-4d67-b6f3-47df06eb844c/console-operator/2.log" Oct 30 16:45:48.687800 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:48.687765 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e5bfb0-8486ccf4c4-kqwsv" Oct 30 16:45:48.846033 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:45:48.845993 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 16:45:49.715908 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:49.715875 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-tjv7v"] Oct 30 16:45:49.764632 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:49.761944 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-tjv7v" Oct 30 16:45:49.809481 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:49.809444 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-tjv7v"] Oct 30 16:45:49.852240 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:49.851990 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-05810b-openshift-default-dockercfg-cf5d9\"" Oct 30 16:45:49.947056 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:49.947016 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39f0bd01-e73f-4f31-a76e-ad6788799f3f-istio-data\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-tjv7v\" (UID: \"39f0bd01-e73f-4f31-a76e-ad6788799f3f\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-tjv7v" Oct 30 16:45:49.947056 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:49.947054 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39f0bd01-e73f-4f31-a76e-ad6788799f3f-istio-token\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-tjv7v\" (UID: \"39f0bd01-e73f-4f31-a76e-ad6788799f3f\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-tjv7v" Oct 30 16:45:49.947275 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:49.947243 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqw26\" (UniqueName: \"kubernetes.io/projected/39f0bd01-e73f-4f31-a76e-ad6788799f3f-kube-api-access-nqw26\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-tjv7v\" (UID: \"39f0bd01-e73f-4f31-a76e-ad6788799f3f\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-tjv7v" Oct 30 16:45:49.947369 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:49.947352 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39f0bd01-e73f-4f31-a76e-ad6788799f3f-istio-envoy\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-tjv7v\" (UID: \"39f0bd01-e73f-4f31-a76e-ad6788799f3f\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-tjv7v" Oct 30 16:45:49.947442 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:49.947425 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39f0bd01-e73f-4f31-a76e-ad6788799f3f-credential-socket\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-tjv7v\" (UID: \"39f0bd01-e73f-4f31-a76e-ad6788799f3f\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-tjv7v" Oct 30 16:45:49.947504 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:49.947472 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39f0bd01-e73f-4f31-a76e-ad6788799f3f-istiod-ca-cert\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-tjv7v\" (UID: \"39f0bd01-e73f-4f31-a76e-ad6788799f3f\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-tjv7v" Oct 30 16:45:49.947566 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:49.947511 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39f0bd01-e73f-4f31-a76e-ad6788799f3f-workload-socket\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-tjv7v\" (UID: \"39f0bd01-e73f-4f31-a76e-ad6788799f3f\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-tjv7v" Oct 30 16:45:49.947716 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:49.947700 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39f0bd01-e73f-4f31-a76e-ad6788799f3f-istio-podinfo\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-tjv7v\" (UID: \"39f0bd01-e73f-4f31-a76e-ad6788799f3f\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-tjv7v" Oct 30 16:45:49.947755 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:49.947734 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39f0bd01-e73f-4f31-a76e-ad6788799f3f-workload-certs\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-tjv7v\" (UID: \"39f0bd01-e73f-4f31-a76e-ad6788799f3f\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-tjv7v" Oct 30 16:45:50.049466 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:50.049377 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39f0bd01-e73f-4f31-a76e-ad6788799f3f-credential-socket\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-tjv7v\" (UID: \"39f0bd01-e73f-4f31-a76e-ad6788799f3f\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-tjv7v" Oct 30 16:45:50.049466 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:50.049436 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39f0bd01-e73f-4f31-a76e-ad6788799f3f-istiod-ca-cert\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-tjv7v\" (UID: \"39f0bd01-e73f-4f31-a76e-ad6788799f3f\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-tjv7v" Oct 30 16:45:50.049738 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:50.049488 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39f0bd01-e73f-4f31-a76e-ad6788799f3f-workload-socket\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-tjv7v\" (UID: \"39f0bd01-e73f-4f31-a76e-ad6788799f3f\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-tjv7v" Oct 30 16:45:50.049738 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:50.049632 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39f0bd01-e73f-4f31-a76e-ad6788799f3f-istio-podinfo\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-tjv7v\" (UID: \"39f0bd01-e73f-4f31-a76e-ad6788799f3f\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-tjv7v" Oct 30 16:45:50.049738 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:50.049671 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39f0bd01-e73f-4f31-a76e-ad6788799f3f-workload-certs\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-tjv7v\" (UID: \"39f0bd01-e73f-4f31-a76e-ad6788799f3f\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-tjv7v" Oct 30 16:45:50.049893 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:50.049788 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39f0bd01-e73f-4f31-a76e-ad6788799f3f-istio-data\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-tjv7v\" (UID: \"39f0bd01-e73f-4f31-a76e-ad6788799f3f\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-tjv7v" Oct 30 16:45:50.049893 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:50.049815 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39f0bd01-e73f-4f31-a76e-ad6788799f3f-istio-token\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-tjv7v\" (UID: \"39f0bd01-e73f-4f31-a76e-ad6788799f3f\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-tjv7v" Oct 30 16:45:50.049996 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:50.049825 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39f0bd01-e73f-4f31-a76e-ad6788799f3f-credential-socket\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-tjv7v\" (UID: \"39f0bd01-e73f-4f31-a76e-ad6788799f3f\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-tjv7v" Oct 30 16:45:50.049996 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:50.049861 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39f0bd01-e73f-4f31-a76e-ad6788799f3f-workload-socket\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-tjv7v\" (UID: \"39f0bd01-e73f-4f31-a76e-ad6788799f3f\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-tjv7v" Oct 30 16:45:50.050093 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:50.049937 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqw26\" (UniqueName: \"kubernetes.io/projected/39f0bd01-e73f-4f31-a76e-ad6788799f3f-kube-api-access-nqw26\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-tjv7v\" (UID: \"39f0bd01-e73f-4f31-a76e-ad6788799f3f\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-tjv7v" Oct 30 16:45:50.050093 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:50.050032 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39f0bd01-e73f-4f31-a76e-ad6788799f3f-workload-certs\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-tjv7v\" (UID: \"39f0bd01-e73f-4f31-a76e-ad6788799f3f\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-tjv7v" Oct 30 16:45:50.050208 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:50.050100 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39f0bd01-e73f-4f31-a76e-ad6788799f3f-istio-data\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-tjv7v\" (UID: \"39f0bd01-e73f-4f31-a76e-ad6788799f3f\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-tjv7v" Oct 30 16:45:50.050208 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:50.050135 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39f0bd01-e73f-4f31-a76e-ad6788799f3f-istio-envoy\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-tjv7v\" (UID: \"39f0bd01-e73f-4f31-a76e-ad6788799f3f\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-tjv7v" Oct 30 16:45:50.050310 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:50.050237 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39f0bd01-e73f-4f31-a76e-ad6788799f3f-istiod-ca-cert\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-tjv7v\" (UID: \"39f0bd01-e73f-4f31-a76e-ad6788799f3f\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-tjv7v" Oct 30 16:45:50.052156 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:50.052128 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39f0bd01-e73f-4f31-a76e-ad6788799f3f-istio-envoy\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-tjv7v\" (UID: \"39f0bd01-e73f-4f31-a76e-ad6788799f3f\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-tjv7v" Oct 30 16:45:50.052274 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:50.052136 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39f0bd01-e73f-4f31-a76e-ad6788799f3f-istio-podinfo\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-tjv7v\" (UID: \"39f0bd01-e73f-4f31-a76e-ad6788799f3f\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-tjv7v" Oct 30 16:45:50.073232 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:50.073203 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39f0bd01-e73f-4f31-a76e-ad6788799f3f-istio-token\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-tjv7v\" (UID: \"39f0bd01-e73f-4f31-a76e-ad6788799f3f\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-tjv7v" Oct 30 16:45:50.073342 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:50.073210 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqw26\" (UniqueName: \"kubernetes.io/projected/39f0bd01-e73f-4f31-a76e-ad6788799f3f-kube-api-access-nqw26\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-tjv7v\" (UID: \"39f0bd01-e73f-4f31-a76e-ad6788799f3f\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-tjv7v" Oct 30 16:45:50.155111 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:50.155071 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-tjv7v" Oct 30 16:45:50.482449 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:50.482412 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-tjv7v"] Oct 30 16:45:50.484312 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:45:50.484282 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39f0bd01_e73f_4f31_a76e_ad6788799f3f.slice/crio-fb9e1b9e24e357792a51ee099bdd070911d70d891f05d5db65745f6a5adc3aa9 WatchSource:0}: Error finding container fb9e1b9e24e357792a51ee099bdd070911d70d891f05d5db65745f6a5adc3aa9: Status 404 returned error can't find the container with id fb9e1b9e24e357792a51ee099bdd070911d70d891f05d5db65745f6a5adc3aa9 Oct 30 16:45:50.487256 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:50.487225 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:45:50.487342 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:50.487292 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:45:50.487342 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:50.487321 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:45:51.006067 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:51.006019 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-tjv7v" event={"ID":"39f0bd01-e73f-4f31-a76e-ad6788799f3f","Type":"ContainerStarted","Data":"afb120f27cfb49af3561b6bddbcabecc7a0086ef4bac62ffe8af92505f99290d"} Oct 30 16:45:51.020796 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:51.020748 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-tjv7v" event={"ID":"39f0bd01-e73f-4f31-a76e-ad6788799f3f","Type":"ContainerStarted","Data":"fb9e1b9e24e357792a51ee099bdd070911d70d891f05d5db65745f6a5adc3aa9"} Oct 30 16:45:51.144390 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:51.144299 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-tjv7v" podStartSLOduration=2.14427608 podStartE2EDuration="2.14427608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:45:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:45:51.122150633 +0000 UTC m=+1506.649485854" watchObservedRunningTime="2026-10-30 16:45:51.14427608 +0000 UTC m=+1506.671611303" Oct 30 16:45:51.159759 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:51.159718 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-tjv7v" Oct 30 16:45:51.163708 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:51.163661 5052 patch_prober.go:29] interesting pod/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-tjv7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Oct 30 16:45:51.163995 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:51.163967 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-tjv7v" podUID="39f0bd01-e73f-4f31-a76e-ad6788799f3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Oct 30 16:45:52.172606 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:52.165943 5052 patch_prober.go:29] interesting pod/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-tjv7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Oct 30 16:45:52.172606 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:52.166095 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-tjv7v" podUID="39f0bd01-e73f-4f31-a76e-ad6788799f3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Oct 30 16:45:53.157609 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:53.157001 5052 patch_prober.go:29] interesting pod/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-tjv7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Oct 30 16:45:53.157609 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:53.157084 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-tjv7v" podUID="39f0bd01-e73f-4f31-a76e-ad6788799f3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Oct 30 16:45:53.214896 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:53.214862 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-x4nqz"] Oct 30 16:45:53.224177 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:53.223517 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-x4nqz" Oct 30 16:45:53.269809 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:53.268406 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w5-1b7bb3-openshift-default-dockercfg-gqvnj\"" Oct 30 16:45:53.294156 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:53.294111 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-x4nqz"] Oct 30 16:45:53.394132 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:53.394089 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dbfcef1-dea9-4360-abd2-d489a86811a3-workload-socket\") pod \"e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-x4nqz\" (UID: \"8dbfcef1-dea9-4360-abd2-d489a86811a3\") " pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-x4nqz" Oct 30 16:45:53.394325 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:53.394147 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8dbfcef1-dea9-4360-abd2-d489a86811a3-istio-data\") pod \"e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-x4nqz\" (UID: \"8dbfcef1-dea9-4360-abd2-d489a86811a3\") " pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-x4nqz" Oct 30 16:45:53.394325 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:53.394274 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8dbfcef1-dea9-4360-abd2-d489a86811a3-istio-token\") pod \"e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-x4nqz\" (UID: \"8dbfcef1-dea9-4360-abd2-d489a86811a3\") " pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-x4nqz" Oct 30 16:45:53.394422 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:53.394340 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8dbfcef1-dea9-4360-abd2-d489a86811a3-istio-envoy\") pod \"e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-x4nqz\" (UID: \"8dbfcef1-dea9-4360-abd2-d489a86811a3\") " pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-x4nqz" Oct 30 16:45:53.394488 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:53.394469 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8dbfcef1-dea9-4360-abd2-d489a86811a3-istiod-ca-cert\") pod \"e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-x4nqz\" (UID: \"8dbfcef1-dea9-4360-abd2-d489a86811a3\") " pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-x4nqz" Oct 30 16:45:53.394530 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:53.394511 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8dbfcef1-dea9-4360-abd2-d489a86811a3-istio-podinfo\") pod \"e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-x4nqz\" (UID: \"8dbfcef1-dea9-4360-abd2-d489a86811a3\") " pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-x4nqz" Oct 30 16:45:53.394571 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:53.394542 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8dbfcef1-dea9-4360-abd2-d489a86811a3-workload-certs\") pod \"e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-x4nqz\" (UID: \"8dbfcef1-dea9-4360-abd2-d489a86811a3\") " pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-x4nqz" Oct 30 16:45:53.394637 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:53.394577 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnj7j\" (UniqueName: \"kubernetes.io/projected/8dbfcef1-dea9-4360-abd2-d489a86811a3-kube-api-access-fnj7j\") pod \"e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-x4nqz\" (UID: \"8dbfcef1-dea9-4360-abd2-d489a86811a3\") " pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-x4nqz" Oct 30 16:45:53.394766 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:53.394750 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dbfcef1-dea9-4360-abd2-d489a86811a3-credential-socket\") pod \"e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-x4nqz\" (UID: \"8dbfcef1-dea9-4360-abd2-d489a86811a3\") " pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-x4nqz" Oct 30 16:45:53.496531 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:53.496452 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dbfcef1-dea9-4360-abd2-d489a86811a3-workload-socket\") pod \"e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-x4nqz\" (UID: \"8dbfcef1-dea9-4360-abd2-d489a86811a3\") " pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-x4nqz" Oct 30 16:45:53.496531 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:53.496494 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8dbfcef1-dea9-4360-abd2-d489a86811a3-istio-data\") pod \"e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-x4nqz\" (UID: \"8dbfcef1-dea9-4360-abd2-d489a86811a3\") " pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-x4nqz" Oct 30 16:45:53.496753 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:53.496614 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8dbfcef1-dea9-4360-abd2-d489a86811a3-istio-token\") pod \"e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-x4nqz\" (UID: \"8dbfcef1-dea9-4360-abd2-d489a86811a3\") " pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-x4nqz" Oct 30 16:45:53.496753 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:53.496673 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8dbfcef1-dea9-4360-abd2-d489a86811a3-istio-envoy\") pod \"e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-x4nqz\" (UID: \"8dbfcef1-dea9-4360-abd2-d489a86811a3\") " pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-x4nqz" Oct 30 16:45:53.496861 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:53.496823 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8dbfcef1-dea9-4360-abd2-d489a86811a3-istiod-ca-cert\") pod \"e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-x4nqz\" (UID: \"8dbfcef1-dea9-4360-abd2-d489a86811a3\") " pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-x4nqz" Oct 30 16:45:53.496861 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:53.496855 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8dbfcef1-dea9-4360-abd2-d489a86811a3-istio-podinfo\") pod \"e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-x4nqz\" (UID: \"8dbfcef1-dea9-4360-abd2-d489a86811a3\") " pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-x4nqz" Oct 30 16:45:53.496970 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:53.496898 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8dbfcef1-dea9-4360-abd2-d489a86811a3-workload-certs\") pod \"e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-x4nqz\" (UID: \"8dbfcef1-dea9-4360-abd2-d489a86811a3\") " pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-x4nqz" Oct 30 16:45:53.496970 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:53.496954 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dbfcef1-dea9-4360-abd2-d489a86811a3-workload-socket\") pod \"e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-x4nqz\" (UID: \"8dbfcef1-dea9-4360-abd2-d489a86811a3\") " pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-x4nqz" Oct 30 16:45:53.497074 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:53.497008 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnj7j\" (UniqueName: \"kubernetes.io/projected/8dbfcef1-dea9-4360-abd2-d489a86811a3-kube-api-access-fnj7j\") pod \"e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-x4nqz\" (UID: \"8dbfcef1-dea9-4360-abd2-d489a86811a3\") " pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-x4nqz" Oct 30 16:45:53.497190 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:53.497170 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8dbfcef1-dea9-4360-abd2-d489a86811a3-workload-certs\") pod \"e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-x4nqz\" (UID: \"8dbfcef1-dea9-4360-abd2-d489a86811a3\") " pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-x4nqz" Oct 30 16:45:53.497314 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:53.497175 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dbfcef1-dea9-4360-abd2-d489a86811a3-credential-socket\") pod \"e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-x4nqz\" (UID: \"8dbfcef1-dea9-4360-abd2-d489a86811a3\") " pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-x4nqz" Oct 30 16:45:53.497398 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:53.497372 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8dbfcef1-dea9-4360-abd2-d489a86811a3-istio-data\") pod \"e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-x4nqz\" (UID: \"8dbfcef1-dea9-4360-abd2-d489a86811a3\") " pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-x4nqz" Oct 30 16:45:53.497469 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:53.497448 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dbfcef1-dea9-4360-abd2-d489a86811a3-credential-socket\") pod \"e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-x4nqz\" (UID: \"8dbfcef1-dea9-4360-abd2-d489a86811a3\") " pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-x4nqz" Oct 30 16:45:53.497620 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:53.497564 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8dbfcef1-dea9-4360-abd2-d489a86811a3-istiod-ca-cert\") pod \"e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-x4nqz\" (UID: \"8dbfcef1-dea9-4360-abd2-d489a86811a3\") " pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-x4nqz" Oct 30 16:45:53.499305 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:53.499279 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8dbfcef1-dea9-4360-abd2-d489a86811a3-istio-envoy\") pod \"e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-x4nqz\" (UID: \"8dbfcef1-dea9-4360-abd2-d489a86811a3\") " pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-x4nqz" Oct 30 16:45:53.499410 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:53.499393 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8dbfcef1-dea9-4360-abd2-d489a86811a3-istio-podinfo\") pod \"e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-x4nqz\" (UID: \"8dbfcef1-dea9-4360-abd2-d489a86811a3\") " pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-x4nqz" Oct 30 16:45:53.519367 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:53.519339 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8dbfcef1-dea9-4360-abd2-d489a86811a3-istio-token\") pod \"e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-x4nqz\" (UID: \"8dbfcef1-dea9-4360-abd2-d489a86811a3\") " pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-x4nqz" Oct 30 16:45:53.519478 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:53.519436 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnj7j\" (UniqueName: \"kubernetes.io/projected/8dbfcef1-dea9-4360-abd2-d489a86811a3-kube-api-access-fnj7j\") pod \"e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-x4nqz\" (UID: \"8dbfcef1-dea9-4360-abd2-d489a86811a3\") " pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-x4nqz" Oct 30 16:45:53.622602 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:53.612876 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-x4nqz" Oct 30 16:45:54.090934 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:54.090854 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-x4nqz"] Oct 30 16:45:54.105929 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:45:54.105905 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dbfcef1_dea9_4360_abd2_d489a86811a3.slice/crio-4101738a563d18fd1e226d6a9ed4ef0b5f5bced184232780f03a640d20408bc3 WatchSource:0}: Error finding container 4101738a563d18fd1e226d6a9ed4ef0b5f5bced184232780f03a640d20408bc3: Status 404 returned error can't find the container with id 4101738a563d18fd1e226d6a9ed4ef0b5f5bced184232780f03a640d20408bc3 Oct 30 16:45:54.113058 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:54.113021 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:45:54.113157 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:54.113096 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:45:54.113157 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:54.113125 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:45:54.155749 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:54.155711 5052 patch_prober.go:29] interesting pod/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-tjv7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Oct 30 16:45:54.155930 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:54.155793 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-tjv7v" podUID="39f0bd01-e73f-4f31-a76e-ad6788799f3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Oct 30 16:45:55.095798 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:55.094924 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-x4nqz" event={"ID":"8dbfcef1-dea9-4360-abd2-d489a86811a3","Type":"ContainerStarted","Data":"11aa7be13d10e30dc7825024f06221482480bb9e2b6d913788ba250ef8577020"} Oct 30 16:45:55.095798 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:55.094997 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-x4nqz" event={"ID":"8dbfcef1-dea9-4360-abd2-d489a86811a3","Type":"ContainerStarted","Data":"4101738a563d18fd1e226d6a9ed4ef0b5f5bced184232780f03a640d20408bc3"} Oct 30 16:45:55.170602 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:55.169675 5052 patch_prober.go:29] interesting pod/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-tjv7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Oct 30 16:45:55.170602 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:55.169731 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-tjv7v" podUID="39f0bd01-e73f-4f31-a76e-ad6788799f3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Oct 30 16:45:55.191504 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:55.191433 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-x4nqz" podStartSLOduration=2.1914149529999998 podStartE2EDuration="2.191414953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:45:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:45:55.168812244 +0000 UTC m=+1510.696147464" watchObservedRunningTime="2026-10-30 16:45:55.191414953 +0000 UTC m=+1510.718750175" Oct 30 16:45:55.647606 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:55.623437 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-x4nqz" Oct 30 16:45:55.648896 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:55.648206 5052 patch_prober.go:29] interesting pod/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-x4nqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Oct 30 16:45:55.648896 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:55.648268 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-x4nqz" podUID="8dbfcef1-dea9-4360-abd2-d489a86811a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Oct 30 16:45:56.158126 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:56.158079 5052 patch_prober.go:29] interesting pod/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-tjv7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Oct 30 16:45:56.158748 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:56.158701 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-tjv7v" podUID="39f0bd01-e73f-4f31-a76e-ad6788799f3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Oct 30 16:45:56.614543 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:56.614464 5052 patch_prober.go:29] interesting pod/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-x4nqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Oct 30 16:45:56.614543 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:56.614518 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-x4nqz" podUID="8dbfcef1-dea9-4360-abd2-d489a86811a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Oct 30 16:45:57.186838 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:57.186801 5052 patch_prober.go:29] interesting pod/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-tjv7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Oct 30 16:45:57.187457 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:57.187423 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-tjv7v" podUID="39f0bd01-e73f-4f31-a76e-ad6788799f3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Oct 30 16:45:57.629615 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:57.620792 5052 patch_prober.go:29] interesting pod/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-x4nqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Oct 30 16:45:57.629615 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:57.620851 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-x4nqz" podUID="8dbfcef1-dea9-4360-abd2-d489a86811a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Oct 30 16:45:58.168424 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:58.167809 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-tjv7v" Oct 30 16:45:58.168715 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:58.168693 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-tjv7v" Oct 30 16:45:58.169759 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:58.169721 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-tjv7v" Oct 30 16:45:58.634069 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:58.633979 5052 patch_prober.go:29] interesting pod/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-x4nqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Oct 30 16:45:58.651720 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:58.651661 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-x4nqz" podUID="8dbfcef1-dea9-4360-abd2-d489a86811a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Oct 30 16:45:59.614384 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:59.614351 5052 patch_prober.go:29] interesting pod/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-x4nqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Oct 30 16:45:59.614566 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:45:59.614401 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-x4nqz" podUID="8dbfcef1-dea9-4360-abd2-d489a86811a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Oct 30 16:46:00.613883 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:00.613851 5052 patch_prober.go:29] interesting pod/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-x4nqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Oct 30 16:46:00.614264 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:00.613898 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-x4nqz" podUID="8dbfcef1-dea9-4360-abd2-d489a86811a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Oct 30 16:46:01.619347 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:01.619316 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-x4nqz" Oct 30 16:46:01.619779 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:01.619760 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-x4nqz" Oct 30 16:46:01.620372 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:01.620354 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-x4nqz" Oct 30 16:46:13.635605 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:13.632216 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cdwcj"] Oct 30 16:46:14.775804 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:14.775761 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8968-1aa1d0"] Oct 30 16:46:14.782600 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:14.782553 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8968-1aa1d0" Oct 30 16:46:14.796349 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:14.796321 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8968-1aa1d0"] Oct 30 16:46:14.838720 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:14.834478 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-xr4c7\"" Oct 30 16:46:14.911912 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:14.911868 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdxqx\" (UniqueName: \"kubernetes.io/projected/3f06bb44-8a96-4687-9901-e509308386c4-kube-api-access-cdxqx\") pod \"test-curl-8968-1aa1d0\" (UID: \"3f06bb44-8a96-4687-9901-e509308386c4\") " pod="opendatahub/test-curl-8968-1aa1d0" Oct 30 16:46:15.165606 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:15.164078 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdxqx\" (UniqueName: \"kubernetes.io/projected/3f06bb44-8a96-4687-9901-e509308386c4-kube-api-access-cdxqx\") pod \"test-curl-8968-1aa1d0\" (UID: \"3f06bb44-8a96-4687-9901-e509308386c4\") " pod="opendatahub/test-curl-8968-1aa1d0" Oct 30 16:46:15.244093 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:15.244059 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdxqx\" (UniqueName: \"kubernetes.io/projected/3f06bb44-8a96-4687-9901-e509308386c4-kube-api-access-cdxqx\") pod \"test-curl-8968-1aa1d0\" (UID: \"3f06bb44-8a96-4687-9901-e509308386c4\") " pod="opendatahub/test-curl-8968-1aa1d0" Oct 30 16:46:15.400152 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:15.400116 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8968-1aa1d0" Oct 30 16:46:17.274428 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:17.254769 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d693d0/distinct-2-w6-d693d0-kserve-6498fdc474-dpbjj" podUID="4b98b8ff-72cd-4ddb-a4b8-67078162e611" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:46:17.292953 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:17.275654 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/distinct-2-w4-263f00-kserve-8464897cf5-q8ck8" podUID="15ed43ef-5cdd-489a-a70e-063239822ff3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 30 16:46:17.336566 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:17.336526 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-750dcb/distinct-w2-750dcb-kserve-86bbdfcbb-bpn77" podUID="c45190af-d795-4e13-8f2f-670b2c597658" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:46:17.488908 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:17.471394 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8968-1aa1d0"] Oct 30 16:46:17.489838 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:17.489802 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8968-1aa1d0" event={"ID":"3f06bb44-8a96-4687-9901-e509308386c4","Type":"ContainerStarted","Data":"d4246daf258a8b51b00f37cb31785af5d0133af9e40067373e4911ff6929e736"} Oct 30 16:46:17.860603 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:17.851396 5052 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-d693d0/distinct-2-w6-d693d0-kserve-6498fdc474-dpbjj" podUID="4b98b8ff-72cd-4ddb-a4b8-67078162e611" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:46:18.359668 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:18.354046 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8968-1aa1d0" event={"ID":"3f06bb44-8a96-4687-9901-e509308386c4","Type":"ContainerStarted","Data":"4b6b706055178f49f862760803becfe8bfc30cc6c7e6eb37c3dfc5a6aecf1086"} Oct 30 16:46:18.453572 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:18.453508 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8968-1aa1d0" podStartSLOduration=4.453490545 podStartE2EDuration="4.453490545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:46:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:46:18.422192217 +0000 UTC m=+1533.949527441" watchObservedRunningTime="2026-10-30 16:46:18.453490545 +0000 UTC m=+1533.980825765" Oct 30 16:46:18.628897 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:18.628816 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b315d4-84cb4fb687-9466n"] Oct 30 16:46:18.633856 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:18.633833 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b315d4-84cb4fb687-9466n" Oct 30 16:46:18.652196 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:18.652168 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-b315d4\"" Oct 30 16:46:18.652827 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:18.652466 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-b315d4-dockercfg-9fjqf\"" Oct 30 16:46:18.658455 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:18.658428 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b315d4-84cb4fb687-9466n"] Oct 30 16:46:18.831922 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:18.818195 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a626d0b1-3ca3-49a1-ab32-dde876466afe-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-b315d4-84cb4fb687-9466n\" (UID: \"a626d0b1-3ca3-49a1-ab32-dde876466afe\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b315d4-84cb4fb687-9466n" Oct 30 16:46:18.884833 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:18.884703 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27c2v\" (UniqueName: \"kubernetes.io/projected/a626d0b1-3ca3-49a1-ab32-dde876466afe-kube-api-access-27c2v\") pod \"payload-pre-processing-e2e-api-a-b315d4-84cb4fb687-9466n\" (UID: \"a626d0b1-3ca3-49a1-ab32-dde876466afe\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b315d4-84cb4fb687-9466n" Oct 30 16:46:18.988167 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:18.988122 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a626d0b1-3ca3-49a1-ab32-dde876466afe-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-b315d4-84cb4fb687-9466n\" (UID: \"a626d0b1-3ca3-49a1-ab32-dde876466afe\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b315d4-84cb4fb687-9466n" Oct 30 16:46:18.991607 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:18.988372 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27c2v\" (UniqueName: \"kubernetes.io/projected/a626d0b1-3ca3-49a1-ab32-dde876466afe-kube-api-access-27c2v\") pod \"payload-pre-processing-e2e-api-a-b315d4-84cb4fb687-9466n\" (UID: \"a626d0b1-3ca3-49a1-ab32-dde876466afe\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b315d4-84cb4fb687-9466n" Oct 30 16:46:19.038472 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:19.038442 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a626d0b1-3ca3-49a1-ab32-dde876466afe-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-b315d4-84cb4fb687-9466n\" (UID: \"a626d0b1-3ca3-49a1-ab32-dde876466afe\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b315d4-84cb4fb687-9466n" Oct 30 16:46:19.086123 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:19.086028 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27c2v\" (UniqueName: \"kubernetes.io/projected/a626d0b1-3ca3-49a1-ab32-dde876466afe-kube-api-access-27c2v\") pod \"payload-pre-processing-e2e-api-a-b315d4-84cb4fb687-9466n\" (UID: \"a626d0b1-3ca3-49a1-ab32-dde876466afe\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b315d4-84cb4fb687-9466n" Oct 30 16:46:19.187629 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:46:19.187563 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 16:46:19.252635 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:19.252422 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b315d4-84cb4fb687-9466n" Oct 30 16:46:19.569153 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:19.569125 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b315d4-84cb4fb687-9466n"] Oct 30 16:46:19.570083 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:46:19.570046 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda626d0b1_3ca3_49a1_ab32_dde876466afe.slice/crio-d5082b56c17936408caf2205792229686844f2df1adfc55e6ca28fd0af7cad84 WatchSource:0}: Error finding container d5082b56c17936408caf2205792229686844f2df1adfc55e6ca28fd0af7cad84: Status 404 returned error can't find the container with id d5082b56c17936408caf2205792229686844f2df1adfc55e6ca28fd0af7cad84 Oct 30 16:46:19.825917 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:19.825785 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8968-1aa1d0"] Oct 30 16:46:19.831013 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:19.830982 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8968-1aa1d0"] Oct 30 16:46:20.031605 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:20.031341 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8968-42007b"] Oct 30 16:46:20.036669 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:20.032675 5052 cpu_manager.go:414] "containerMap: removing container" podUID="3f06bb44-8a96-4687-9901-e509308386c4" containerName="test-curl-8968-1aa1d0" Oct 30 16:46:20.036669 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:20.032702 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f06bb44-8a96-4687-9901-e509308386c4" containerName="test-curl-8968-1aa1d0" Oct 30 16:46:20.036669 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:20.032934 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f06bb44-8a96-4687-9901-e509308386c4" containerName="test-curl-8968-1aa1d0" Oct 30 16:46:20.043213 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:20.042995 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8968-42007b" Oct 30 16:46:20.049078 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:20.047106 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8968-42007b"] Oct 30 16:46:20.148125 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:20.148098 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thpz7\" (UniqueName: \"kubernetes.io/projected/738da916-49d4-4378-a88b-3f9572860dcd-kube-api-access-thpz7\") pod \"test-curl-8968-42007b\" (UID: \"738da916-49d4-4378-a88b-3f9572860dcd\") " pod="opendatahub/test-curl-8968-42007b" Oct 30 16:46:20.260605 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:20.259230 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thpz7\" (UniqueName: \"kubernetes.io/projected/738da916-49d4-4378-a88b-3f9572860dcd-kube-api-access-thpz7\") pod \"test-curl-8968-42007b\" (UID: \"738da916-49d4-4378-a88b-3f9572860dcd\") " pod="opendatahub/test-curl-8968-42007b" Oct 30 16:46:20.308625 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:20.306340 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thpz7\" (UniqueName: \"kubernetes.io/projected/738da916-49d4-4378-a88b-3f9572860dcd-kube-api-access-thpz7\") pod \"test-curl-8968-42007b\" (UID: \"738da916-49d4-4378-a88b-3f9572860dcd\") " pod="opendatahub/test-curl-8968-42007b" Oct 30 16:46:20.368442 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:20.368410 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8968-42007b" Oct 30 16:46:20.376658 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:20.376623 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b315d4-84cb4fb687-9466n" event={"ID":"a626d0b1-3ca3-49a1-ab32-dde876466afe","Type":"ContainerStarted","Data":"7b70434e3d66c791346ec315a70ca9e411d9ffc4c8c0a71955a03eae2604d7da"} Oct 30 16:46:20.376794 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:20.376687 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b315d4-84cb4fb687-9466n" event={"ID":"a626d0b1-3ca3-49a1-ab32-dde876466afe","Type":"ContainerStarted","Data":"d5082b56c17936408caf2205792229686844f2df1adfc55e6ca28fd0af7cad84"} Oct 30 16:46:20.376794 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:20.376733 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b315d4-84cb4fb687-9466n" Oct 30 16:46:20.376986 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:20.376963 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8968-1aa1d0" podUID="3f06bb44-8a96-4687-9901-e509308386c4" containerName="test-curl-8968-1aa1d0" containerID="cri-o://4b6b706055178f49f862760803becfe8bfc30cc6c7e6eb37c3dfc5a6aecf1086" gracePeriod=2 Oct 30 16:46:20.420196 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:20.420097 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b315d4-84cb4fb687-9466n" podStartSLOduration=2.420084177 podStartE2EDuration="2.420084177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:46:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:46:20.415805033 +0000 UTC m=+1535.943140252" watchObservedRunningTime="2026-10-30 16:46:20.420084177 +0000 UTC m=+1535.947419395" Oct 30 16:46:20.428654 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:46:20.428624 5052 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8968-1aa1d0\" is forbidden: User \"system:node:ip-10-0-131-136.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-136.ec2.internal' and this object" podUID="3f06bb44-8a96-4687-9901-e509308386c4" pod="opendatahub/test-curl-8968-1aa1d0" Oct 30 16:46:20.715851 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:20.715827 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8968-42007b"] Oct 30 16:46:20.718127 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:46:20.718082 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod738da916_49d4_4378_a88b_3f9572860dcd.slice/crio-0d4fc78389a1f8f0db94cc7d6bc2b1f59c23469f03fda363b86d7361e607745a WatchSource:0}: Error finding container 0d4fc78389a1f8f0db94cc7d6bc2b1f59c23469f03fda363b86d7361e607745a: Status 404 returned error can't find the container with id 0d4fc78389a1f8f0db94cc7d6bc2b1f59c23469f03fda363b86d7361e607745a Oct 30 16:46:21.388332 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:21.388292 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8968-42007b" event={"ID":"738da916-49d4-4378-a88b-3f9572860dcd","Type":"ContainerStarted","Data":"3ab328c940b31026bf6e0b12b2c53062bfb5e847ab190d2d61a250a91da2ac7b"} Oct 30 16:46:21.388500 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:21.388348 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8968-42007b" event={"ID":"738da916-49d4-4378-a88b-3f9572860dcd","Type":"ContainerStarted","Data":"0d4fc78389a1f8f0db94cc7d6bc2b1f59c23469f03fda363b86d7361e607745a"} Oct 30 16:46:21.428540 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:21.428489 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8968-42007b" podStartSLOduration=2.428475655 podStartE2EDuration="2.428475655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:46:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:46:21.424789472 +0000 UTC m=+1536.952124692" watchObservedRunningTime="2026-10-30 16:46:21.428475655 +0000 UTC m=+1536.955810874" Oct 30 16:46:22.388063 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:22.388029 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8968-42007b"] Oct 30 16:46:22.400722 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:22.400681 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8968-42007b"] Oct 30 16:46:22.408664 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:22.408630 5052 generic.go:356] "Generic (PLEG): container finished" podID="3f06bb44-8a96-4687-9901-e509308386c4" containerID="4b6b706055178f49f862760803becfe8bfc30cc6c7e6eb37c3dfc5a6aecf1086" exitCode=137 Oct 30 16:46:22.547697 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:22.547665 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8968-efdfc3"] Oct 30 16:46:22.548855 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:22.548836 5052 cpu_manager.go:414] "containerMap: removing container" podUID="738da916-49d4-4378-a88b-3f9572860dcd" containerName="test-curl-8968-42007b" Oct 30 16:46:22.548919 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:22.548857 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="738da916-49d4-4378-a88b-3f9572860dcd" containerName="test-curl-8968-42007b" Oct 30 16:46:22.549066 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:22.549054 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="738da916-49d4-4378-a88b-3f9572860dcd" containerName="test-curl-8968-42007b" Oct 30 16:46:22.553266 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:22.553247 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8968-efdfc3" Oct 30 16:46:22.566830 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:22.566806 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8968-efdfc3"] Oct 30 16:46:22.608105 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:22.608079 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8968-1aa1d0" Oct 30 16:46:22.711681 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:22.711630 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f06bb44-8a96-4687-9901-e509308386c4-kube-api-access-cdxqx\" (UniqueName: \"kubernetes.io/projected/3f06bb44-8a96-4687-9901-e509308386c4-kube-api-access-cdxqx\") pod \"3f06bb44-8a96-4687-9901-e509308386c4\" (UID: \"3f06bb44-8a96-4687-9901-e509308386c4\") " Oct 30 16:46:22.712296 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:22.712272 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbcw2\" (UniqueName: \"kubernetes.io/projected/40a41028-8743-4f22-b989-ae80435a8cfa-kube-api-access-sbcw2\") pod \"test-curl-8968-efdfc3\" (UID: \"40a41028-8743-4f22-b989-ae80435a8cfa\") " pod="opendatahub/test-curl-8968-efdfc3" Oct 30 16:46:22.713667 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:22.713642 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f06bb44-8a96-4687-9901-e509308386c4-kube-api-access-cdxqx" pod "3f06bb44-8a96-4687-9901-e509308386c4" (UID: "3f06bb44-8a96-4687-9901-e509308386c4"). InnerVolumeSpecName "kube-api-access-cdxqx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:46:22.814010 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:22.813979 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbcw2\" (UniqueName: \"kubernetes.io/projected/40a41028-8743-4f22-b989-ae80435a8cfa-kube-api-access-sbcw2\") pod \"test-curl-8968-efdfc3\" (UID: \"40a41028-8743-4f22-b989-ae80435a8cfa\") " pod="opendatahub/test-curl-8968-efdfc3" Oct 30 16:46:22.814267 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:22.814252 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdxqx\" (UniqueName: \"kubernetes.io/projected/3f06bb44-8a96-4687-9901-e509308386c4-kube-api-access-cdxqx\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:46:22.836308 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:22.836278 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbcw2\" (UniqueName: \"kubernetes.io/projected/40a41028-8743-4f22-b989-ae80435a8cfa-kube-api-access-sbcw2\") pod \"test-curl-8968-efdfc3\" (UID: \"40a41028-8743-4f22-b989-ae80435a8cfa\") " pod="opendatahub/test-curl-8968-efdfc3" Oct 30 16:46:22.917060 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:22.917033 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8968-efdfc3" Oct 30 16:46:23.255193 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:23.255156 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8968-efdfc3"] Oct 30 16:46:23.255872 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:46:23.255845 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod40a41028_8743_4f22_b989_ae80435a8cfa.slice/crio-063e32c0b4decca1956c3c702618348a399e8e935a19b74a96127d37a51aad43 WatchSource:0}: Error finding container 063e32c0b4decca1956c3c702618348a399e8e935a19b74a96127d37a51aad43: Status 404 returned error can't find the container with id 063e32c0b4decca1956c3c702618348a399e8e935a19b74a96127d37a51aad43 Oct 30 16:46:23.426611 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:23.426566 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8968-1aa1d0" Oct 30 16:46:23.426611 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:23.426620 5052 scope.go:122] "RemoveContainer" containerID="4b6b706055178f49f862760803becfe8bfc30cc6c7e6eb37c3dfc5a6aecf1086" Oct 30 16:46:23.428646 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:23.428614 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8968-efdfc3" event={"ID":"40a41028-8743-4f22-b989-ae80435a8cfa","Type":"ContainerStarted","Data":"c75eebae6cc62a99bde47771a21ef4808f373368c83163be7a686bbfbc3720d9"} Oct 30 16:46:23.428835 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:23.428661 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8968-efdfc3" event={"ID":"40a41028-8743-4f22-b989-ae80435a8cfa","Type":"ContainerStarted","Data":"063e32c0b4decca1956c3c702618348a399e8e935a19b74a96127d37a51aad43"} Oct 30 16:46:23.428835 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:23.428753 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8968-42007b" podUID="738da916-49d4-4378-a88b-3f9572860dcd" containerName="test-curl-8968-42007b" containerID="cri-o://3ab328c940b31026bf6e0b12b2c53062bfb5e847ab190d2d61a250a91da2ac7b" gracePeriod=2 Oct 30 16:46:23.469156 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:23.469114 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8968-efdfc3" podStartSLOduration=1.469100803 podStartE2EDuration="1.469100803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:46:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:46:23.46466579 +0000 UTC m=+1538.992001010" watchObservedRunningTime="2026-10-30 16:46:23.469100803 +0000 UTC m=+1538.996436022" Oct 30 16:46:24.373717 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:24.373681 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8968-efdfc3"] Oct 30 16:46:24.379951 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:24.379922 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8968-efdfc3"] Oct 30 16:46:24.535372 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:24.535336 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8968-f1788c"] Oct 30 16:46:24.536256 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:24.536236 5052 cpu_manager.go:414] "containerMap: removing container" podUID="40a41028-8743-4f22-b989-ae80435a8cfa" containerName="test-curl-8968-efdfc3" Oct 30 16:46:24.536307 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:24.536257 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40a41028-8743-4f22-b989-ae80435a8cfa" containerName="test-curl-8968-efdfc3" Oct 30 16:46:24.536418 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:24.536408 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="40a41028-8743-4f22-b989-ae80435a8cfa" containerName="test-curl-8968-efdfc3" Oct 30 16:46:24.541833 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:24.541807 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8968-f1788c" Oct 30 16:46:24.557494 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:24.557462 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8968-f1788c"] Oct 30 16:46:24.605310 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:24.605273 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f06bb44-8a96-4687-9901-e509308386c4" path="/var/lib/kubelet/pods/3f06bb44-8a96-4687-9901-e509308386c4/volumes" Oct 30 16:46:24.620492 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:24.620463 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6dqx\" (UniqueName: \"kubernetes.io/projected/dcfefc5b-513f-4bfb-8083-c3e4110e5123-kube-api-access-t6dqx\") pod \"test-curl-8968-f1788c\" (UID: \"dcfefc5b-513f-4bfb-8083-c3e4110e5123\") " pod="opendatahub/test-curl-8968-f1788c" Oct 30 16:46:24.722266 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:24.722234 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6dqx\" (UniqueName: \"kubernetes.io/projected/dcfefc5b-513f-4bfb-8083-c3e4110e5123-kube-api-access-t6dqx\") pod \"test-curl-8968-f1788c\" (UID: \"dcfefc5b-513f-4bfb-8083-c3e4110e5123\") " pod="opendatahub/test-curl-8968-f1788c" Oct 30 16:46:24.745836 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:24.745802 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6dqx\" (UniqueName: \"kubernetes.io/projected/dcfefc5b-513f-4bfb-8083-c3e4110e5123-kube-api-access-t6dqx\") pod \"test-curl-8968-f1788c\" (UID: \"dcfefc5b-513f-4bfb-8083-c3e4110e5123\") " pod="opendatahub/test-curl-8968-f1788c" Oct 30 16:46:24.871544 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:24.871513 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8968-f1788c" Oct 30 16:46:25.218462 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:25.218432 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8968-f1788c"] Oct 30 16:46:25.219835 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:46:25.219807 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddcfefc5b_513f_4bfb_8083_c3e4110e5123.slice/crio-2a200f3ba9289ebe9c975e4ce3d0db1c6c4b1defe97d1580b3a06c9a4a7c44b4 WatchSource:0}: Error finding container 2a200f3ba9289ebe9c975e4ce3d0db1c6c4b1defe97d1580b3a06c9a4a7c44b4: Status 404 returned error can't find the container with id 2a200f3ba9289ebe9c975e4ce3d0db1c6c4b1defe97d1580b3a06c9a4a7c44b4 Oct 30 16:46:25.448978 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:25.448950 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8968-f1788c" event={"ID":"dcfefc5b-513f-4bfb-8083-c3e4110e5123","Type":"ContainerStarted","Data":"8af5405514e602e4b95343e9b3083cf748e2c4a13c5a084e83f69105d2ca1b63"} Oct 30 16:46:25.449127 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:25.448988 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8968-f1788c" event={"ID":"dcfefc5b-513f-4bfb-8083-c3e4110e5123","Type":"ContainerStarted","Data":"2a200f3ba9289ebe9c975e4ce3d0db1c6c4b1defe97d1580b3a06c9a4a7c44b4"} Oct 30 16:46:25.449266 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:25.449218 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8968-efdfc3" podUID="40a41028-8743-4f22-b989-ae80435a8cfa" containerName="test-curl-8968-efdfc3" containerID="cri-o://c75eebae6cc62a99bde47771a21ef4808f373368c83163be7a686bbfbc3720d9" gracePeriod=2 Oct 30 16:46:25.486226 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:25.486172 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8968-f1788c" podStartSLOduration=1.486154623 podStartE2EDuration="1.486154623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:46:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:46:25.484716015 +0000 UTC m=+1541.012051234" watchObservedRunningTime="2026-10-30 16:46:25.486154623 +0000 UTC m=+1541.013489844" Oct 30 16:46:25.635691 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:25.635668 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8968-42007b" Oct 30 16:46:25.750618 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:25.748864 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/738da916-49d4-4378-a88b-3f9572860dcd-kube-api-access-thpz7\" (UniqueName: \"kubernetes.io/projected/738da916-49d4-4378-a88b-3f9572860dcd-kube-api-access-thpz7\") pod \"738da916-49d4-4378-a88b-3f9572860dcd\" (UID: \"738da916-49d4-4378-a88b-3f9572860dcd\") " Oct 30 16:46:25.751461 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:25.751432 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/738da916-49d4-4378-a88b-3f9572860dcd-kube-api-access-thpz7" pod "738da916-49d4-4378-a88b-3f9572860dcd" (UID: "738da916-49d4-4378-a88b-3f9572860dcd"). InnerVolumeSpecName "kube-api-access-thpz7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:46:25.867812 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:25.862993 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thpz7\" (UniqueName: \"kubernetes.io/projected/738da916-49d4-4378-a88b-3f9572860dcd-kube-api-access-thpz7\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:46:26.402376 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:26.402339 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8968-f1788c"] Oct 30 16:46:26.408720 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:26.408693 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8968-f1788c"] Oct 30 16:46:26.460596 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:26.460543 5052 generic.go:356] "Generic (PLEG): container finished" podID="738da916-49d4-4378-a88b-3f9572860dcd" containerID="3ab328c940b31026bf6e0b12b2c53062bfb5e847ab190d2d61a250a91da2ac7b" exitCode=137 Oct 30 16:46:26.460759 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:26.460614 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8968-42007b" Oct 30 16:46:26.460759 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:26.460631 5052 scope.go:122] "RemoveContainer" containerID="3ab328c940b31026bf6e0b12b2c53062bfb5e847ab190d2d61a250a91da2ac7b" Oct 30 16:46:26.474976 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:26.474953 5052 scope.go:122] "RemoveContainer" containerID="3ab328c940b31026bf6e0b12b2c53062bfb5e847ab190d2d61a250a91da2ac7b" Oct 30 16:46:26.475311 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:46:26.475283 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ab328c940b31026bf6e0b12b2c53062bfb5e847ab190d2d61a250a91da2ac7b\": container with ID starting with 3ab328c940b31026bf6e0b12b2c53062bfb5e847ab190d2d61a250a91da2ac7b not found: ID does not exist" containerID="3ab328c940b31026bf6e0b12b2c53062bfb5e847ab190d2d61a250a91da2ac7b" Oct 30 16:46:26.475404 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:26.475316 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ab328c940b31026bf6e0b12b2c53062bfb5e847ab190d2d61a250a91da2ac7b"} err="failed to get container status \"3ab328c940b31026bf6e0b12b2c53062bfb5e847ab190d2d61a250a91da2ac7b\": rpc error: code = NotFound desc = could not find container \"3ab328c940b31026bf6e0b12b2c53062bfb5e847ab190d2d61a250a91da2ac7b\": container with ID starting with 3ab328c940b31026bf6e0b12b2c53062bfb5e847ab190d2d61a250a91da2ac7b not found: ID does not exist" Oct 30 16:46:26.491347 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:46:26.491211 5052 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8968-42007b\" is forbidden: User \"system:node:ip-10-0-131-136.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-136.ec2.internal' and this object" podUID="738da916-49d4-4378-a88b-3f9572860dcd" pod="opendatahub/test-curl-8968-42007b" Oct 30 16:46:26.621798 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:26.621098 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="738da916-49d4-4378-a88b-3f9572860dcd" path="/var/lib/kubelet/pods/738da916-49d4-4378-a88b-3f9572860dcd/volumes" Oct 30 16:46:27.562670 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:27.560069 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8968-f1788c" podUID="dcfefc5b-513f-4bfb-8083-c3e4110e5123" containerName="test-curl-8968-f1788c" containerID="cri-o://8af5405514e602e4b95343e9b3083cf748e2c4a13c5a084e83f69105d2ca1b63" gracePeriod=2 Oct 30 16:46:27.866018 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:27.865992 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8968-efdfc3" Oct 30 16:46:27.907943 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:27.889908 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40a41028-8743-4f22-b989-ae80435a8cfa-kube-api-access-sbcw2\" (UniqueName: \"kubernetes.io/projected/40a41028-8743-4f22-b989-ae80435a8cfa-kube-api-access-sbcw2\") pod \"40a41028-8743-4f22-b989-ae80435a8cfa\" (UID: \"40a41028-8743-4f22-b989-ae80435a8cfa\") " Oct 30 16:46:27.907943 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:27.894680 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40a41028-8743-4f22-b989-ae80435a8cfa-kube-api-access-sbcw2" pod "40a41028-8743-4f22-b989-ae80435a8cfa" (UID: "40a41028-8743-4f22-b989-ae80435a8cfa"). InnerVolumeSpecName "kube-api-access-sbcw2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:46:28.005664 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:28.005535 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbcw2\" (UniqueName: \"kubernetes.io/projected/40a41028-8743-4f22-b989-ae80435a8cfa-kube-api-access-sbcw2\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:46:28.569781 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:28.569741 5052 generic.go:356] "Generic (PLEG): container finished" podID="40a41028-8743-4f22-b989-ae80435a8cfa" containerID="c75eebae6cc62a99bde47771a21ef4808f373368c83163be7a686bbfbc3720d9" exitCode=137 Oct 30 16:46:28.570296 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:28.569800 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8968-efdfc3" Oct 30 16:46:28.570296 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:28.569828 5052 scope.go:122] "RemoveContainer" containerID="c75eebae6cc62a99bde47771a21ef4808f373368c83163be7a686bbfbc3720d9" Oct 30 16:46:28.582853 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:28.582755 5052 scope.go:122] "RemoveContainer" containerID="c75eebae6cc62a99bde47771a21ef4808f373368c83163be7a686bbfbc3720d9" Oct 30 16:46:28.583030 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:46:28.583008 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c75eebae6cc62a99bde47771a21ef4808f373368c83163be7a686bbfbc3720d9\": container with ID starting with c75eebae6cc62a99bde47771a21ef4808f373368c83163be7a686bbfbc3720d9 not found: ID does not exist" containerID="c75eebae6cc62a99bde47771a21ef4808f373368c83163be7a686bbfbc3720d9" Oct 30 16:46:28.583082 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:28.583040 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c75eebae6cc62a99bde47771a21ef4808f373368c83163be7a686bbfbc3720d9"} err="failed to get container status \"c75eebae6cc62a99bde47771a21ef4808f373368c83163be7a686bbfbc3720d9\": rpc error: code = NotFound desc = could not find container \"c75eebae6cc62a99bde47771a21ef4808f373368c83163be7a686bbfbc3720d9\": container with ID starting with c75eebae6cc62a99bde47771a21ef4808f373368c83163be7a686bbfbc3720d9 not found: ID does not exist" Oct 30 16:46:28.593627 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:46:28.593595 5052 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8968-efdfc3\" is forbidden: User \"system:node:ip-10-0-131-136.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-136.ec2.internal' and this object" podUID="40a41028-8743-4f22-b989-ae80435a8cfa" pod="opendatahub/test-curl-8968-efdfc3" Oct 30 16:46:28.599871 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:28.599849 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40a41028-8743-4f22-b989-ae80435a8cfa" path="/var/lib/kubelet/pods/40a41028-8743-4f22-b989-ae80435a8cfa/volumes" Oct 30 16:46:29.790177 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:29.790149 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8968-f1788c" Oct 30 16:46:29.862476 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:29.862394 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcfefc5b-513f-4bfb-8083-c3e4110e5123-kube-api-access-t6dqx\" (UniqueName: \"kubernetes.io/projected/dcfefc5b-513f-4bfb-8083-c3e4110e5123-kube-api-access-t6dqx\") pod \"dcfefc5b-513f-4bfb-8083-c3e4110e5123\" (UID: \"dcfefc5b-513f-4bfb-8083-c3e4110e5123\") " Oct 30 16:46:29.864611 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:29.864553 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcfefc5b-513f-4bfb-8083-c3e4110e5123-kube-api-access-t6dqx" pod "dcfefc5b-513f-4bfb-8083-c3e4110e5123" (UID: "dcfefc5b-513f-4bfb-8083-c3e4110e5123"). InnerVolumeSpecName "kube-api-access-t6dqx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:46:29.965099 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:29.965060 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6dqx\" (UniqueName: \"kubernetes.io/projected/dcfefc5b-513f-4bfb-8083-c3e4110e5123-kube-api-access-t6dqx\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:46:30.588101 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:30.588060 5052 generic.go:356] "Generic (PLEG): container finished" podID="dcfefc5b-513f-4bfb-8083-c3e4110e5123" containerID="8af5405514e602e4b95343e9b3083cf748e2c4a13c5a084e83f69105d2ca1b63" exitCode=137 Oct 30 16:46:30.588271 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:30.588122 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8968-f1788c" Oct 30 16:46:30.588271 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:30.588161 5052 scope.go:122] "RemoveContainer" containerID="8af5405514e602e4b95343e9b3083cf748e2c4a13c5a084e83f69105d2ca1b63" Oct 30 16:46:30.599862 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:30.599841 5052 scope.go:122] "RemoveContainer" containerID="8af5405514e602e4b95343e9b3083cf748e2c4a13c5a084e83f69105d2ca1b63" Oct 30 16:46:30.600122 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:46:30.600101 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8af5405514e602e4b95343e9b3083cf748e2c4a13c5a084e83f69105d2ca1b63\": container with ID starting with 8af5405514e602e4b95343e9b3083cf748e2c4a13c5a084e83f69105d2ca1b63 not found: ID does not exist" containerID="8af5405514e602e4b95343e9b3083cf748e2c4a13c5a084e83f69105d2ca1b63" Oct 30 16:46:30.600183 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:30.600135 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8af5405514e602e4b95343e9b3083cf748e2c4a13c5a084e83f69105d2ca1b63"} err="failed to get container status \"8af5405514e602e4b95343e9b3083cf748e2c4a13c5a084e83f69105d2ca1b63\": rpc error: code = NotFound desc = could not find container \"8af5405514e602e4b95343e9b3083cf748e2c4a13c5a084e83f69105d2ca1b63\": container with ID starting with 8af5405514e602e4b95343e9b3083cf748e2c4a13c5a084e83f69105d2ca1b63 not found: ID does not exist" Oct 30 16:46:30.604073 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:30.604047 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcfefc5b-513f-4bfb-8083-c3e4110e5123" path="/var/lib/kubelet/pods/dcfefc5b-513f-4bfb-8083-c3e4110e5123/volumes" Oct 30 16:46:30.611358 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:46:30.611333 5052 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8968-f1788c\" is forbidden: User \"system:node:ip-10-0-131-136.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-136.ec2.internal' and this object" podUID="dcfefc5b-513f-4bfb-8083-c3e4110e5123" pod="opendatahub/test-curl-8968-f1788c" Oct 30 16:46:31.392500 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:31.392470 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b315d4-84cb4fb687-9466n" Oct 30 16:46:34.842500 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:34.842460 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dthnj"] Oct 30 16:46:34.843759 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:34.843735 5052 cpu_manager.go:414] "containerMap: removing container" podUID="dcfefc5b-513f-4bfb-8083-c3e4110e5123" containerName="test-curl-8968-f1788c" Oct 30 16:46:34.843872 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:34.843759 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcfefc5b-513f-4bfb-8083-c3e4110e5123" containerName="test-curl-8968-f1788c" Oct 30 16:46:34.843985 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:34.843971 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcfefc5b-513f-4bfb-8083-c3e4110e5123" containerName="test-curl-8968-f1788c" Oct 30 16:46:34.850293 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:34.850266 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dthnj" Oct 30 16:46:34.871014 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:34.870981 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dthnj"] Oct 30 16:46:35.248605 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:35.225246 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55ab8e2a-2823-4ff1-a391-572ce6586084-credential-socket\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dthnj\" (UID: \"55ab8e2a-2823-4ff1-a391-572ce6586084\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dthnj" Oct 30 16:46:35.252033 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:35.252007 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55ab8e2a-2823-4ff1-a391-572ce6586084-istio-podinfo\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dthnj\" (UID: \"55ab8e2a-2823-4ff1-a391-572ce6586084\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dthnj" Oct 30 16:46:35.252256 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:35.252239 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55ab8e2a-2823-4ff1-a391-572ce6586084-workload-socket\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dthnj\" (UID: \"55ab8e2a-2823-4ff1-a391-572ce6586084\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dthnj" Oct 30 16:46:35.252381 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:35.252368 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55ab8e2a-2823-4ff1-a391-572ce6586084-istio-token\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dthnj\" (UID: \"55ab8e2a-2823-4ff1-a391-572ce6586084\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dthnj" Oct 30 16:46:35.252541 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:35.252525 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtmtb\" (UniqueName: \"kubernetes.io/projected/55ab8e2a-2823-4ff1-a391-572ce6586084-kube-api-access-vtmtb\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dthnj\" (UID: \"55ab8e2a-2823-4ff1-a391-572ce6586084\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dthnj" Oct 30 16:46:35.271868 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:35.271829 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55ab8e2a-2823-4ff1-a391-572ce6586084-workload-certs\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dthnj\" (UID: \"55ab8e2a-2823-4ff1-a391-572ce6586084\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dthnj" Oct 30 16:46:35.272146 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:35.272122 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55ab8e2a-2823-4ff1-a391-572ce6586084-istiod-ca-cert\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dthnj\" (UID: \"55ab8e2a-2823-4ff1-a391-572ce6586084\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dthnj" Oct 30 16:46:35.272411 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:35.272394 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55ab8e2a-2823-4ff1-a391-572ce6586084-istio-data\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dthnj\" (UID: \"55ab8e2a-2823-4ff1-a391-572ce6586084\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dthnj" Oct 30 16:46:35.272642 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:35.272624 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55ab8e2a-2823-4ff1-a391-572ce6586084-istio-envoy\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dthnj\" (UID: \"55ab8e2a-2823-4ff1-a391-572ce6586084\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dthnj" Oct 30 16:46:35.377604 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:35.376773 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55ab8e2a-2823-4ff1-a391-572ce6586084-istio-podinfo\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dthnj\" (UID: \"55ab8e2a-2823-4ff1-a391-572ce6586084\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dthnj" Oct 30 16:46:35.377604 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:35.376829 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55ab8e2a-2823-4ff1-a391-572ce6586084-workload-socket\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dthnj\" (UID: \"55ab8e2a-2823-4ff1-a391-572ce6586084\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dthnj" Oct 30 16:46:35.377604 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:35.376864 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55ab8e2a-2823-4ff1-a391-572ce6586084-istio-token\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dthnj\" (UID: \"55ab8e2a-2823-4ff1-a391-572ce6586084\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dthnj" Oct 30 16:46:35.377604 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:35.376920 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtmtb\" (UniqueName: \"kubernetes.io/projected/55ab8e2a-2823-4ff1-a391-572ce6586084-kube-api-access-vtmtb\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dthnj\" (UID: \"55ab8e2a-2823-4ff1-a391-572ce6586084\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dthnj" Oct 30 16:46:35.377604 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:35.376992 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55ab8e2a-2823-4ff1-a391-572ce6586084-workload-certs\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dthnj\" (UID: \"55ab8e2a-2823-4ff1-a391-572ce6586084\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dthnj" Oct 30 16:46:35.377604 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:35.377022 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55ab8e2a-2823-4ff1-a391-572ce6586084-istiod-ca-cert\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dthnj\" (UID: \"55ab8e2a-2823-4ff1-a391-572ce6586084\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dthnj" Oct 30 16:46:35.377604 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:35.377157 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55ab8e2a-2823-4ff1-a391-572ce6586084-istio-data\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dthnj\" (UID: \"55ab8e2a-2823-4ff1-a391-572ce6586084\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dthnj" Oct 30 16:46:35.377604 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:35.377201 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55ab8e2a-2823-4ff1-a391-572ce6586084-istio-envoy\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dthnj\" (UID: \"55ab8e2a-2823-4ff1-a391-572ce6586084\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dthnj" Oct 30 16:46:35.377604 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:35.377333 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55ab8e2a-2823-4ff1-a391-572ce6586084-credential-socket\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dthnj\" (UID: \"55ab8e2a-2823-4ff1-a391-572ce6586084\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dthnj" Oct 30 16:46:35.389633 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:35.382944 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55ab8e2a-2823-4ff1-a391-572ce6586084-workload-socket\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dthnj\" (UID: \"55ab8e2a-2823-4ff1-a391-572ce6586084\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dthnj" Oct 30 16:46:35.390932 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:35.390901 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55ab8e2a-2823-4ff1-a391-572ce6586084-istiod-ca-cert\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dthnj\" (UID: \"55ab8e2a-2823-4ff1-a391-572ce6586084\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dthnj" Oct 30 16:46:35.392197 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:35.391523 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55ab8e2a-2823-4ff1-a391-572ce6586084-istio-podinfo\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dthnj\" (UID: \"55ab8e2a-2823-4ff1-a391-572ce6586084\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dthnj" Oct 30 16:46:35.392327 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:35.391753 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55ab8e2a-2823-4ff1-a391-572ce6586084-istio-data\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dthnj\" (UID: \"55ab8e2a-2823-4ff1-a391-572ce6586084\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dthnj" Oct 30 16:46:35.392405 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:35.391934 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55ab8e2a-2823-4ff1-a391-572ce6586084-workload-certs\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dthnj\" (UID: \"55ab8e2a-2823-4ff1-a391-572ce6586084\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dthnj" Oct 30 16:46:35.396271 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:35.392706 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55ab8e2a-2823-4ff1-a391-572ce6586084-credential-socket\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dthnj\" (UID: \"55ab8e2a-2823-4ff1-a391-572ce6586084\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dthnj" Oct 30 16:46:35.396430 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:35.395677 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55ab8e2a-2823-4ff1-a391-572ce6586084-istio-envoy\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dthnj\" (UID: \"55ab8e2a-2823-4ff1-a391-572ce6586084\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dthnj" Oct 30 16:46:35.431606 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:35.413644 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtmtb\" (UniqueName: \"kubernetes.io/projected/55ab8e2a-2823-4ff1-a391-572ce6586084-kube-api-access-vtmtb\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dthnj\" (UID: \"55ab8e2a-2823-4ff1-a391-572ce6586084\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dthnj" Oct 30 16:46:35.448150 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:35.426076 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55ab8e2a-2823-4ff1-a391-572ce6586084-istio-token\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dthnj\" (UID: \"55ab8e2a-2823-4ff1-a391-572ce6586084\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dthnj" Oct 30 16:46:35.473992 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:35.473950 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dthnj" Oct 30 16:46:36.066738 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:46:36.066698 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55ab8e2a_2823_4ff1_a391_572ce6586084.slice/crio-cd1956d9fb1bb537f0e839ef55b2f30083d8414219ac3e7613cad66424a4d69d WatchSource:0}: Error finding container cd1956d9fb1bb537f0e839ef55b2f30083d8414219ac3e7613cad66424a4d69d: Status 404 returned error can't find the container with id cd1956d9fb1bb537f0e839ef55b2f30083d8414219ac3e7613cad66424a4d69d Oct 30 16:46:36.070279 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:36.070255 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dthnj"] Oct 30 16:46:36.071220 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:36.071186 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:46:36.071317 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:36.071266 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:46:36.071317 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:36.071311 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:46:36.713619 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:36.713553 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dthnj" event={"ID":"55ab8e2a-2823-4ff1-a391-572ce6586084","Type":"ContainerStarted","Data":"da5810e192dfc520f72419d10cb863e67ea65086148476efe3e13a5426cada1d"} Oct 30 16:46:36.713769 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:36.713624 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dthnj" event={"ID":"55ab8e2a-2823-4ff1-a391-572ce6586084","Type":"ContainerStarted","Data":"cd1956d9fb1bb537f0e839ef55b2f30083d8414219ac3e7613cad66424a4d69d"} Oct 30 16:46:36.798811 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:36.798762 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dthnj" podStartSLOduration=2.798747802 podStartE2EDuration="2.798747802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:46:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:46:36.784289734 +0000 UTC m=+1552.311624954" watchObservedRunningTime="2026-10-30 16:46:36.798747802 +0000 UTC m=+1552.326083020" Oct 30 16:46:37.475295 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:37.475256 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dthnj" Oct 30 16:46:37.476462 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:37.476437 5052 patch_prober.go:29] interesting pod/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dthnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Oct 30 16:46:37.476520 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:37.476488 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dthnj" podUID="55ab8e2a-2823-4ff1-a391-572ce6586084" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Oct 30 16:46:38.474992 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:38.474951 5052 patch_prober.go:29] interesting pod/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dthnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Oct 30 16:46:38.475270 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:38.475240 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dthnj" podUID="55ab8e2a-2823-4ff1-a391-572ce6586084" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Oct 30 16:46:39.480561 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:39.480523 5052 patch_prober.go:29] interesting pod/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dthnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Oct 30 16:46:39.481509 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:39.481474 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dthnj" podUID="55ab8e2a-2823-4ff1-a391-572ce6586084" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Oct 30 16:46:40.475162 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:40.475121 5052 patch_prober.go:29] interesting pod/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dthnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Oct 30 16:46:40.475345 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:40.475190 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dthnj" podUID="55ab8e2a-2823-4ff1-a391-572ce6586084" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Oct 30 16:46:41.475874 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:41.475843 5052 patch_prober.go:29] interesting pod/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dthnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Oct 30 16:46:41.476244 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:41.475896 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dthnj" podUID="55ab8e2a-2823-4ff1-a391-572ce6586084" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Oct 30 16:46:42.475353 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:42.475313 5052 patch_prober.go:29] interesting pod/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dthnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Oct 30 16:46:42.475553 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:42.475396 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dthnj" podUID="55ab8e2a-2823-4ff1-a391-572ce6586084" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Oct 30 16:46:43.479956 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:43.479925 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dthnj" Oct 30 16:46:43.480399 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:43.480384 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dthnj" Oct 30 16:46:43.480995 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:43.480972 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dthnj" Oct 30 16:46:45.001105 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:45.001068 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-00fd1a-86b946bb7f-dtk29"] Oct 30 16:46:45.006306 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:45.006282 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-00fd1a-86b946bb7f-dtk29" Oct 30 16:46:45.027647 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:45.027623 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-00fd1a-86b946bb7f-dtk29"] Oct 30 16:46:45.176620 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:45.172467 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-00fd1a-dockercfg-69rwx\"" Oct 30 16:46:45.176620 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:45.172536 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-00fd1a\"" Oct 30 16:46:45.324309 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:45.312664 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1e2767a-41a6-453f-8ac7-4bc93b77b143-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-00fd1a-86b946bb7f-dtk29\" (UID: \"a1e2767a-41a6-453f-8ac7-4bc93b77b143\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-00fd1a-86b946bb7f-dtk29" Oct 30 16:46:45.324309 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:45.312866 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68zcp\" (UniqueName: \"kubernetes.io/projected/a1e2767a-41a6-453f-8ac7-4bc93b77b143-kube-api-access-68zcp\") pod \"payload-pre-processing-e2e-api-a-00fd1a-86b946bb7f-dtk29\" (UID: \"a1e2767a-41a6-453f-8ac7-4bc93b77b143\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-00fd1a-86b946bb7f-dtk29" Oct 30 16:46:45.483080 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:45.481863 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1e2767a-41a6-453f-8ac7-4bc93b77b143-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-00fd1a-86b946bb7f-dtk29\" (UID: \"a1e2767a-41a6-453f-8ac7-4bc93b77b143\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-00fd1a-86b946bb7f-dtk29" Oct 30 16:46:45.493047 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:45.493014 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68zcp\" (UniqueName: \"kubernetes.io/projected/a1e2767a-41a6-453f-8ac7-4bc93b77b143-kube-api-access-68zcp\") pod \"payload-pre-processing-e2e-api-a-00fd1a-86b946bb7f-dtk29\" (UID: \"a1e2767a-41a6-453f-8ac7-4bc93b77b143\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-00fd1a-86b946bb7f-dtk29" Oct 30 16:46:45.496493 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:45.484215 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1e2767a-41a6-453f-8ac7-4bc93b77b143-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-00fd1a-86b946bb7f-dtk29\" (UID: \"a1e2767a-41a6-453f-8ac7-4bc93b77b143\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-00fd1a-86b946bb7f-dtk29" Oct 30 16:46:45.559908 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:45.559857 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68zcp\" (UniqueName: \"kubernetes.io/projected/a1e2767a-41a6-453f-8ac7-4bc93b77b143-kube-api-access-68zcp\") pod \"payload-pre-processing-e2e-api-a-00fd1a-86b946bb7f-dtk29\" (UID: \"a1e2767a-41a6-453f-8ac7-4bc93b77b143\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-00fd1a-86b946bb7f-dtk29" Oct 30 16:46:45.627985 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:45.627948 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-00fd1a-86b946bb7f-dtk29" Oct 30 16:46:45.949646 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:46:45.949611 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1e2767a_41a6_453f_8ac7_4bc93b77b143.slice/crio-a3530ffdb87ad7a34b69a0f3a9044a4949600d2ef621bfe3c467c4e2437c297a WatchSource:0}: Error finding container a3530ffdb87ad7a34b69a0f3a9044a4949600d2ef621bfe3c467c4e2437c297a: Status 404 returned error can't find the container with id a3530ffdb87ad7a34b69a0f3a9044a4949600d2ef621bfe3c467c4e2437c297a Oct 30 16:46:45.949785 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:45.949711 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-00fd1a-86b946bb7f-dtk29"] Oct 30 16:46:46.933096 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:46.933057 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-00fd1a-86b946bb7f-dtk29" event={"ID":"a1e2767a-41a6-453f-8ac7-4bc93b77b143","Type":"ContainerStarted","Data":"6882094be201b85d43522fe5d438a0306cf2236c06b32757a497439be3f82349"} Oct 30 16:46:46.933483 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:46.933121 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-00fd1a-86b946bb7f-dtk29" event={"ID":"a1e2767a-41a6-453f-8ac7-4bc93b77b143","Type":"ContainerStarted","Data":"a3530ffdb87ad7a34b69a0f3a9044a4949600d2ef621bfe3c467c4e2437c297a"} Oct 30 16:46:46.933483 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:46.933149 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-00fd1a-86b946bb7f-dtk29" Oct 30 16:46:46.978725 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:46.978667 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-00fd1a-86b946bb7f-dtk29" podStartSLOduration=2.978650719 podStartE2EDuration="2.978650719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:46:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:46:46.977917267 +0000 UTC m=+1562.505252487" watchObservedRunningTime="2026-10-30 16:46:46.978650719 +0000 UTC m=+1562.505985941" Oct 30 16:46:48.842087 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:46:48.842051 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 16:46:56.889605 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:56.869497 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cdwcj"] Oct 30 16:46:57.944520 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:46:57.944493 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-00fd1a-86b946bb7f-dtk29" Oct 30 16:47:05.027303 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:05.027267 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8962-2f0f36"] Oct 30 16:47:05.033631 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:05.033607 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8962-2f0f36" Oct 30 16:47:05.049041 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:05.049013 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8962-2f0f36"] Oct 30 16:47:05.056818 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:05.056793 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-xr4c7\"" Oct 30 16:47:05.066728 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:05.066702 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x57wm\" (UniqueName: \"kubernetes.io/projected/46e348ba-6e79-4665-a691-c7632a581520-kube-api-access-x57wm\") pod \"test-curl-8962-2f0f36\" (UID: \"46e348ba-6e79-4665-a691-c7632a581520\") " pod="opendatahub/test-curl-8962-2f0f36" Oct 30 16:47:05.168179 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:05.168142 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x57wm\" (UniqueName: \"kubernetes.io/projected/46e348ba-6e79-4665-a691-c7632a581520-kube-api-access-x57wm\") pod \"test-curl-8962-2f0f36\" (UID: \"46e348ba-6e79-4665-a691-c7632a581520\") " pod="opendatahub/test-curl-8962-2f0f36" Oct 30 16:47:05.192491 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:05.192467 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x57wm\" (UniqueName: \"kubernetes.io/projected/46e348ba-6e79-4665-a691-c7632a581520-kube-api-access-x57wm\") pod \"test-curl-8962-2f0f36\" (UID: \"46e348ba-6e79-4665-a691-c7632a581520\") " pod="opendatahub/test-curl-8962-2f0f36" Oct 30 16:47:05.350735 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:05.350658 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8962-2f0f36" Oct 30 16:47:05.661417 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:05.661392 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8962-2f0f36"] Oct 30 16:47:05.665338 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:47:05.662063 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod46e348ba_6e79_4665_a691_c7632a581520.slice/crio-36e3b403fe13d866912736063c28e4606ad066f5aa87ccfe4ea9e12e000fe263 WatchSource:0}: Error finding container 36e3b403fe13d866912736063c28e4606ad066f5aa87ccfe4ea9e12e000fe263: Status 404 returned error can't find the container with id 36e3b403fe13d866912736063c28e4606ad066f5aa87ccfe4ea9e12e000fe263 Oct 30 16:47:06.118841 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:06.118742 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8962-2f0f36" event={"ID":"46e348ba-6e79-4665-a691-c7632a581520","Type":"ContainerStarted","Data":"24d11b5d9c612f0f73aeae9716d6f5a68b1f51ae57261e7f237fde75434418cc"} Oct 30 16:47:06.118841 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:06.118793 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8962-2f0f36" event={"ID":"46e348ba-6e79-4665-a691-c7632a581520","Type":"ContainerStarted","Data":"36e3b403fe13d866912736063c28e4606ad066f5aa87ccfe4ea9e12e000fe263"} Oct 30 16:47:06.160316 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:06.160266 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8962-2f0f36" podStartSLOduration=1.160248235 podStartE2EDuration="1.160248235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:47:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:47:06.158831183 +0000 UTC m=+1581.686166402" watchObservedRunningTime="2026-10-30 16:47:06.160248235 +0000 UTC m=+1581.687583453" Oct 30 16:47:07.102949 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:07.102902 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8962-2f0f36"] Oct 30 16:47:07.109089 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:07.109055 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8962-2f0f36"] Oct 30 16:47:08.139248 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:08.139208 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8962-2f0f36" podUID="46e348ba-6e79-4665-a691-c7632a581520" containerName="test-curl-8962-2f0f36" containerID="cri-o://24d11b5d9c612f0f73aeae9716d6f5a68b1f51ae57261e7f237fde75434418cc" gracePeriod=2 Oct 30 16:47:08.154117 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:47:08.154073 5052 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8962-2f0f36\" is forbidden: User \"system:node:ip-10-0-131-136.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-136.ec2.internal' and this object" podUID="46e348ba-6e79-4665-a691-c7632a581520" pod="opendatahub/test-curl-8962-2f0f36" Oct 30 16:47:10.521867 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:10.521833 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8962-2f0f36" Oct 30 16:47:10.535235 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:10.535185 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46e348ba-6e79-4665-a691-c7632a581520-kube-api-access-x57wm\" (UniqueName: \"kubernetes.io/projected/46e348ba-6e79-4665-a691-c7632a581520-kube-api-access-x57wm\") pod \"46e348ba-6e79-4665-a691-c7632a581520\" (UID: \"46e348ba-6e79-4665-a691-c7632a581520\") " Oct 30 16:47:10.537642 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:10.537609 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46e348ba-6e79-4665-a691-c7632a581520-kube-api-access-x57wm" pod "46e348ba-6e79-4665-a691-c7632a581520" (UID: "46e348ba-6e79-4665-a691-c7632a581520"). InnerVolumeSpecName "kube-api-access-x57wm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:47:10.600437 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:10.600403 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46e348ba-6e79-4665-a691-c7632a581520" path="/var/lib/kubelet/pods/46e348ba-6e79-4665-a691-c7632a581520/volumes" Oct 30 16:47:10.638597 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:10.638546 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x57wm\" (UniqueName: \"kubernetes.io/projected/46e348ba-6e79-4665-a691-c7632a581520-kube-api-access-x57wm\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:47:11.167776 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:11.167741 5052 generic.go:356] "Generic (PLEG): container finished" podID="46e348ba-6e79-4665-a691-c7632a581520" containerID="24d11b5d9c612f0f73aeae9716d6f5a68b1f51ae57261e7f237fde75434418cc" exitCode=137 Oct 30 16:47:11.167946 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:11.167797 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8962-2f0f36" Oct 30 16:47:11.167946 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:11.167842 5052 scope.go:122] "RemoveContainer" containerID="24d11b5d9c612f0f73aeae9716d6f5a68b1f51ae57261e7f237fde75434418cc" Oct 30 16:47:11.178943 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:11.178923 5052 scope.go:122] "RemoveContainer" containerID="24d11b5d9c612f0f73aeae9716d6f5a68b1f51ae57261e7f237fde75434418cc" Oct 30 16:47:11.179228 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:47:11.179209 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24d11b5d9c612f0f73aeae9716d6f5a68b1f51ae57261e7f237fde75434418cc\": container with ID starting with 24d11b5d9c612f0f73aeae9716d6f5a68b1f51ae57261e7f237fde75434418cc not found: ID does not exist" containerID="24d11b5d9c612f0f73aeae9716d6f5a68b1f51ae57261e7f237fde75434418cc" Oct 30 16:47:11.179299 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:11.179239 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24d11b5d9c612f0f73aeae9716d6f5a68b1f51ae57261e7f237fde75434418cc"} err="failed to get container status \"24d11b5d9c612f0f73aeae9716d6f5a68b1f51ae57261e7f237fde75434418cc\": rpc error: code = NotFound desc = could not find container \"24d11b5d9c612f0f73aeae9716d6f5a68b1f51ae57261e7f237fde75434418cc\": container with ID starting with 24d11b5d9c612f0f73aeae9716d6f5a68b1f51ae57261e7f237fde75434418cc not found: ID does not exist" Oct 30 16:47:18.201761 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:18.201726 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b8af660d-787d8764b-h9rfs"] Oct 30 16:47:18.933214 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:47:18.933168 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 16:47:20.474011 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:20.473952 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-xjhj4"] Oct 30 16:47:20.474536 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:20.474233 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-xjhj4" podUID="029d295c-a65a-4191-9754-a421c21464ba" containerName="istio-proxy" containerID="cri-o://8b4392bec4b7bb73bffe133bc736111a10345438eb82677a89f00d7358eb3889" gracePeriod=30 Oct 30 16:47:20.487699 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:20.487671 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-vbvvh"] Oct 30 16:47:20.487917 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:20.487892 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-vbvvh" podUID="0a5f0a2a-f2f2-4bea-98e8-16e6794028de" containerName="istio-proxy" containerID="cri-o://48c4105bf5427439a31b5cd61c955f82d1e0fa0e1b84b87baeed6c3b947b0793" gracePeriod=30 Oct 30 16:47:20.491843 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:20.491818 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-46brl"] Oct 30 16:47:20.491983 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:20.491966 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-46brl" podUID="4b8235a9-5724-4354-ad7b-eaef26b3966f" containerName="istio-proxy" containerID="cri-o://06eae2719515f348df2fe76d5997e5a8ea46ffb83a0d24cce2f7885dfa88b985" gracePeriod=30 Oct 30 16:47:20.502481 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:20.502455 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-kznnt"] Oct 30 16:47:20.502651 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:20.502633 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-kznnt" podUID="c254bba7-008e-4724-80a8-8292ae42b7e1" containerName="istio-proxy" containerID="cri-o://cf30a6a0225cb105f4c7c0096f168d0f62a4cdbc453dbb9fe995055af8d31aef" gracePeriod=30 Oct 30 16:47:22.480601 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:22.478773 5052 patch_prober.go:29] interesting pod/e2e-worker-w3-0b2348-openshift-default-85679bc47c-46brl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 16:47:22.480601 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:22.478839 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-46brl" podUID="4b8235a9-5724-4354-ad7b-eaef26b3966f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 16:47:23.213662 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:23.211273 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-b8af660d-787d8764b-h9rfs" podUID="2c983762-43df-4410-9001-d5d25c7a45f3" containerName="payload-processing" containerID="cri-o://1cea34d2c5be1ab53a7ea1aff002b811b72bf2029f09335db701ce6f718c1587" gracePeriod=25 Oct 30 16:47:25.880885 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:25.880856 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-vbvvh" Oct 30 16:47:25.883826 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:25.883802 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-xjhj4" Oct 30 16:47:25.957866 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:25.957841 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-46brl" Oct 30 16:47:25.965467 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:25.965446 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-kznnt" Oct 30 16:47:26.061071 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.061032 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/029d295c-a65a-4191-9754-a421c21464ba-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/029d295c-a65a-4191-9754-a421c21464ba-credential-socket\") pod \"029d295c-a65a-4191-9754-a421c21464ba\" (UID: \"029d295c-a65a-4191-9754-a421c21464ba\") " Oct 30 16:47:26.061525 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.061495 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/029d295c-a65a-4191-9754-a421c21464ba-credential-socket" pod "029d295c-a65a-4191-9754-a421c21464ba" (UID: "029d295c-a65a-4191-9754-a421c21464ba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:47:26.061664 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.061565 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a5f0a2a-f2f2-4bea-98e8-16e6794028de-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a5f0a2a-f2f2-4bea-98e8-16e6794028de-istio-data\") pod \"0a5f0a2a-f2f2-4bea-98e8-16e6794028de\" (UID: \"0a5f0a2a-f2f2-4bea-98e8-16e6794028de\") " Oct 30 16:47:26.061729 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.061710 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b8235a9-5724-4354-ad7b-eaef26b3966f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b8235a9-5724-4354-ad7b-eaef26b3966f-workload-socket\") pod \"4b8235a9-5724-4354-ad7b-eaef26b3966f\" (UID: \"4b8235a9-5724-4354-ad7b-eaef26b3966f\") " Oct 30 16:47:26.061793 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.061782 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/029d295c-a65a-4191-9754-a421c21464ba-kube-api-access-ghmvk\" (UniqueName: \"kubernetes.io/projected/029d295c-a65a-4191-9754-a421c21464ba-kube-api-access-ghmvk\") pod \"029d295c-a65a-4191-9754-a421c21464ba\" (UID: \"029d295c-a65a-4191-9754-a421c21464ba\") " Oct 30 16:47:26.062243 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.062216 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b8235a9-5724-4354-ad7b-eaef26b3966f-workload-socket" pod "4b8235a9-5724-4354-ad7b-eaef26b3966f" (UID: "4b8235a9-5724-4354-ad7b-eaef26b3966f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:47:26.062332 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.062275 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b8235a9-5724-4354-ad7b-eaef26b3966f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b8235a9-5724-4354-ad7b-eaef26b3966f-istio-envoy\") pod \"4b8235a9-5724-4354-ad7b-eaef26b3966f\" (UID: \"4b8235a9-5724-4354-ad7b-eaef26b3966f\") " Oct 30 16:47:26.062450 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.062349 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0a5f0a2a-f2f2-4bea-98e8-16e6794028de-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a5f0a2a-f2f2-4bea-98e8-16e6794028de-istio-podinfo\") pod \"0a5f0a2a-f2f2-4bea-98e8-16e6794028de\" (UID: \"0a5f0a2a-f2f2-4bea-98e8-16e6794028de\") " Oct 30 16:47:26.062450 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.062387 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a5f0a2a-f2f2-4bea-98e8-16e6794028de-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a5f0a2a-f2f2-4bea-98e8-16e6794028de-istiod-ca-cert\") pod \"0a5f0a2a-f2f2-4bea-98e8-16e6794028de\" (UID: \"0a5f0a2a-f2f2-4bea-98e8-16e6794028de\") " Oct 30 16:47:26.062450 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.062435 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a5f0a2a-f2f2-4bea-98e8-16e6794028de-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a5f0a2a-f2f2-4bea-98e8-16e6794028de-istio-envoy\") pod \"0a5f0a2a-f2f2-4bea-98e8-16e6794028de\" (UID: \"0a5f0a2a-f2f2-4bea-98e8-16e6794028de\") " Oct 30 16:47:26.062660 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.062465 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/029d295c-a65a-4191-9754-a421c21464ba-istio-token\" (UniqueName: \"kubernetes.io/projected/029d295c-a65a-4191-9754-a421c21464ba-istio-token\") pod \"029d295c-a65a-4191-9754-a421c21464ba\" (UID: \"029d295c-a65a-4191-9754-a421c21464ba\") " Oct 30 16:47:26.062660 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.062494 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a5f0a2a-f2f2-4bea-98e8-16e6794028de-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a5f0a2a-f2f2-4bea-98e8-16e6794028de-credential-socket\") pod \"0a5f0a2a-f2f2-4bea-98e8-16e6794028de\" (UID: \"0a5f0a2a-f2f2-4bea-98e8-16e6794028de\") " Oct 30 16:47:26.062660 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.062524 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a5f0a2a-f2f2-4bea-98e8-16e6794028de-kube-api-access-nt8qt\" (UniqueName: \"kubernetes.io/projected/0a5f0a2a-f2f2-4bea-98e8-16e6794028de-kube-api-access-nt8qt\") pod \"0a5f0a2a-f2f2-4bea-98e8-16e6794028de\" (UID: \"0a5f0a2a-f2f2-4bea-98e8-16e6794028de\") " Oct 30 16:47:26.062660 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.062626 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b8235a9-5724-4354-ad7b-eaef26b3966f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b8235a9-5724-4354-ad7b-eaef26b3966f-workload-certs\") pod \"4b8235a9-5724-4354-ad7b-eaef26b3966f\" (UID: \"4b8235a9-5724-4354-ad7b-eaef26b3966f\") " Oct 30 16:47:26.062868 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.062668 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/029d295c-a65a-4191-9754-a421c21464ba-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/029d295c-a65a-4191-9754-a421c21464ba-istio-envoy\") pod \"029d295c-a65a-4191-9754-a421c21464ba\" (UID: \"029d295c-a65a-4191-9754-a421c21464ba\") " Oct 30 16:47:26.062868 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.062720 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c254bba7-008e-4724-80a8-8292ae42b7e1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c254bba7-008e-4724-80a8-8292ae42b7e1-credential-socket\") pod \"c254bba7-008e-4724-80a8-8292ae42b7e1\" (UID: \"c254bba7-008e-4724-80a8-8292ae42b7e1\") " Oct 30 16:47:26.062868 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.062765 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/029d295c-a65a-4191-9754-a421c21464ba-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/029d295c-a65a-4191-9754-a421c21464ba-istio-podinfo\") pod \"029d295c-a65a-4191-9754-a421c21464ba\" (UID: \"029d295c-a65a-4191-9754-a421c21464ba\") " Oct 30 16:47:26.062868 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.062799 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c254bba7-008e-4724-80a8-8292ae42b7e1-kube-api-access-dk6pt\" (UniqueName: \"kubernetes.io/projected/c254bba7-008e-4724-80a8-8292ae42b7e1-kube-api-access-dk6pt\") pod \"c254bba7-008e-4724-80a8-8292ae42b7e1\" (UID: \"c254bba7-008e-4724-80a8-8292ae42b7e1\") " Oct 30 16:47:26.062868 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.062852 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c254bba7-008e-4724-80a8-8292ae42b7e1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c254bba7-008e-4724-80a8-8292ae42b7e1-istio-podinfo\") pod \"c254bba7-008e-4724-80a8-8292ae42b7e1\" (UID: \"c254bba7-008e-4724-80a8-8292ae42b7e1\") " Oct 30 16:47:26.063125 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.062894 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b8235a9-5724-4354-ad7b-eaef26b3966f-kube-api-access-w24fw\" (UniqueName: \"kubernetes.io/projected/4b8235a9-5724-4354-ad7b-eaef26b3966f-kube-api-access-w24fw\") pod \"4b8235a9-5724-4354-ad7b-eaef26b3966f\" (UID: \"4b8235a9-5724-4354-ad7b-eaef26b3966f\") " Oct 30 16:47:26.063125 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.062968 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c254bba7-008e-4724-80a8-8292ae42b7e1-istio-token\" (UniqueName: \"kubernetes.io/projected/c254bba7-008e-4724-80a8-8292ae42b7e1-istio-token\") pod \"c254bba7-008e-4724-80a8-8292ae42b7e1\" (UID: \"c254bba7-008e-4724-80a8-8292ae42b7e1\") " Oct 30 16:47:26.063125 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.062998 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c254bba7-008e-4724-80a8-8292ae42b7e1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c254bba7-008e-4724-80a8-8292ae42b7e1-istio-envoy\") pod \"c254bba7-008e-4724-80a8-8292ae42b7e1\" (UID: \"c254bba7-008e-4724-80a8-8292ae42b7e1\") " Oct 30 16:47:26.063125 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.063025 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c254bba7-008e-4724-80a8-8292ae42b7e1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c254bba7-008e-4724-80a8-8292ae42b7e1-istio-data\") pod \"c254bba7-008e-4724-80a8-8292ae42b7e1\" (UID: \"c254bba7-008e-4724-80a8-8292ae42b7e1\") " Oct 30 16:47:26.063125 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.063086 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b8235a9-5724-4354-ad7b-eaef26b3966f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b8235a9-5724-4354-ad7b-eaef26b3966f-istiod-ca-cert\") pod \"4b8235a9-5724-4354-ad7b-eaef26b3966f\" (UID: \"4b8235a9-5724-4354-ad7b-eaef26b3966f\") " Oct 30 16:47:26.063125 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.063115 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/029d295c-a65a-4191-9754-a421c21464ba-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/029d295c-a65a-4191-9754-a421c21464ba-istiod-ca-cert\") pod \"029d295c-a65a-4191-9754-a421c21464ba\" (UID: \"029d295c-a65a-4191-9754-a421c21464ba\") " Oct 30 16:47:26.063416 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.063154 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a5f0a2a-f2f2-4bea-98e8-16e6794028de-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a5f0a2a-f2f2-4bea-98e8-16e6794028de-workload-certs\") pod \"0a5f0a2a-f2f2-4bea-98e8-16e6794028de\" (UID: \"0a5f0a2a-f2f2-4bea-98e8-16e6794028de\") " Oct 30 16:47:26.063416 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.063182 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a5f0a2a-f2f2-4bea-98e8-16e6794028de-istio-token\" (UniqueName: \"kubernetes.io/projected/0a5f0a2a-f2f2-4bea-98e8-16e6794028de-istio-token\") pod \"0a5f0a2a-f2f2-4bea-98e8-16e6794028de\" (UID: \"0a5f0a2a-f2f2-4bea-98e8-16e6794028de\") " Oct 30 16:47:26.063416 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.063235 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c254bba7-008e-4724-80a8-8292ae42b7e1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c254bba7-008e-4724-80a8-8292ae42b7e1-istiod-ca-cert\") pod \"c254bba7-008e-4724-80a8-8292ae42b7e1\" (UID: \"c254bba7-008e-4724-80a8-8292ae42b7e1\") " Oct 30 16:47:26.063416 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.063361 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/029d295c-a65a-4191-9754-a421c21464ba-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/029d295c-a65a-4191-9754-a421c21464ba-workload-socket\") pod \"029d295c-a65a-4191-9754-a421c21464ba\" (UID: \"029d295c-a65a-4191-9754-a421c21464ba\") " Oct 30 16:47:26.063769 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.063434 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4b8235a9-5724-4354-ad7b-eaef26b3966f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b8235a9-5724-4354-ad7b-eaef26b3966f-istio-podinfo\") pod \"4b8235a9-5724-4354-ad7b-eaef26b3966f\" (UID: \"4b8235a9-5724-4354-ad7b-eaef26b3966f\") " Oct 30 16:47:26.063769 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.063517 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/029d295c-a65a-4191-9754-a421c21464ba-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/029d295c-a65a-4191-9754-a421c21464ba-istio-data\") pod \"029d295c-a65a-4191-9754-a421c21464ba\" (UID: \"029d295c-a65a-4191-9754-a421c21464ba\") " Oct 30 16:47:26.063769 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.063551 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a5f0a2a-f2f2-4bea-98e8-16e6794028de-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a5f0a2a-f2f2-4bea-98e8-16e6794028de-workload-socket\") pod \"0a5f0a2a-f2f2-4bea-98e8-16e6794028de\" (UID: \"0a5f0a2a-f2f2-4bea-98e8-16e6794028de\") " Oct 30 16:47:26.063769 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.063678 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c254bba7-008e-4724-80a8-8292ae42b7e1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c254bba7-008e-4724-80a8-8292ae42b7e1-workload-socket\") pod \"c254bba7-008e-4724-80a8-8292ae42b7e1\" (UID: \"c254bba7-008e-4724-80a8-8292ae42b7e1\") " Oct 30 16:47:26.063769 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.063722 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b8235a9-5724-4354-ad7b-eaef26b3966f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b8235a9-5724-4354-ad7b-eaef26b3966f-istio-data\") pod \"4b8235a9-5724-4354-ad7b-eaef26b3966f\" (UID: \"4b8235a9-5724-4354-ad7b-eaef26b3966f\") " Oct 30 16:47:26.066761 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.063783 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b8235a9-5724-4354-ad7b-eaef26b3966f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b8235a9-5724-4354-ad7b-eaef26b3966f-credential-socket\") pod \"4b8235a9-5724-4354-ad7b-eaef26b3966f\" (UID: \"4b8235a9-5724-4354-ad7b-eaef26b3966f\") " Oct 30 16:47:26.066761 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.063818 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/029d295c-a65a-4191-9754-a421c21464ba-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/029d295c-a65a-4191-9754-a421c21464ba-workload-certs\") pod \"029d295c-a65a-4191-9754-a421c21464ba\" (UID: \"029d295c-a65a-4191-9754-a421c21464ba\") " Oct 30 16:47:26.066761 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.063893 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c254bba7-008e-4724-80a8-8292ae42b7e1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c254bba7-008e-4724-80a8-8292ae42b7e1-workload-certs\") pod \"c254bba7-008e-4724-80a8-8292ae42b7e1\" (UID: \"c254bba7-008e-4724-80a8-8292ae42b7e1\") " Oct 30 16:47:26.066761 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.063998 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b8235a9-5724-4354-ad7b-eaef26b3966f-istio-token\" (UniqueName: \"kubernetes.io/projected/4b8235a9-5724-4354-ad7b-eaef26b3966f-istio-token\") pod \"4b8235a9-5724-4354-ad7b-eaef26b3966f\" (UID: \"4b8235a9-5724-4354-ad7b-eaef26b3966f\") " Oct 30 16:47:26.066761 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.064068 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a5f0a2a-f2f2-4bea-98e8-16e6794028de-istio-data" pod "0a5f0a2a-f2f2-4bea-98e8-16e6794028de" (UID: "0a5f0a2a-f2f2-4bea-98e8-16e6794028de"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:47:26.066761 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.065102 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/029d295c-a65a-4191-9754-a421c21464ba-workload-certs" pod "029d295c-a65a-4191-9754-a421c21464ba" (UID: "029d295c-a65a-4191-9754-a421c21464ba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:47:26.066761 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.065129 5052 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/029d295c-a65a-4191-9754-a421c21464ba-credential-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:47:26.066761 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.065150 5052 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a5f0a2a-f2f2-4bea-98e8-16e6794028de-istio-data\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:47:26.066761 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.065167 5052 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b8235a9-5724-4354-ad7b-eaef26b3966f-workload-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:47:26.066761 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.065326 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a5f0a2a-f2f2-4bea-98e8-16e6794028de-workload-socket" pod "0a5f0a2a-f2f2-4bea-98e8-16e6794028de" (UID: "0a5f0a2a-f2f2-4bea-98e8-16e6794028de"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:47:26.066761 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.065522 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b8235a9-5724-4354-ad7b-eaef26b3966f-istio-data" pod "4b8235a9-5724-4354-ad7b-eaef26b3966f" (UID: "4b8235a9-5724-4354-ad7b-eaef26b3966f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:47:26.066761 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.065723 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/029d295c-a65a-4191-9754-a421c21464ba-istio-data" pod "029d295c-a65a-4191-9754-a421c21464ba" (UID: "029d295c-a65a-4191-9754-a421c21464ba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:47:26.066761 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.065904 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b8235a9-5724-4354-ad7b-eaef26b3966f-credential-socket" pod "4b8235a9-5724-4354-ad7b-eaef26b3966f" (UID: "4b8235a9-5724-4354-ad7b-eaef26b3966f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:47:26.066761 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.066226 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c254bba7-008e-4724-80a8-8292ae42b7e1-workload-socket" pod "c254bba7-008e-4724-80a8-8292ae42b7e1" (UID: "c254bba7-008e-4724-80a8-8292ae42b7e1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:47:26.066761 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.066594 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b8235a9-5724-4354-ad7b-eaef26b3966f-workload-certs" pod "4b8235a9-5724-4354-ad7b-eaef26b3966f" (UID: "4b8235a9-5724-4354-ad7b-eaef26b3966f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:47:26.067554 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.067508 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a5f0a2a-f2f2-4bea-98e8-16e6794028de-workload-certs" pod "0a5f0a2a-f2f2-4bea-98e8-16e6794028de" (UID: "0a5f0a2a-f2f2-4bea-98e8-16e6794028de"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:47:26.067712 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.067691 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c254bba7-008e-4724-80a8-8292ae42b7e1-istio-data" pod "c254bba7-008e-4724-80a8-8292ae42b7e1" (UID: "c254bba7-008e-4724-80a8-8292ae42b7e1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:47:26.067995 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.067965 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b8235a9-5724-4354-ad7b-eaef26b3966f-istiod-ca-cert" pod "4b8235a9-5724-4354-ad7b-eaef26b3966f" (UID: "4b8235a9-5724-4354-ad7b-eaef26b3966f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:47:26.069176 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.068309 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/029d295c-a65a-4191-9754-a421c21464ba-istiod-ca-cert" pod "029d295c-a65a-4191-9754-a421c21464ba" (UID: "029d295c-a65a-4191-9754-a421c21464ba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:47:26.069176 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.068314 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/029d295c-a65a-4191-9754-a421c21464ba-kube-api-access-ghmvk" pod "029d295c-a65a-4191-9754-a421c21464ba" (UID: "029d295c-a65a-4191-9754-a421c21464ba"). InnerVolumeSpecName "kube-api-access-ghmvk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:47:26.069176 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.068558 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c254bba7-008e-4724-80a8-8292ae42b7e1-workload-certs" pod "c254bba7-008e-4724-80a8-8292ae42b7e1" (UID: "c254bba7-008e-4724-80a8-8292ae42b7e1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:47:26.069176 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.068666 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c254bba7-008e-4724-80a8-8292ae42b7e1-istio-envoy" pod "c254bba7-008e-4724-80a8-8292ae42b7e1" (UID: "c254bba7-008e-4724-80a8-8292ae42b7e1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:47:26.069176 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.068768 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a5f0a2a-f2f2-4bea-98e8-16e6794028de-credential-socket" pod "0a5f0a2a-f2f2-4bea-98e8-16e6794028de" (UID: "0a5f0a2a-f2f2-4bea-98e8-16e6794028de"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:47:26.069176 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.068876 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c254bba7-008e-4724-80a8-8292ae42b7e1-credential-socket" pod "c254bba7-008e-4724-80a8-8292ae42b7e1" (UID: "c254bba7-008e-4724-80a8-8292ae42b7e1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:47:26.069176 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.068928 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b8235a9-5724-4354-ad7b-eaef26b3966f-istio-token" pod "4b8235a9-5724-4354-ad7b-eaef26b3966f" (UID: "4b8235a9-5724-4354-ad7b-eaef26b3966f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:47:26.069650 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.069203 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c254bba7-008e-4724-80a8-8292ae42b7e1-istiod-ca-cert" pod "c254bba7-008e-4724-80a8-8292ae42b7e1" (UID: "c254bba7-008e-4724-80a8-8292ae42b7e1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:47:26.071226 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.071195 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a5f0a2a-f2f2-4bea-98e8-16e6794028de-istio-token" pod "0a5f0a2a-f2f2-4bea-98e8-16e6794028de" (UID: "0a5f0a2a-f2f2-4bea-98e8-16e6794028de"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:47:26.071653 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.071558 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/029d295c-a65a-4191-9754-a421c21464ba-workload-socket" pod "029d295c-a65a-4191-9754-a421c21464ba" (UID: "029d295c-a65a-4191-9754-a421c21464ba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:47:26.073872 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.073728 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/029d295c-a65a-4191-9754-a421c21464ba-istio-podinfo" pod "029d295c-a65a-4191-9754-a421c21464ba" (UID: "029d295c-a65a-4191-9754-a421c21464ba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:47:26.073994 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.073974 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/029d295c-a65a-4191-9754-a421c21464ba-istio-envoy" pod "029d295c-a65a-4191-9754-a421c21464ba" (UID: "029d295c-a65a-4191-9754-a421c21464ba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:47:26.074088 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.074069 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a5f0a2a-f2f2-4bea-98e8-16e6794028de-istiod-ca-cert" pod "0a5f0a2a-f2f2-4bea-98e8-16e6794028de" (UID: "0a5f0a2a-f2f2-4bea-98e8-16e6794028de"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:47:26.074340 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.074291 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c254bba7-008e-4724-80a8-8292ae42b7e1-kube-api-access-dk6pt" pod "c254bba7-008e-4724-80a8-8292ae42b7e1" (UID: "c254bba7-008e-4724-80a8-8292ae42b7e1"). InnerVolumeSpecName "kube-api-access-dk6pt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:47:26.074524 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.074507 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c254bba7-008e-4724-80a8-8292ae42b7e1-istio-podinfo" pod "c254bba7-008e-4724-80a8-8292ae42b7e1" (UID: "c254bba7-008e-4724-80a8-8292ae42b7e1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:47:26.074764 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.074534 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/029d295c-a65a-4191-9754-a421c21464ba-istio-token" pod "029d295c-a65a-4191-9754-a421c21464ba" (UID: "029d295c-a65a-4191-9754-a421c21464ba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:47:26.074895 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.074707 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4b8235a9-5724-4354-ad7b-eaef26b3966f-istio-podinfo" pod "4b8235a9-5724-4354-ad7b-eaef26b3966f" (UID: "4b8235a9-5724-4354-ad7b-eaef26b3966f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:47:26.075091 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.074960 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a5f0a2a-f2f2-4bea-98e8-16e6794028de-istio-envoy" pod "0a5f0a2a-f2f2-4bea-98e8-16e6794028de" (UID: "0a5f0a2a-f2f2-4bea-98e8-16e6794028de"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:47:26.075907 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.075886 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b8235a9-5724-4354-ad7b-eaef26b3966f-kube-api-access-w24fw" pod "4b8235a9-5724-4354-ad7b-eaef26b3966f" (UID: "4b8235a9-5724-4354-ad7b-eaef26b3966f"). InnerVolumeSpecName "kube-api-access-w24fw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:47:26.076113 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.076096 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0a5f0a2a-f2f2-4bea-98e8-16e6794028de-istio-podinfo" pod "0a5f0a2a-f2f2-4bea-98e8-16e6794028de" (UID: "0a5f0a2a-f2f2-4bea-98e8-16e6794028de"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:47:26.076173 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.076121 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a5f0a2a-f2f2-4bea-98e8-16e6794028de-kube-api-access-nt8qt" pod "0a5f0a2a-f2f2-4bea-98e8-16e6794028de" (UID: "0a5f0a2a-f2f2-4bea-98e8-16e6794028de"). InnerVolumeSpecName "kube-api-access-nt8qt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:47:26.076474 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.076458 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c254bba7-008e-4724-80a8-8292ae42b7e1-istio-token" pod "c254bba7-008e-4724-80a8-8292ae42b7e1" (UID: "c254bba7-008e-4724-80a8-8292ae42b7e1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:47:26.076713 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.076697 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b8235a9-5724-4354-ad7b-eaef26b3966f-istio-envoy" pod "4b8235a9-5724-4354-ad7b-eaef26b3966f" (UID: "4b8235a9-5724-4354-ad7b-eaef26b3966f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:47:26.166749 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.166718 5052 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b8235a9-5724-4354-ad7b-eaef26b3966f-credential-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:47:26.166749 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.166747 5052 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/029d295c-a65a-4191-9754-a421c21464ba-workload-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:47:26.166999 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.166760 5052 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c254bba7-008e-4724-80a8-8292ae42b7e1-workload-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:47:26.166999 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.166771 5052 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b8235a9-5724-4354-ad7b-eaef26b3966f-istio-token\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:47:26.166999 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.166782 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghmvk\" (UniqueName: \"kubernetes.io/projected/029d295c-a65a-4191-9754-a421c21464ba-kube-api-access-ghmvk\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:47:26.166999 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.166793 5052 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b8235a9-5724-4354-ad7b-eaef26b3966f-istio-envoy\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:47:26.166999 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.166806 5052 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a5f0a2a-f2f2-4bea-98e8-16e6794028de-istio-podinfo\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:47:26.166999 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.166817 5052 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a5f0a2a-f2f2-4bea-98e8-16e6794028de-istiod-ca-cert\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:47:26.166999 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.166828 5052 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a5f0a2a-f2f2-4bea-98e8-16e6794028de-istio-envoy\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:47:26.166999 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.166839 5052 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/029d295c-a65a-4191-9754-a421c21464ba-istio-token\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:47:26.166999 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.166851 5052 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a5f0a2a-f2f2-4bea-98e8-16e6794028de-credential-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:47:26.166999 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.166863 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nt8qt\" (UniqueName: \"kubernetes.io/projected/0a5f0a2a-f2f2-4bea-98e8-16e6794028de-kube-api-access-nt8qt\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:47:26.166999 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.166875 5052 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b8235a9-5724-4354-ad7b-eaef26b3966f-workload-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:47:26.166999 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.166888 5052 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/029d295c-a65a-4191-9754-a421c21464ba-istio-envoy\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:47:26.166999 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.166899 5052 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c254bba7-008e-4724-80a8-8292ae42b7e1-credential-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:47:26.166999 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.166910 5052 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/029d295c-a65a-4191-9754-a421c21464ba-istio-podinfo\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:47:26.166999 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.166922 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dk6pt\" (UniqueName: \"kubernetes.io/projected/c254bba7-008e-4724-80a8-8292ae42b7e1-kube-api-access-dk6pt\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:47:26.166999 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.166934 5052 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c254bba7-008e-4724-80a8-8292ae42b7e1-istio-podinfo\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:47:26.166999 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.166947 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w24fw\" (UniqueName: \"kubernetes.io/projected/4b8235a9-5724-4354-ad7b-eaef26b3966f-kube-api-access-w24fw\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:47:26.166999 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.166960 5052 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c254bba7-008e-4724-80a8-8292ae42b7e1-istio-token\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:47:26.166999 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.166976 5052 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c254bba7-008e-4724-80a8-8292ae42b7e1-istio-envoy\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:47:26.166999 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.166989 5052 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c254bba7-008e-4724-80a8-8292ae42b7e1-istio-data\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:47:26.166999 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.167000 5052 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b8235a9-5724-4354-ad7b-eaef26b3966f-istiod-ca-cert\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:47:26.167689 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.167012 5052 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/029d295c-a65a-4191-9754-a421c21464ba-istiod-ca-cert\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:47:26.167689 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.167024 5052 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a5f0a2a-f2f2-4bea-98e8-16e6794028de-workload-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:47:26.167689 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.167036 5052 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a5f0a2a-f2f2-4bea-98e8-16e6794028de-istio-token\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:47:26.167689 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.167049 5052 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c254bba7-008e-4724-80a8-8292ae42b7e1-istiod-ca-cert\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:47:26.167689 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.167061 5052 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/029d295c-a65a-4191-9754-a421c21464ba-workload-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:47:26.167689 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.167074 5052 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b8235a9-5724-4354-ad7b-eaef26b3966f-istio-podinfo\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:47:26.167689 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.167086 5052 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/029d295c-a65a-4191-9754-a421c21464ba-istio-data\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:47:26.167689 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.167100 5052 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a5f0a2a-f2f2-4bea-98e8-16e6794028de-workload-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:47:26.167689 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.167112 5052 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c254bba7-008e-4724-80a8-8292ae42b7e1-workload-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:47:26.167689 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.167125 5052 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b8235a9-5724-4354-ad7b-eaef26b3966f-istio-data\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:47:26.407140 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.407101 5052 generic.go:356] "Generic (PLEG): container finished" podID="4b8235a9-5724-4354-ad7b-eaef26b3966f" containerID="06eae2719515f348df2fe76d5997e5a8ea46ffb83a0d24cce2f7885dfa88b985" exitCode=0 Oct 30 16:47:26.407649 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.407457 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-46brl" Oct 30 16:47:26.432422 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.407479 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-46brl" event={"ID":"4b8235a9-5724-4354-ad7b-eaef26b3966f","Type":"ContainerDied","Data":"06eae2719515f348df2fe76d5997e5a8ea46ffb83a0d24cce2f7885dfa88b985"} Oct 30 16:47:26.432422 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.427430 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-46brl" event={"ID":"4b8235a9-5724-4354-ad7b-eaef26b3966f","Type":"ContainerDied","Data":"116f02411087ca7344713291a12ea7df517231d5d0f667bfa85825ab2c0e949b"} Oct 30 16:47:26.432422 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.427474 5052 scope.go:122] "RemoveContainer" containerID="06eae2719515f348df2fe76d5997e5a8ea46ffb83a0d24cce2f7885dfa88b985" Oct 30 16:47:26.448314 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.448276 5052 generic.go:356] "Generic (PLEG): container finished" podID="029d295c-a65a-4191-9754-a421c21464ba" containerID="8b4392bec4b7bb73bffe133bc736111a10345438eb82677a89f00d7358eb3889" exitCode=0 Oct 30 16:47:26.459645 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.448984 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-xjhj4" Oct 30 16:47:26.459645 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.449719 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-xjhj4" event={"ID":"029d295c-a65a-4191-9754-a421c21464ba","Type":"ContainerDied","Data":"8b4392bec4b7bb73bffe133bc736111a10345438eb82677a89f00d7358eb3889"} Oct 30 16:47:26.459645 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.451720 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-xjhj4" event={"ID":"029d295c-a65a-4191-9754-a421c21464ba","Type":"ContainerDied","Data":"2816532f7db38518153d64bee4ebbd4dcfdd2d08948225428c32a364c50d9443"} Oct 30 16:47:26.469129 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.468903 5052 generic.go:356] "Generic (PLEG): container finished" podID="0a5f0a2a-f2f2-4bea-98e8-16e6794028de" containerID="48c4105bf5427439a31b5cd61c955f82d1e0fa0e1b84b87baeed6c3b947b0793" exitCode=0 Oct 30 16:47:26.469129 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.468978 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-vbvvh" event={"ID":"0a5f0a2a-f2f2-4bea-98e8-16e6794028de","Type":"ContainerDied","Data":"48c4105bf5427439a31b5cd61c955f82d1e0fa0e1b84b87baeed6c3b947b0793"} Oct 30 16:47:26.469129 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.469017 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-vbvvh" event={"ID":"0a5f0a2a-f2f2-4bea-98e8-16e6794028de","Type":"ContainerDied","Data":"9d0ccf93dffcf6bc05ec61a46436061eb042d70da54bc8ee611c6b1449b20885"} Oct 30 16:47:26.469573 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.469428 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-vbvvh" Oct 30 16:47:26.473843 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.473739 5052 generic.go:356] "Generic (PLEG): container finished" podID="c254bba7-008e-4724-80a8-8292ae42b7e1" containerID="cf30a6a0225cb105f4c7c0096f168d0f62a4cdbc453dbb9fe995055af8d31aef" exitCode=0 Oct 30 16:47:26.473843 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.473786 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-kznnt" event={"ID":"c254bba7-008e-4724-80a8-8292ae42b7e1","Type":"ContainerDied","Data":"cf30a6a0225cb105f4c7c0096f168d0f62a4cdbc453dbb9fe995055af8d31aef"} Oct 30 16:47:26.473843 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.473794 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-kznnt" Oct 30 16:47:26.473843 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.473824 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-kznnt" event={"ID":"c254bba7-008e-4724-80a8-8292ae42b7e1","Type":"ContainerDied","Data":"8de520285b3e0900170f1e2e37791b4f279bec3b841969b612f0e3a1d88c0206"} Oct 30 16:47:26.483646 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.483505 5052 scope.go:122] "RemoveContainer" containerID="71f6f0d2b5d722966a3978154591e1372551da04dc4e5d1c33602851e57f1f3c" Oct 30 16:47:26.494173 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.494153 5052 scope.go:122] "RemoveContainer" containerID="06eae2719515f348df2fe76d5997e5a8ea46ffb83a0d24cce2f7885dfa88b985" Oct 30 16:47:26.494441 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:47:26.494421 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06eae2719515f348df2fe76d5997e5a8ea46ffb83a0d24cce2f7885dfa88b985\": container with ID starting with 06eae2719515f348df2fe76d5997e5a8ea46ffb83a0d24cce2f7885dfa88b985 not found: ID does not exist" containerID="06eae2719515f348df2fe76d5997e5a8ea46ffb83a0d24cce2f7885dfa88b985" Oct 30 16:47:26.494497 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.494450 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06eae2719515f348df2fe76d5997e5a8ea46ffb83a0d24cce2f7885dfa88b985"} err="failed to get container status \"06eae2719515f348df2fe76d5997e5a8ea46ffb83a0d24cce2f7885dfa88b985\": rpc error: code = NotFound desc = could not find container \"06eae2719515f348df2fe76d5997e5a8ea46ffb83a0d24cce2f7885dfa88b985\": container with ID starting with 06eae2719515f348df2fe76d5997e5a8ea46ffb83a0d24cce2f7885dfa88b985 not found: ID does not exist" Oct 30 16:47:26.494497 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.494467 5052 scope.go:122] "RemoveContainer" containerID="71f6f0d2b5d722966a3978154591e1372551da04dc4e5d1c33602851e57f1f3c" Oct 30 16:47:26.494766 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:47:26.494745 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71f6f0d2b5d722966a3978154591e1372551da04dc4e5d1c33602851e57f1f3c\": container with ID starting with 71f6f0d2b5d722966a3978154591e1372551da04dc4e5d1c33602851e57f1f3c not found: ID does not exist" containerID="71f6f0d2b5d722966a3978154591e1372551da04dc4e5d1c33602851e57f1f3c" Oct 30 16:47:26.494847 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.494770 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71f6f0d2b5d722966a3978154591e1372551da04dc4e5d1c33602851e57f1f3c"} err="failed to get container status \"71f6f0d2b5d722966a3978154591e1372551da04dc4e5d1c33602851e57f1f3c\": rpc error: code = NotFound desc = could not find container \"71f6f0d2b5d722966a3978154591e1372551da04dc4e5d1c33602851e57f1f3c\": container with ID starting with 71f6f0d2b5d722966a3978154591e1372551da04dc4e5d1c33602851e57f1f3c not found: ID does not exist" Oct 30 16:47:26.494847 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.494787 5052 scope.go:122] "RemoveContainer" containerID="8b4392bec4b7bb73bffe133bc736111a10345438eb82677a89f00d7358eb3889" Oct 30 16:47:26.534619 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.534595 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-xjhj4"] Oct 30 16:47:26.540294 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.540271 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-xjhj4"] Oct 30 16:47:26.541494 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.541479 5052 scope.go:122] "RemoveContainer" containerID="8b4392bec4b7bb73bffe133bc736111a10345438eb82677a89f00d7358eb3889" Oct 30 16:47:26.541805 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:47:26.541785 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b4392bec4b7bb73bffe133bc736111a10345438eb82677a89f00d7358eb3889\": container with ID starting with 8b4392bec4b7bb73bffe133bc736111a10345438eb82677a89f00d7358eb3889 not found: ID does not exist" containerID="8b4392bec4b7bb73bffe133bc736111a10345438eb82677a89f00d7358eb3889" Oct 30 16:47:26.541890 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.541810 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b4392bec4b7bb73bffe133bc736111a10345438eb82677a89f00d7358eb3889"} err="failed to get container status \"8b4392bec4b7bb73bffe133bc736111a10345438eb82677a89f00d7358eb3889\": rpc error: code = NotFound desc = could not find container \"8b4392bec4b7bb73bffe133bc736111a10345438eb82677a89f00d7358eb3889\": container with ID starting with 8b4392bec4b7bb73bffe133bc736111a10345438eb82677a89f00d7358eb3889 not found: ID does not exist" Oct 30 16:47:26.541890 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.541827 5052 scope.go:122] "RemoveContainer" containerID="48c4105bf5427439a31b5cd61c955f82d1e0fa0e1b84b87baeed6c3b947b0793" Oct 30 16:47:26.552468 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.552452 5052 scope.go:122] "RemoveContainer" containerID="48c4105bf5427439a31b5cd61c955f82d1e0fa0e1b84b87baeed6c3b947b0793" Oct 30 16:47:26.552779 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:47:26.552753 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48c4105bf5427439a31b5cd61c955f82d1e0fa0e1b84b87baeed6c3b947b0793\": container with ID starting with 48c4105bf5427439a31b5cd61c955f82d1e0fa0e1b84b87baeed6c3b947b0793 not found: ID does not exist" containerID="48c4105bf5427439a31b5cd61c955f82d1e0fa0e1b84b87baeed6c3b947b0793" Oct 30 16:47:26.552831 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.552788 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48c4105bf5427439a31b5cd61c955f82d1e0fa0e1b84b87baeed6c3b947b0793"} err="failed to get container status \"48c4105bf5427439a31b5cd61c955f82d1e0fa0e1b84b87baeed6c3b947b0793\": rpc error: code = NotFound desc = could not find container \"48c4105bf5427439a31b5cd61c955f82d1e0fa0e1b84b87baeed6c3b947b0793\": container with ID starting with 48c4105bf5427439a31b5cd61c955f82d1e0fa0e1b84b87baeed6c3b947b0793 not found: ID does not exist" Oct 30 16:47:26.552831 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.552806 5052 scope.go:122] "RemoveContainer" containerID="cf30a6a0225cb105f4c7c0096f168d0f62a4cdbc453dbb9fe995055af8d31aef" Oct 30 16:47:26.562777 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.562761 5052 scope.go:122] "RemoveContainer" containerID="cf30a6a0225cb105f4c7c0096f168d0f62a4cdbc453dbb9fe995055af8d31aef" Oct 30 16:47:26.563028 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:47:26.563008 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf30a6a0225cb105f4c7c0096f168d0f62a4cdbc453dbb9fe995055af8d31aef\": container with ID starting with cf30a6a0225cb105f4c7c0096f168d0f62a4cdbc453dbb9fe995055af8d31aef not found: ID does not exist" containerID="cf30a6a0225cb105f4c7c0096f168d0f62a4cdbc453dbb9fe995055af8d31aef" Oct 30 16:47:26.563080 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.563036 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf30a6a0225cb105f4c7c0096f168d0f62a4cdbc453dbb9fe995055af8d31aef"} err="failed to get container status \"cf30a6a0225cb105f4c7c0096f168d0f62a4cdbc453dbb9fe995055af8d31aef\": rpc error: code = NotFound desc = could not find container \"cf30a6a0225cb105f4c7c0096f168d0f62a4cdbc453dbb9fe995055af8d31aef\": container with ID starting with cf30a6a0225cb105f4c7c0096f168d0f62a4cdbc453dbb9fe995055af8d31aef not found: ID does not exist" Oct 30 16:47:26.577542 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.577516 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-46brl"] Oct 30 16:47:26.582260 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.582241 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-46brl"] Oct 30 16:47:26.600038 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.600013 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="029d295c-a65a-4191-9754-a421c21464ba" path="/var/lib/kubelet/pods/029d295c-a65a-4191-9754-a421c21464ba/volumes" Oct 30 16:47:26.600520 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.600506 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b8235a9-5724-4354-ad7b-eaef26b3966f" path="/var/lib/kubelet/pods/4b8235a9-5724-4354-ad7b-eaef26b3966f/volumes" Oct 30 16:47:26.619070 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.619045 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-vbvvh"] Oct 30 16:47:26.625718 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.625699 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-vbvvh"] Oct 30 16:47:26.662376 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.662233 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-kznnt"] Oct 30 16:47:26.665986 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:26.665966 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-kznnt"] Oct 30 16:47:28.599860 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:28.599831 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a5f0a2a-f2f2-4bea-98e8-16e6794028de" path="/var/lib/kubelet/pods/0a5f0a2a-f2f2-4bea-98e8-16e6794028de/volumes" Oct 30 16:47:28.600522 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:28.600505 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c254bba7-008e-4724-80a8-8292ae42b7e1" path="/var/lib/kubelet/pods/c254bba7-008e-4724-80a8-8292ae42b7e1/volumes" Oct 30 16:47:29.642858 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:29.642738 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-b8af660d-787d8764b-h9rfs" podUID="2c983762-43df-4410-9001-d5d25c7a45f3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 16:47:31.736119 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:31.736082 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e5674c-5969bf95bc-jrsfj"] Oct 30 16:47:31.736916 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:31.736899 5052 cpu_manager.go:414] "containerMap: removing container" podUID="c254bba7-008e-4724-80a8-8292ae42b7e1" containerName="istio-proxy" Oct 30 16:47:31.736983 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:31.736916 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c254bba7-008e-4724-80a8-8292ae42b7e1" containerName="istio-proxy" Oct 30 16:47:31.736983 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:31.736928 5052 cpu_manager.go:414] "containerMap: removing container" podUID="4b8235a9-5724-4354-ad7b-eaef26b3966f" containerName="istio-proxy" Oct 30 16:47:31.736983 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:31.736934 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b8235a9-5724-4354-ad7b-eaef26b3966f" containerName="istio-proxy" Oct 30 16:47:31.736983 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:31.736941 5052 cpu_manager.go:414] "containerMap: removing container" podUID="029d295c-a65a-4191-9754-a421c21464ba" containerName="istio-proxy" Oct 30 16:47:31.736983 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:31.736946 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="029d295c-a65a-4191-9754-a421c21464ba" containerName="istio-proxy" Oct 30 16:47:31.736983 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:31.736960 5052 cpu_manager.go:414] "containerMap: removing container" podUID="46e348ba-6e79-4665-a691-c7632a581520" containerName="test-curl-8962-2f0f36" Oct 30 16:47:31.736983 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:31.736965 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46e348ba-6e79-4665-a691-c7632a581520" containerName="test-curl-8962-2f0f36" Oct 30 16:47:31.737200 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:31.736989 5052 cpu_manager.go:414] "containerMap: removing container" podUID="0a5f0a2a-f2f2-4bea-98e8-16e6794028de" containerName="istio-proxy" Oct 30 16:47:31.737200 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:31.736993 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a5f0a2a-f2f2-4bea-98e8-16e6794028de" containerName="istio-proxy" Oct 30 16:47:31.737200 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:31.737003 5052 cpu_manager.go:414] "containerMap: removing container" podUID="4b8235a9-5724-4354-ad7b-eaef26b3966f" containerName="istio-proxy" Oct 30 16:47:31.737200 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:31.737007 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b8235a9-5724-4354-ad7b-eaef26b3966f" containerName="istio-proxy" Oct 30 16:47:31.737200 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:31.737110 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a5f0a2a-f2f2-4bea-98e8-16e6794028de" containerName="istio-proxy" Oct 30 16:47:31.737200 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:31.737120 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b8235a9-5724-4354-ad7b-eaef26b3966f" containerName="istio-proxy" Oct 30 16:47:31.737200 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:31.737132 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="c254bba7-008e-4724-80a8-8292ae42b7e1" containerName="istio-proxy" Oct 30 16:47:31.737200 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:31.737137 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b8235a9-5724-4354-ad7b-eaef26b3966f" containerName="istio-proxy" Oct 30 16:47:31.737200 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:31.737143 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="029d295c-a65a-4191-9754-a421c21464ba" containerName="istio-proxy" Oct 30 16:47:31.737200 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:31.737150 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="46e348ba-6e79-4665-a691-c7632a581520" containerName="test-curl-8962-2f0f36" Oct 30 16:47:31.743271 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:31.743246 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e5674c-5969bf95bc-jrsfj" Oct 30 16:47:31.765051 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:31.764732 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-e5674c-serving-cert\"" Oct 30 16:47:31.765465 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:31.765442 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e5674c-5969bf95bc-jrsfj"] Oct 30 16:47:31.813833 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:31.813798 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4wld\" (UniqueName: \"kubernetes.io/projected/8dbf7c5d-6618-43f3-8a40-94e3a7d14662-kube-api-access-c4wld\") pod \"maas-api-e2e-api-b-e5674c-5969bf95bc-jrsfj\" (UID: \"8dbf7c5d-6618-43f3-8a40-94e3a7d14662\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e5674c-5969bf95bc-jrsfj" Oct 30 16:47:31.814002 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:31.813982 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8dbf7c5d-6618-43f3-8a40-94e3a7d14662-maas-api-tls\") pod \"maas-api-e2e-api-b-e5674c-5969bf95bc-jrsfj\" (UID: \"8dbf7c5d-6618-43f3-8a40-94e3a7d14662\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e5674c-5969bf95bc-jrsfj" Oct 30 16:47:31.814096 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:31.814062 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8dbf7c5d-6618-43f3-8a40-94e3a7d14662-openshift-service-ca\") pod \"maas-api-e2e-api-b-e5674c-5969bf95bc-jrsfj\" (UID: \"8dbf7c5d-6618-43f3-8a40-94e3a7d14662\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e5674c-5969bf95bc-jrsfj" Oct 30 16:47:31.814287 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:31.814254 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8dbf7c5d-6618-43f3-8a40-94e3a7d14662-metrics-certs\") pod \"maas-api-e2e-api-b-e5674c-5969bf95bc-jrsfj\" (UID: \"8dbf7c5d-6618-43f3-8a40-94e3a7d14662\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e5674c-5969bf95bc-jrsfj" Oct 30 16:47:31.924274 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:31.924216 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4wld\" (UniqueName: \"kubernetes.io/projected/8dbf7c5d-6618-43f3-8a40-94e3a7d14662-kube-api-access-c4wld\") pod \"maas-api-e2e-api-b-e5674c-5969bf95bc-jrsfj\" (UID: \"8dbf7c5d-6618-43f3-8a40-94e3a7d14662\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e5674c-5969bf95bc-jrsfj" Oct 30 16:47:31.924496 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:31.924396 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8dbf7c5d-6618-43f3-8a40-94e3a7d14662-maas-api-tls\") pod \"maas-api-e2e-api-b-e5674c-5969bf95bc-jrsfj\" (UID: \"8dbf7c5d-6618-43f3-8a40-94e3a7d14662\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e5674c-5969bf95bc-jrsfj" Oct 30 16:47:31.924496 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:31.924427 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8dbf7c5d-6618-43f3-8a40-94e3a7d14662-openshift-service-ca\") pod \"maas-api-e2e-api-b-e5674c-5969bf95bc-jrsfj\" (UID: \"8dbf7c5d-6618-43f3-8a40-94e3a7d14662\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e5674c-5969bf95bc-jrsfj" Oct 30 16:47:31.924676 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:31.924645 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8dbf7c5d-6618-43f3-8a40-94e3a7d14662-metrics-certs\") pod \"maas-api-e2e-api-b-e5674c-5969bf95bc-jrsfj\" (UID: \"8dbf7c5d-6618-43f3-8a40-94e3a7d14662\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e5674c-5969bf95bc-jrsfj" Oct 30 16:47:31.926397 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:31.926372 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8dbf7c5d-6618-43f3-8a40-94e3a7d14662-openshift-service-ca\") pod \"maas-api-e2e-api-b-e5674c-5969bf95bc-jrsfj\" (UID: \"8dbf7c5d-6618-43f3-8a40-94e3a7d14662\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e5674c-5969bf95bc-jrsfj" Oct 30 16:47:31.943397 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:31.943366 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8dbf7c5d-6618-43f3-8a40-94e3a7d14662-maas-api-tls\") pod \"maas-api-e2e-api-b-e5674c-5969bf95bc-jrsfj\" (UID: \"8dbf7c5d-6618-43f3-8a40-94e3a7d14662\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e5674c-5969bf95bc-jrsfj" Oct 30 16:47:31.975002 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:31.962621 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4wld\" (UniqueName: \"kubernetes.io/projected/8dbf7c5d-6618-43f3-8a40-94e3a7d14662-kube-api-access-c4wld\") pod \"maas-api-e2e-api-b-e5674c-5969bf95bc-jrsfj\" (UID: \"8dbf7c5d-6618-43f3-8a40-94e3a7d14662\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e5674c-5969bf95bc-jrsfj" Oct 30 16:47:31.985711 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:31.985677 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8dbf7c5d-6618-43f3-8a40-94e3a7d14662-metrics-certs\") pod \"maas-api-e2e-api-b-e5674c-5969bf95bc-jrsfj\" (UID: \"8dbf7c5d-6618-43f3-8a40-94e3a7d14662\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e5674c-5969bf95bc-jrsfj" Oct 30 16:47:32.069657 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:32.066745 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e5674c-5969bf95bc-jrsfj" Oct 30 16:47:32.406209 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:47:32.406181 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dbf7c5d_6618_43f3_8a40_94e3a7d14662.slice/crio-cc194b36e87717c632b78a60195cc42833dd9863f7f7d92466f715561061f4d2 WatchSource:0}: Error finding container cc194b36e87717c632b78a60195cc42833dd9863f7f7d92466f715561061f4d2: Status 404 returned error can't find the container with id cc194b36e87717c632b78a60195cc42833dd9863f7f7d92466f715561061f4d2 Oct 30 16:47:32.406788 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:32.406764 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e5674c-5969bf95bc-jrsfj"] Oct 30 16:47:32.407959 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:32.407937 5052 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 16:47:32.526552 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:32.526516 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e5674c-5969bf95bc-jrsfj" event={"ID":"8dbf7c5d-6618-43f3-8a40-94e3a7d14662","Type":"ContainerStarted","Data":"cc194b36e87717c632b78a60195cc42833dd9863f7f7d92466f715561061f4d2"} Oct 30 16:47:33.583462 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:33.583428 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e5674c-5969bf95bc-jrsfj" event={"ID":"8dbf7c5d-6618-43f3-8a40-94e3a7d14662","Type":"ContainerStarted","Data":"75373b1db1fd33a998a1ffb62471e48229104d78c1378ea07fc83a518f4b9447"} Oct 30 16:47:33.583889 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:33.583635 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e5674c-5969bf95bc-jrsfj" Oct 30 16:47:33.638307 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:33.638241 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e5674c-5969bf95bc-jrsfj" podStartSLOduration=2.264939827 podStartE2EDuration="2.638227571s" totalImagesPullingTime="373.287744ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:47:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:47:33.637181806 +0000 UTC m=+1609.164517017" watchObservedRunningTime="2026-10-30 16:47:33.638227571 +0000 UTC m=+1609.165562789" Oct 30 16:47:34.722714 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:34.722677 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-ca5971-openshift-default-f54d56b47-qrn9w"] Oct 30 16:47:34.727920 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:34.727897 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ca5971-openshift-default-f54d56b47-qrn9w" Oct 30 16:47:34.756527 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:34.756480 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-ca5971-openshift-default-f54d56b47-qrn9w"] Oct 30 16:47:34.806799 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:34.806763 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbb6daa4-cde3-4e3f-8343-1b8ca62f8390-credential-socket\") pod \"e2e-api-b-ca5971-openshift-default-f54d56b47-qrn9w\" (UID: \"cbb6daa4-cde3-4e3f-8343-1b8ca62f8390\") " pod="openshift-ingress/e2e-api-b-ca5971-openshift-default-f54d56b47-qrn9w" Oct 30 16:47:34.806998 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:34.806819 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbb6daa4-cde3-4e3f-8343-1b8ca62f8390-istio-podinfo\") pod \"e2e-api-b-ca5971-openshift-default-f54d56b47-qrn9w\" (UID: \"cbb6daa4-cde3-4e3f-8343-1b8ca62f8390\") " pod="openshift-ingress/e2e-api-b-ca5971-openshift-default-f54d56b47-qrn9w" Oct 30 16:47:34.806998 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:34.806880 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbb6daa4-cde3-4e3f-8343-1b8ca62f8390-workload-certs\") pod \"e2e-api-b-ca5971-openshift-default-f54d56b47-qrn9w\" (UID: \"cbb6daa4-cde3-4e3f-8343-1b8ca62f8390\") " pod="openshift-ingress/e2e-api-b-ca5971-openshift-default-f54d56b47-qrn9w" Oct 30 16:47:34.806998 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:34.806909 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbb6daa4-cde3-4e3f-8343-1b8ca62f8390-istio-token\") pod \"e2e-api-b-ca5971-openshift-default-f54d56b47-qrn9w\" (UID: \"cbb6daa4-cde3-4e3f-8343-1b8ca62f8390\") " pod="openshift-ingress/e2e-api-b-ca5971-openshift-default-f54d56b47-qrn9w" Oct 30 16:47:34.806998 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:34.806930 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbb6daa4-cde3-4e3f-8343-1b8ca62f8390-istio-envoy\") pod \"e2e-api-b-ca5971-openshift-default-f54d56b47-qrn9w\" (UID: \"cbb6daa4-cde3-4e3f-8343-1b8ca62f8390\") " pod="openshift-ingress/e2e-api-b-ca5971-openshift-default-f54d56b47-qrn9w" Oct 30 16:47:34.807217 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:34.807037 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbb6daa4-cde3-4e3f-8343-1b8ca62f8390-istiod-ca-cert\") pod \"e2e-api-b-ca5971-openshift-default-f54d56b47-qrn9w\" (UID: \"cbb6daa4-cde3-4e3f-8343-1b8ca62f8390\") " pod="openshift-ingress/e2e-api-b-ca5971-openshift-default-f54d56b47-qrn9w" Oct 30 16:47:34.807217 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:34.807093 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dh6pq\" (UniqueName: \"kubernetes.io/projected/cbb6daa4-cde3-4e3f-8343-1b8ca62f8390-kube-api-access-dh6pq\") pod \"e2e-api-b-ca5971-openshift-default-f54d56b47-qrn9w\" (UID: \"cbb6daa4-cde3-4e3f-8343-1b8ca62f8390\") " pod="openshift-ingress/e2e-api-b-ca5971-openshift-default-f54d56b47-qrn9w" Oct 30 16:47:34.807217 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:34.807112 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbb6daa4-cde3-4e3f-8343-1b8ca62f8390-istio-data\") pod \"e2e-api-b-ca5971-openshift-default-f54d56b47-qrn9w\" (UID: \"cbb6daa4-cde3-4e3f-8343-1b8ca62f8390\") " pod="openshift-ingress/e2e-api-b-ca5971-openshift-default-f54d56b47-qrn9w" Oct 30 16:47:34.807217 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:34.807186 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbb6daa4-cde3-4e3f-8343-1b8ca62f8390-workload-socket\") pod \"e2e-api-b-ca5971-openshift-default-f54d56b47-qrn9w\" (UID: \"cbb6daa4-cde3-4e3f-8343-1b8ca62f8390\") " pod="openshift-ingress/e2e-api-b-ca5971-openshift-default-f54d56b47-qrn9w" Oct 30 16:47:34.809794 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:34.809764 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-ca5971-openshift-default-dockercfg-6lckq\"" Oct 30 16:47:34.937602 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:34.935057 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbb6daa4-cde3-4e3f-8343-1b8ca62f8390-workload-socket\") pod \"e2e-api-b-ca5971-openshift-default-f54d56b47-qrn9w\" (UID: \"cbb6daa4-cde3-4e3f-8343-1b8ca62f8390\") " pod="openshift-ingress/e2e-api-b-ca5971-openshift-default-f54d56b47-qrn9w" Oct 30 16:47:34.945616 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:34.944791 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbb6daa4-cde3-4e3f-8343-1b8ca62f8390-workload-socket\") pod \"e2e-api-b-ca5971-openshift-default-f54d56b47-qrn9w\" (UID: \"cbb6daa4-cde3-4e3f-8343-1b8ca62f8390\") " pod="openshift-ingress/e2e-api-b-ca5971-openshift-default-f54d56b47-qrn9w" Oct 30 16:47:34.945616 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:34.945066 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbb6daa4-cde3-4e3f-8343-1b8ca62f8390-credential-socket\") pod \"e2e-api-b-ca5971-openshift-default-f54d56b47-qrn9w\" (UID: \"cbb6daa4-cde3-4e3f-8343-1b8ca62f8390\") " pod="openshift-ingress/e2e-api-b-ca5971-openshift-default-f54d56b47-qrn9w" Oct 30 16:47:34.945616 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:34.945125 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbb6daa4-cde3-4e3f-8343-1b8ca62f8390-istio-podinfo\") pod \"e2e-api-b-ca5971-openshift-default-f54d56b47-qrn9w\" (UID: \"cbb6daa4-cde3-4e3f-8343-1b8ca62f8390\") " pod="openshift-ingress/e2e-api-b-ca5971-openshift-default-f54d56b47-qrn9w" Oct 30 16:47:34.945616 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:34.945155 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbb6daa4-cde3-4e3f-8343-1b8ca62f8390-workload-certs\") pod \"e2e-api-b-ca5971-openshift-default-f54d56b47-qrn9w\" (UID: \"cbb6daa4-cde3-4e3f-8343-1b8ca62f8390\") " pod="openshift-ingress/e2e-api-b-ca5971-openshift-default-f54d56b47-qrn9w" Oct 30 16:47:34.945616 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:34.945193 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbb6daa4-cde3-4e3f-8343-1b8ca62f8390-istio-token\") pod \"e2e-api-b-ca5971-openshift-default-f54d56b47-qrn9w\" (UID: \"cbb6daa4-cde3-4e3f-8343-1b8ca62f8390\") " pod="openshift-ingress/e2e-api-b-ca5971-openshift-default-f54d56b47-qrn9w" Oct 30 16:47:34.945616 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:34.945219 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbb6daa4-cde3-4e3f-8343-1b8ca62f8390-istio-envoy\") pod \"e2e-api-b-ca5971-openshift-default-f54d56b47-qrn9w\" (UID: \"cbb6daa4-cde3-4e3f-8343-1b8ca62f8390\") " pod="openshift-ingress/e2e-api-b-ca5971-openshift-default-f54d56b47-qrn9w" Oct 30 16:47:34.945616 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:34.945355 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbb6daa4-cde3-4e3f-8343-1b8ca62f8390-istiod-ca-cert\") pod \"e2e-api-b-ca5971-openshift-default-f54d56b47-qrn9w\" (UID: \"cbb6daa4-cde3-4e3f-8343-1b8ca62f8390\") " pod="openshift-ingress/e2e-api-b-ca5971-openshift-default-f54d56b47-qrn9w" Oct 30 16:47:34.945616 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:34.945451 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dh6pq\" (UniqueName: \"kubernetes.io/projected/cbb6daa4-cde3-4e3f-8343-1b8ca62f8390-kube-api-access-dh6pq\") pod \"e2e-api-b-ca5971-openshift-default-f54d56b47-qrn9w\" (UID: \"cbb6daa4-cde3-4e3f-8343-1b8ca62f8390\") " pod="openshift-ingress/e2e-api-b-ca5971-openshift-default-f54d56b47-qrn9w" Oct 30 16:47:34.945616 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:34.945483 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbb6daa4-cde3-4e3f-8343-1b8ca62f8390-istio-data\") pod \"e2e-api-b-ca5971-openshift-default-f54d56b47-qrn9w\" (UID: \"cbb6daa4-cde3-4e3f-8343-1b8ca62f8390\") " pod="openshift-ingress/e2e-api-b-ca5971-openshift-default-f54d56b47-qrn9w" Oct 30 16:47:34.946307 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:34.946094 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbb6daa4-cde3-4e3f-8343-1b8ca62f8390-istio-data\") pod \"e2e-api-b-ca5971-openshift-default-f54d56b47-qrn9w\" (UID: \"cbb6daa4-cde3-4e3f-8343-1b8ca62f8390\") " pod="openshift-ingress/e2e-api-b-ca5971-openshift-default-f54d56b47-qrn9w" Oct 30 16:47:34.946372 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:34.946358 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbb6daa4-cde3-4e3f-8343-1b8ca62f8390-credential-socket\") pod \"e2e-api-b-ca5971-openshift-default-f54d56b47-qrn9w\" (UID: \"cbb6daa4-cde3-4e3f-8343-1b8ca62f8390\") " pod="openshift-ingress/e2e-api-b-ca5971-openshift-default-f54d56b47-qrn9w" Oct 30 16:47:34.961608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:34.947616 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbb6daa4-cde3-4e3f-8343-1b8ca62f8390-istiod-ca-cert\") pod \"e2e-api-b-ca5971-openshift-default-f54d56b47-qrn9w\" (UID: \"cbb6daa4-cde3-4e3f-8343-1b8ca62f8390\") " pod="openshift-ingress/e2e-api-b-ca5971-openshift-default-f54d56b47-qrn9w" Oct 30 16:47:34.961608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:34.948450 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbb6daa4-cde3-4e3f-8343-1b8ca62f8390-workload-certs\") pod \"e2e-api-b-ca5971-openshift-default-f54d56b47-qrn9w\" (UID: \"cbb6daa4-cde3-4e3f-8343-1b8ca62f8390\") " pod="openshift-ingress/e2e-api-b-ca5971-openshift-default-f54d56b47-qrn9w" Oct 30 16:47:34.983303 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:34.983217 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbb6daa4-cde3-4e3f-8343-1b8ca62f8390-istio-podinfo\") pod \"e2e-api-b-ca5971-openshift-default-f54d56b47-qrn9w\" (UID: \"cbb6daa4-cde3-4e3f-8343-1b8ca62f8390\") " pod="openshift-ingress/e2e-api-b-ca5971-openshift-default-f54d56b47-qrn9w" Oct 30 16:47:35.006609 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:35.003496 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbb6daa4-cde3-4e3f-8343-1b8ca62f8390-istio-envoy\") pod \"e2e-api-b-ca5971-openshift-default-f54d56b47-qrn9w\" (UID: \"cbb6daa4-cde3-4e3f-8343-1b8ca62f8390\") " pod="openshift-ingress/e2e-api-b-ca5971-openshift-default-f54d56b47-qrn9w" Oct 30 16:47:35.006609 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:35.005789 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dh6pq\" (UniqueName: \"kubernetes.io/projected/cbb6daa4-cde3-4e3f-8343-1b8ca62f8390-kube-api-access-dh6pq\") pod \"e2e-api-b-ca5971-openshift-default-f54d56b47-qrn9w\" (UID: \"cbb6daa4-cde3-4e3f-8343-1b8ca62f8390\") " pod="openshift-ingress/e2e-api-b-ca5971-openshift-default-f54d56b47-qrn9w" Oct 30 16:47:35.012420 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:35.012391 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbb6daa4-cde3-4e3f-8343-1b8ca62f8390-istio-token\") pod \"e2e-api-b-ca5971-openshift-default-f54d56b47-qrn9w\" (UID: \"cbb6daa4-cde3-4e3f-8343-1b8ca62f8390\") " pod="openshift-ingress/e2e-api-b-ca5971-openshift-default-f54d56b47-qrn9w" Oct 30 16:47:35.049218 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:35.048924 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ca5971-openshift-default-f54d56b47-qrn9w" Oct 30 16:47:35.675184 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:35.675156 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-ca5971-openshift-default-f54d56b47-qrn9w"] Oct 30 16:47:35.675469 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:47:35.675436 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbb6daa4_cde3_4e3f_8343_1b8ca62f8390.slice/crio-ce4f146644a650cf6d64d0d656f90824e6011f7fabd8c666d6c7a4349fe21630 WatchSource:0}: Error finding container ce4f146644a650cf6d64d0d656f90824e6011f7fabd8c666d6c7a4349fe21630: Status 404 returned error can't find the container with id ce4f146644a650cf6d64d0d656f90824e6011f7fabd8c666d6c7a4349fe21630 Oct 30 16:47:35.678477 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:35.678443 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:47:35.678575 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:35.678514 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:47:35.678575 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:35.678543 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:47:36.610833 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:36.610799 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ca5971-openshift-default-f54d56b47-qrn9w" event={"ID":"cbb6daa4-cde3-4e3f-8343-1b8ca62f8390","Type":"ContainerStarted","Data":"aed4f211ad9931452ee7d8b380b6e6031a24664f56d8843560dad937ceeb9356"} Oct 30 16:47:36.611236 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:36.610849 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ca5971-openshift-default-f54d56b47-qrn9w" event={"ID":"cbb6daa4-cde3-4e3f-8343-1b8ca62f8390","Type":"ContainerStarted","Data":"ce4f146644a650cf6d64d0d656f90824e6011f7fabd8c666d6c7a4349fe21630"} Oct 30 16:47:36.683358 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:36.683288 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-ca5971-openshift-default-f54d56b47-qrn9w" podStartSLOduration=2.683267708 podStartE2EDuration="2.683267708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:47:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:47:36.681044013 +0000 UTC m=+1612.208379232" watchObservedRunningTime="2026-10-30 16:47:36.683267708 +0000 UTC m=+1612.210602928" Oct 30 16:47:37.051324 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:37.051292 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-ca5971-openshift-default-f54d56b47-qrn9w" Oct 30 16:47:37.057446 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:37.057420 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-ca5971-openshift-default-f54d56b47-qrn9w" Oct 30 16:47:37.619050 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:37.619018 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-ca5971-openshift-default-f54d56b47-qrn9w" Oct 30 16:47:37.620082 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:37.620059 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-ca5971-openshift-default-f54d56b47-qrn9w" Oct 30 16:47:39.609935 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:39.609872 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-b8af660d-787d8764b-h9rfs" podUID="2c983762-43df-4410-9001-d5d25c7a45f3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 16:47:39.631272 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:39.620117 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e5674c-5969bf95bc-jrsfj" Oct 30 16:47:43.619503 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:43.619475 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-b8af660d-787d8764b-h9rfs" Oct 30 16:47:43.740883 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:43.740801 5052 generic.go:356] "Generic (PLEG): container finished" podID="2c983762-43df-4410-9001-d5d25c7a45f3" containerID="1cea34d2c5be1ab53a7ea1aff002b811b72bf2029f09335db701ce6f718c1587" exitCode=0 Oct 30 16:47:43.740883 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:43.740860 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-b8af660d-787d8764b-h9rfs" Oct 30 16:47:43.741099 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:43.740886 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-b8af660d-787d8764b-h9rfs" event={"ID":"2c983762-43df-4410-9001-d5d25c7a45f3","Type":"ContainerDied","Data":"1cea34d2c5be1ab53a7ea1aff002b811b72bf2029f09335db701ce6f718c1587"} Oct 30 16:47:43.741099 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:43.740951 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-b8af660d-787d8764b-h9rfs" event={"ID":"2c983762-43df-4410-9001-d5d25c7a45f3","Type":"ContainerDied","Data":"4bcdcea078d035126fbeab46ed7b1f13d3e66e176376a2d4bf6de315664ecb9f"} Oct 30 16:47:43.741099 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:43.740982 5052 scope.go:122] "RemoveContainer" containerID="1cea34d2c5be1ab53a7ea1aff002b811b72bf2029f09335db701ce6f718c1587" Oct 30 16:47:43.748901 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:43.748860 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c983762-43df-4410-9001-d5d25c7a45f3-kube-api-access-hvvb8\" (UniqueName: \"kubernetes.io/projected/2c983762-43df-4410-9001-d5d25c7a45f3-kube-api-access-hvvb8\") pod \"2c983762-43df-4410-9001-d5d25c7a45f3\" (UID: \"2c983762-43df-4410-9001-d5d25c7a45f3\") " Oct 30 16:47:43.749112 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:43.749093 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c983762-43df-4410-9001-d5d25c7a45f3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c983762-43df-4410-9001-d5d25c7a45f3-plugins-config-volume\") pod \"2c983762-43df-4410-9001-d5d25c7a45f3\" (UID: \"2c983762-43df-4410-9001-d5d25c7a45f3\") " Oct 30 16:47:43.749494 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:43.749468 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c983762-43df-4410-9001-d5d25c7a45f3-plugins-config-volume" pod "2c983762-43df-4410-9001-d5d25c7a45f3" (UID: "2c983762-43df-4410-9001-d5d25c7a45f3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:47:43.750012 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:43.749988 5052 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c983762-43df-4410-9001-d5d25c7a45f3-plugins-config-volume\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:47:43.751092 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:43.751073 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c983762-43df-4410-9001-d5d25c7a45f3-kube-api-access-hvvb8" pod "2c983762-43df-4410-9001-d5d25c7a45f3" (UID: "2c983762-43df-4410-9001-d5d25c7a45f3"). InnerVolumeSpecName "kube-api-access-hvvb8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:47:43.754669 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:43.754648 5052 scope.go:122] "RemoveContainer" containerID="1cea34d2c5be1ab53a7ea1aff002b811b72bf2029f09335db701ce6f718c1587" Oct 30 16:47:43.754963 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:47:43.754938 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cea34d2c5be1ab53a7ea1aff002b811b72bf2029f09335db701ce6f718c1587\": container with ID starting with 1cea34d2c5be1ab53a7ea1aff002b811b72bf2029f09335db701ce6f718c1587 not found: ID does not exist" containerID="1cea34d2c5be1ab53a7ea1aff002b811b72bf2029f09335db701ce6f718c1587" Oct 30 16:47:43.755024 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:43.754962 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cea34d2c5be1ab53a7ea1aff002b811b72bf2029f09335db701ce6f718c1587"} err="failed to get container status \"1cea34d2c5be1ab53a7ea1aff002b811b72bf2029f09335db701ce6f718c1587\": rpc error: code = NotFound desc = could not find container \"1cea34d2c5be1ab53a7ea1aff002b811b72bf2029f09335db701ce6f718c1587\": container with ID starting with 1cea34d2c5be1ab53a7ea1aff002b811b72bf2029f09335db701ce6f718c1587 not found: ID does not exist" Oct 30 16:47:43.851259 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:43.851225 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvvb8\" (UniqueName: \"kubernetes.io/projected/2c983762-43df-4410-9001-d5d25c7a45f3-kube-api-access-hvvb8\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:47:44.091530 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:44.091499 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b8af660d-787d8764b-h9rfs"] Oct 30 16:47:44.099148 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:44.099122 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b8af660d-787d8764b-h9rfs"] Oct 30 16:47:44.606333 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:44.606301 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c983762-43df-4410-9001-d5d25c7a45f3" path="/var/lib/kubelet/pods/2c983762-43df-4410-9001-d5d25c7a45f3/volumes" Oct 30 16:47:45.090886 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:45.090848 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cdwcj"] Oct 30 16:47:45.588059 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:45.588022 5052 scope.go:122] "RemoveContainer" containerID="e84bdf7f621343f796307bed370d4da876f8582157df6f2132fe164ade1eff4a" Oct 30 16:47:49.200843 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:47:49.200795 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 16:47:59.771755 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:59.771094 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-407eab-openshift-default-85f774ff9c-nldmk"] Oct 30 16:47:59.772339 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:59.772318 5052 cpu_manager.go:414] "containerMap: removing container" podUID="2c983762-43df-4410-9001-d5d25c7a45f3" containerName="payload-processing" Oct 30 16:47:59.772420 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:59.772341 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c983762-43df-4410-9001-d5d25c7a45f3" containerName="payload-processing" Oct 30 16:47:59.772622 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:59.772601 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c983762-43df-4410-9001-d5d25c7a45f3" containerName="payload-processing" Oct 30 16:47:59.777106 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:59.777085 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-407eab-openshift-default-85f774ff9c-nldmk" Oct 30 16:47:59.799228 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:59.799202 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-407eab-openshift-default-85f774ff9c-nldmk"] Oct 30 16:47:59.806155 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:59.806128 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-407eab-openshift-default-dockercfg-2l6h6\"" Oct 30 16:47:59.946609 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:59.940551 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/218cbb21-a450-4352-b4b3-f7e8558688bb-istio-token\") pod \"e2e-api-b-407eab-openshift-default-85f774ff9c-nldmk\" (UID: \"218cbb21-a450-4352-b4b3-f7e8558688bb\") " pod="openshift-ingress/e2e-api-b-407eab-openshift-default-85f774ff9c-nldmk" Oct 30 16:47:59.954609 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:59.948143 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nztq\" (UniqueName: \"kubernetes.io/projected/218cbb21-a450-4352-b4b3-f7e8558688bb-kube-api-access-2nztq\") pod \"e2e-api-b-407eab-openshift-default-85f774ff9c-nldmk\" (UID: \"218cbb21-a450-4352-b4b3-f7e8558688bb\") " pod="openshift-ingress/e2e-api-b-407eab-openshift-default-85f774ff9c-nldmk" Oct 30 16:47:59.968600 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:59.962778 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/218cbb21-a450-4352-b4b3-f7e8558688bb-istiod-ca-cert\") pod \"e2e-api-b-407eab-openshift-default-85f774ff9c-nldmk\" (UID: \"218cbb21-a450-4352-b4b3-f7e8558688bb\") " pod="openshift-ingress/e2e-api-b-407eab-openshift-default-85f774ff9c-nldmk" Oct 30 16:47:59.968600 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:59.962847 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/218cbb21-a450-4352-b4b3-f7e8558688bb-istio-podinfo\") pod \"e2e-api-b-407eab-openshift-default-85f774ff9c-nldmk\" (UID: \"218cbb21-a450-4352-b4b3-f7e8558688bb\") " pod="openshift-ingress/e2e-api-b-407eab-openshift-default-85f774ff9c-nldmk" Oct 30 16:47:59.968600 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:59.963097 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/218cbb21-a450-4352-b4b3-f7e8558688bb-workload-certs\") pod \"e2e-api-b-407eab-openshift-default-85f774ff9c-nldmk\" (UID: \"218cbb21-a450-4352-b4b3-f7e8558688bb\") " pod="openshift-ingress/e2e-api-b-407eab-openshift-default-85f774ff9c-nldmk" Oct 30 16:47:59.968600 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:59.963309 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/218cbb21-a450-4352-b4b3-f7e8558688bb-credential-socket\") pod \"e2e-api-b-407eab-openshift-default-85f774ff9c-nldmk\" (UID: \"218cbb21-a450-4352-b4b3-f7e8558688bb\") " pod="openshift-ingress/e2e-api-b-407eab-openshift-default-85f774ff9c-nldmk" Oct 30 16:47:59.968600 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:59.963351 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/218cbb21-a450-4352-b4b3-f7e8558688bb-istio-envoy\") pod \"e2e-api-b-407eab-openshift-default-85f774ff9c-nldmk\" (UID: \"218cbb21-a450-4352-b4b3-f7e8558688bb\") " pod="openshift-ingress/e2e-api-b-407eab-openshift-default-85f774ff9c-nldmk" Oct 30 16:47:59.968600 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:59.963382 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/218cbb21-a450-4352-b4b3-f7e8558688bb-istio-data\") pod \"e2e-api-b-407eab-openshift-default-85f774ff9c-nldmk\" (UID: \"218cbb21-a450-4352-b4b3-f7e8558688bb\") " pod="openshift-ingress/e2e-api-b-407eab-openshift-default-85f774ff9c-nldmk" Oct 30 16:47:59.968600 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:47:59.963470 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/218cbb21-a450-4352-b4b3-f7e8558688bb-workload-socket\") pod \"e2e-api-b-407eab-openshift-default-85f774ff9c-nldmk\" (UID: \"218cbb21-a450-4352-b4b3-f7e8558688bb\") " pod="openshift-ingress/e2e-api-b-407eab-openshift-default-85f774ff9c-nldmk" Oct 30 16:48:00.095650 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:00.084421 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/218cbb21-a450-4352-b4b3-f7e8558688bb-credential-socket\") pod \"e2e-api-b-407eab-openshift-default-85f774ff9c-nldmk\" (UID: \"218cbb21-a450-4352-b4b3-f7e8558688bb\") " pod="openshift-ingress/e2e-api-b-407eab-openshift-default-85f774ff9c-nldmk" Oct 30 16:48:00.095650 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:00.084502 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/218cbb21-a450-4352-b4b3-f7e8558688bb-istio-envoy\") pod \"e2e-api-b-407eab-openshift-default-85f774ff9c-nldmk\" (UID: \"218cbb21-a450-4352-b4b3-f7e8558688bb\") " pod="openshift-ingress/e2e-api-b-407eab-openshift-default-85f774ff9c-nldmk" Oct 30 16:48:00.095650 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:00.084547 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/218cbb21-a450-4352-b4b3-f7e8558688bb-istio-data\") pod \"e2e-api-b-407eab-openshift-default-85f774ff9c-nldmk\" (UID: \"218cbb21-a450-4352-b4b3-f7e8558688bb\") " pod="openshift-ingress/e2e-api-b-407eab-openshift-default-85f774ff9c-nldmk" Oct 30 16:48:00.095650 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:00.084652 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/218cbb21-a450-4352-b4b3-f7e8558688bb-workload-socket\") pod \"e2e-api-b-407eab-openshift-default-85f774ff9c-nldmk\" (UID: \"218cbb21-a450-4352-b4b3-f7e8558688bb\") " pod="openshift-ingress/e2e-api-b-407eab-openshift-default-85f774ff9c-nldmk" Oct 30 16:48:00.095650 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:00.084794 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/218cbb21-a450-4352-b4b3-f7e8558688bb-istio-token\") pod \"e2e-api-b-407eab-openshift-default-85f774ff9c-nldmk\" (UID: \"218cbb21-a450-4352-b4b3-f7e8558688bb\") " pod="openshift-ingress/e2e-api-b-407eab-openshift-default-85f774ff9c-nldmk" Oct 30 16:48:00.095650 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:00.084843 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nztq\" (UniqueName: \"kubernetes.io/projected/218cbb21-a450-4352-b4b3-f7e8558688bb-kube-api-access-2nztq\") pod \"e2e-api-b-407eab-openshift-default-85f774ff9c-nldmk\" (UID: \"218cbb21-a450-4352-b4b3-f7e8558688bb\") " pod="openshift-ingress/e2e-api-b-407eab-openshift-default-85f774ff9c-nldmk" Oct 30 16:48:00.095650 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:00.084913 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/218cbb21-a450-4352-b4b3-f7e8558688bb-istiod-ca-cert\") pod \"e2e-api-b-407eab-openshift-default-85f774ff9c-nldmk\" (UID: \"218cbb21-a450-4352-b4b3-f7e8558688bb\") " pod="openshift-ingress/e2e-api-b-407eab-openshift-default-85f774ff9c-nldmk" Oct 30 16:48:00.095650 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:00.084959 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/218cbb21-a450-4352-b4b3-f7e8558688bb-istio-podinfo\") pod \"e2e-api-b-407eab-openshift-default-85f774ff9c-nldmk\" (UID: \"218cbb21-a450-4352-b4b3-f7e8558688bb\") " pod="openshift-ingress/e2e-api-b-407eab-openshift-default-85f774ff9c-nldmk" Oct 30 16:48:00.095650 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:00.086045 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/218cbb21-a450-4352-b4b3-f7e8558688bb-workload-socket\") pod \"e2e-api-b-407eab-openshift-default-85f774ff9c-nldmk\" (UID: \"218cbb21-a450-4352-b4b3-f7e8558688bb\") " pod="openshift-ingress/e2e-api-b-407eab-openshift-default-85f774ff9c-nldmk" Oct 30 16:48:00.095650 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:00.090022 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/218cbb21-a450-4352-b4b3-f7e8558688bb-workload-certs\") pod \"e2e-api-b-407eab-openshift-default-85f774ff9c-nldmk\" (UID: \"218cbb21-a450-4352-b4b3-f7e8558688bb\") " pod="openshift-ingress/e2e-api-b-407eab-openshift-default-85f774ff9c-nldmk" Oct 30 16:48:00.095650 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:00.090635 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/218cbb21-a450-4352-b4b3-f7e8558688bb-workload-certs\") pod \"e2e-api-b-407eab-openshift-default-85f774ff9c-nldmk\" (UID: \"218cbb21-a450-4352-b4b3-f7e8558688bb\") " pod="openshift-ingress/e2e-api-b-407eab-openshift-default-85f774ff9c-nldmk" Oct 30 16:48:00.095650 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:00.090898 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/218cbb21-a450-4352-b4b3-f7e8558688bb-credential-socket\") pod \"e2e-api-b-407eab-openshift-default-85f774ff9c-nldmk\" (UID: \"218cbb21-a450-4352-b4b3-f7e8558688bb\") " pod="openshift-ingress/e2e-api-b-407eab-openshift-default-85f774ff9c-nldmk" Oct 30 16:48:00.095650 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:00.092110 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/218cbb21-a450-4352-b4b3-f7e8558688bb-istio-data\") pod \"e2e-api-b-407eab-openshift-default-85f774ff9c-nldmk\" (UID: \"218cbb21-a450-4352-b4b3-f7e8558688bb\") " pod="openshift-ingress/e2e-api-b-407eab-openshift-default-85f774ff9c-nldmk" Oct 30 16:48:00.095650 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:00.092869 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/218cbb21-a450-4352-b4b3-f7e8558688bb-istiod-ca-cert\") pod \"e2e-api-b-407eab-openshift-default-85f774ff9c-nldmk\" (UID: \"218cbb21-a450-4352-b4b3-f7e8558688bb\") " pod="openshift-ingress/e2e-api-b-407eab-openshift-default-85f774ff9c-nldmk" Oct 30 16:48:00.118567 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:00.117050 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/218cbb21-a450-4352-b4b3-f7e8558688bb-istio-podinfo\") pod \"e2e-api-b-407eab-openshift-default-85f774ff9c-nldmk\" (UID: \"218cbb21-a450-4352-b4b3-f7e8558688bb\") " pod="openshift-ingress/e2e-api-b-407eab-openshift-default-85f774ff9c-nldmk" Oct 30 16:48:00.151606 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:00.150098 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/218cbb21-a450-4352-b4b3-f7e8558688bb-istio-envoy\") pod \"e2e-api-b-407eab-openshift-default-85f774ff9c-nldmk\" (UID: \"218cbb21-a450-4352-b4b3-f7e8558688bb\") " pod="openshift-ingress/e2e-api-b-407eab-openshift-default-85f774ff9c-nldmk" Oct 30 16:48:00.151606 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:00.150789 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nztq\" (UniqueName: \"kubernetes.io/projected/218cbb21-a450-4352-b4b3-f7e8558688bb-kube-api-access-2nztq\") pod \"e2e-api-b-407eab-openshift-default-85f774ff9c-nldmk\" (UID: \"218cbb21-a450-4352-b4b3-f7e8558688bb\") " pod="openshift-ingress/e2e-api-b-407eab-openshift-default-85f774ff9c-nldmk" Oct 30 16:48:00.159428 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:00.157224 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/218cbb21-a450-4352-b4b3-f7e8558688bb-istio-token\") pod \"e2e-api-b-407eab-openshift-default-85f774ff9c-nldmk\" (UID: \"218cbb21-a450-4352-b4b3-f7e8558688bb\") " pod="openshift-ingress/e2e-api-b-407eab-openshift-default-85f774ff9c-nldmk" Oct 30 16:48:00.411875 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:00.411518 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-407eab-openshift-default-85f774ff9c-nldmk" Oct 30 16:48:00.764102 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:48:00.764068 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod218cbb21_a450_4352_b4b3_f7e8558688bb.slice/crio-c222361e8b16ea70109b930ef59793198858cbd634c16172dbc547c11ff32bcb WatchSource:0}: Error finding container c222361e8b16ea70109b930ef59793198858cbd634c16172dbc547c11ff32bcb: Status 404 returned error can't find the container with id c222361e8b16ea70109b930ef59793198858cbd634c16172dbc547c11ff32bcb Oct 30 16:48:00.764274 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:00.764251 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-407eab-openshift-default-85f774ff9c-nldmk"] Oct 30 16:48:00.767113 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:00.767085 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:48:00.767180 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:00.767147 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:48:00.767180 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:00.767176 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:48:01.283285 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:01.283253 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-407eab-openshift-default-85f774ff9c-nldmk" event={"ID":"218cbb21-a450-4352-b4b3-f7e8558688bb","Type":"ContainerStarted","Data":"5533cf1b92cd792dfcb6a1953f72d0abedaee4cb23f48905192a29a1642e75f3"} Oct 30 16:48:01.283725 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:01.283314 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-407eab-openshift-default-85f774ff9c-nldmk" event={"ID":"218cbb21-a450-4352-b4b3-f7e8558688bb","Type":"ContainerStarted","Data":"c222361e8b16ea70109b930ef59793198858cbd634c16172dbc547c11ff32bcb"} Oct 30 16:48:01.337182 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:01.337136 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-407eab-openshift-default-85f774ff9c-nldmk" podStartSLOduration=2.337121496 podStartE2EDuration="2.337121496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:47:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:48:01.333718628 +0000 UTC m=+1636.861053871" watchObservedRunningTime="2026-10-30 16:48:01.337121496 +0000 UTC m=+1636.864456715" Oct 30 16:48:01.412675 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:01.412641 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-407eab-openshift-default-85f774ff9c-nldmk" Oct 30 16:48:01.419251 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:01.419230 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-407eab-openshift-default-85f774ff9c-nldmk" Oct 30 16:48:02.290896 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:02.290864 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-407eab-openshift-default-85f774ff9c-nldmk" Oct 30 16:48:02.291829 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:02.291809 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-407eab-openshift-default-85f774ff9c-nldmk" Oct 30 16:48:05.336023 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:05.335993 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-bg8br"] Oct 30 16:48:05.336413 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:05.336204 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-bg8br" podUID="597c97ef-00c9-46b1-afbb-8b05363df559" containerName="istio-proxy" containerID="cri-o://3dcd2c17c569e4716e96bf386a1abf72d5329514b6ae4a6a3fde68a0fbce0037" gracePeriod=30 Oct 30 16:48:05.351731 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:05.351699 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hkcvj"] Oct 30 16:48:05.351949 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:05.351924 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hkcvj" podUID="545986bf-3dc2-4204-9fd1-ac7085df215b" containerName="istio-proxy" containerID="cri-o://533bd54ab34a531074d3353f760a16a82f42ac44d0c44ba8d18b40240a57427f" gracePeriod=30 Oct 30 16:48:10.650679 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:10.650653 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hkcvj" Oct 30 16:48:10.759320 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:10.759283 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/545986bf-3dc2-4204-9fd1-ac7085df215b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/545986bf-3dc2-4204-9fd1-ac7085df215b-workload-socket\") pod \"545986bf-3dc2-4204-9fd1-ac7085df215b\" (UID: \"545986bf-3dc2-4204-9fd1-ac7085df215b\") " Oct 30 16:48:10.759514 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:10.759391 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/545986bf-3dc2-4204-9fd1-ac7085df215b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/545986bf-3dc2-4204-9fd1-ac7085df215b-istio-envoy\") pod \"545986bf-3dc2-4204-9fd1-ac7085df215b\" (UID: \"545986bf-3dc2-4204-9fd1-ac7085df215b\") " Oct 30 16:48:10.759732 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:10.759704 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/545986bf-3dc2-4204-9fd1-ac7085df215b-workload-socket" pod "545986bf-3dc2-4204-9fd1-ac7085df215b" (UID: "545986bf-3dc2-4204-9fd1-ac7085df215b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:48:10.759910 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:10.759891 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/545986bf-3dc2-4204-9fd1-ac7085df215b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/545986bf-3dc2-4204-9fd1-ac7085df215b-istio-podinfo\") pod \"545986bf-3dc2-4204-9fd1-ac7085df215b\" (UID: \"545986bf-3dc2-4204-9fd1-ac7085df215b\") " Oct 30 16:48:10.760394 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:10.760350 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/545986bf-3dc2-4204-9fd1-ac7085df215b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/545986bf-3dc2-4204-9fd1-ac7085df215b-credential-socket\") pod \"545986bf-3dc2-4204-9fd1-ac7085df215b\" (UID: \"545986bf-3dc2-4204-9fd1-ac7085df215b\") " Oct 30 16:48:10.760633 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:10.760607 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/545986bf-3dc2-4204-9fd1-ac7085df215b-credential-socket" pod "545986bf-3dc2-4204-9fd1-ac7085df215b" (UID: "545986bf-3dc2-4204-9fd1-ac7085df215b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:48:10.760758 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:10.760669 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/545986bf-3dc2-4204-9fd1-ac7085df215b-istio-token\" (UniqueName: \"kubernetes.io/projected/545986bf-3dc2-4204-9fd1-ac7085df215b-istio-token\") pod \"545986bf-3dc2-4204-9fd1-ac7085df215b\" (UID: \"545986bf-3dc2-4204-9fd1-ac7085df215b\") " Oct 30 16:48:10.761271 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:10.761250 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/545986bf-3dc2-4204-9fd1-ac7085df215b-kube-api-access-rm22w\" (UniqueName: \"kubernetes.io/projected/545986bf-3dc2-4204-9fd1-ac7085df215b-kube-api-access-rm22w\") pod \"545986bf-3dc2-4204-9fd1-ac7085df215b\" (UID: \"545986bf-3dc2-4204-9fd1-ac7085df215b\") " Oct 30 16:48:10.761555 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:10.761536 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/545986bf-3dc2-4204-9fd1-ac7085df215b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/545986bf-3dc2-4204-9fd1-ac7085df215b-istio-data\") pod \"545986bf-3dc2-4204-9fd1-ac7085df215b\" (UID: \"545986bf-3dc2-4204-9fd1-ac7085df215b\") " Oct 30 16:48:10.761837 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:10.761803 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/545986bf-3dc2-4204-9fd1-ac7085df215b-istio-data" pod "545986bf-3dc2-4204-9fd1-ac7085df215b" (UID: "545986bf-3dc2-4204-9fd1-ac7085df215b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:48:10.761964 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:10.761950 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/545986bf-3dc2-4204-9fd1-ac7085df215b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/545986bf-3dc2-4204-9fd1-ac7085df215b-istiod-ca-cert\") pod \"545986bf-3dc2-4204-9fd1-ac7085df215b\" (UID: \"545986bf-3dc2-4204-9fd1-ac7085df215b\") " Oct 30 16:48:10.762859 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:10.762841 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/545986bf-3dc2-4204-9fd1-ac7085df215b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/545986bf-3dc2-4204-9fd1-ac7085df215b-workload-certs\") pod \"545986bf-3dc2-4204-9fd1-ac7085df215b\" (UID: \"545986bf-3dc2-4204-9fd1-ac7085df215b\") " Oct 30 16:48:10.764144 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:10.764125 5052 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/545986bf-3dc2-4204-9fd1-ac7085df215b-istio-data\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:48:10.765466 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:10.765450 5052 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/545986bf-3dc2-4204-9fd1-ac7085df215b-workload-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:48:10.765605 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:10.765591 5052 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/545986bf-3dc2-4204-9fd1-ac7085df215b-credential-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:48:10.765694 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:10.762322 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/545986bf-3dc2-4204-9fd1-ac7085df215b-istiod-ca-cert" pod "545986bf-3dc2-4204-9fd1-ac7085df215b" (UID: "545986bf-3dc2-4204-9fd1-ac7085df215b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:48:10.765778 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:10.763200 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/545986bf-3dc2-4204-9fd1-ac7085df215b-workload-certs" pod "545986bf-3dc2-4204-9fd1-ac7085df215b" (UID: "545986bf-3dc2-4204-9fd1-ac7085df215b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:48:10.771749 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:10.771724 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/545986bf-3dc2-4204-9fd1-ac7085df215b-istio-envoy" pod "545986bf-3dc2-4204-9fd1-ac7085df215b" (UID: "545986bf-3dc2-4204-9fd1-ac7085df215b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:48:10.771847 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:10.771743 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/545986bf-3dc2-4204-9fd1-ac7085df215b-istio-podinfo" pod "545986bf-3dc2-4204-9fd1-ac7085df215b" (UID: "545986bf-3dc2-4204-9fd1-ac7085df215b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:48:10.771920 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:10.771900 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/545986bf-3dc2-4204-9fd1-ac7085df215b-kube-api-access-rm22w" pod "545986bf-3dc2-4204-9fd1-ac7085df215b" (UID: "545986bf-3dc2-4204-9fd1-ac7085df215b"). InnerVolumeSpecName "kube-api-access-rm22w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:48:10.777629 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:10.774489 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/545986bf-3dc2-4204-9fd1-ac7085df215b-istio-token" pod "545986bf-3dc2-4204-9fd1-ac7085df215b" (UID: "545986bf-3dc2-4204-9fd1-ac7085df215b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:48:10.868166 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:10.868085 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rm22w\" (UniqueName: \"kubernetes.io/projected/545986bf-3dc2-4204-9fd1-ac7085df215b-kube-api-access-rm22w\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:48:10.868166 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:10.868123 5052 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/545986bf-3dc2-4204-9fd1-ac7085df215b-istiod-ca-cert\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:48:10.868166 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:10.868138 5052 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/545986bf-3dc2-4204-9fd1-ac7085df215b-workload-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:48:10.868166 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:10.868150 5052 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/545986bf-3dc2-4204-9fd1-ac7085df215b-istio-envoy\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:48:10.868166 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:10.868163 5052 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/545986bf-3dc2-4204-9fd1-ac7085df215b-istio-podinfo\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:48:10.868764 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:10.868174 5052 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/545986bf-3dc2-4204-9fd1-ac7085df215b-istio-token\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:48:11.538841 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:11.538755 5052 generic.go:356] "Generic (PLEG): container finished" podID="545986bf-3dc2-4204-9fd1-ac7085df215b" containerID="533bd54ab34a531074d3353f760a16a82f42ac44d0c44ba8d18b40240a57427f" exitCode=0 Oct 30 16:48:11.539006 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:11.538879 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hkcvj" event={"ID":"545986bf-3dc2-4204-9fd1-ac7085df215b","Type":"ContainerDied","Data":"533bd54ab34a531074d3353f760a16a82f42ac44d0c44ba8d18b40240a57427f"} Oct 30 16:48:11.539006 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:11.538932 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hkcvj" event={"ID":"545986bf-3dc2-4204-9fd1-ac7085df215b","Type":"ContainerDied","Data":"aac02cdc261f7cffdd80ff723363b3f46bd8f5dc468b97a2fe4efa224955970f"} Oct 30 16:48:11.539006 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:11.538964 5052 scope.go:122] "RemoveContainer" containerID="533bd54ab34a531074d3353f760a16a82f42ac44d0c44ba8d18b40240a57427f" Oct 30 16:48:11.539216 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:11.539195 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hkcvj" Oct 30 16:48:11.588612 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:11.588273 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-bg8br" Oct 30 16:48:11.613614 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:11.608063 5052 generic.go:356] "Generic (PLEG): container finished" podID="597c97ef-00c9-46b1-afbb-8b05363df559" containerID="3dcd2c17c569e4716e96bf386a1abf72d5329514b6ae4a6a3fde68a0fbce0037" exitCode=0 Oct 30 16:48:11.613614 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:11.608356 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-bg8br" event={"ID":"597c97ef-00c9-46b1-afbb-8b05363df559","Type":"ContainerDied","Data":"3dcd2c17c569e4716e96bf386a1abf72d5329514b6ae4a6a3fde68a0fbce0037"} Oct 30 16:48:11.675674 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:11.674767 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/597c97ef-00c9-46b1-afbb-8b05363df559-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/597c97ef-00c9-46b1-afbb-8b05363df559-credential-socket\") pod \"597c97ef-00c9-46b1-afbb-8b05363df559\" (UID: \"597c97ef-00c9-46b1-afbb-8b05363df559\") " Oct 30 16:48:11.675674 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:11.674850 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/597c97ef-00c9-46b1-afbb-8b05363df559-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/597c97ef-00c9-46b1-afbb-8b05363df559-workload-certs\") pod \"597c97ef-00c9-46b1-afbb-8b05363df559\" (UID: \"597c97ef-00c9-46b1-afbb-8b05363df559\") " Oct 30 16:48:11.675674 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:11.675008 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/597c97ef-00c9-46b1-afbb-8b05363df559-kube-api-access-zqhz2\" (UniqueName: \"kubernetes.io/projected/597c97ef-00c9-46b1-afbb-8b05363df559-kube-api-access-zqhz2\") pod \"597c97ef-00c9-46b1-afbb-8b05363df559\" (UID: \"597c97ef-00c9-46b1-afbb-8b05363df559\") " Oct 30 16:48:11.675674 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:11.675102 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/597c97ef-00c9-46b1-afbb-8b05363df559-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/597c97ef-00c9-46b1-afbb-8b05363df559-workload-socket\") pod \"597c97ef-00c9-46b1-afbb-8b05363df559\" (UID: \"597c97ef-00c9-46b1-afbb-8b05363df559\") " Oct 30 16:48:11.675674 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:11.675174 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/597c97ef-00c9-46b1-afbb-8b05363df559-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/597c97ef-00c9-46b1-afbb-8b05363df559-istio-data\") pod \"597c97ef-00c9-46b1-afbb-8b05363df559\" (UID: \"597c97ef-00c9-46b1-afbb-8b05363df559\") " Oct 30 16:48:11.675674 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:11.675301 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/597c97ef-00c9-46b1-afbb-8b05363df559-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/597c97ef-00c9-46b1-afbb-8b05363df559-istio-envoy\") pod \"597c97ef-00c9-46b1-afbb-8b05363df559\" (UID: \"597c97ef-00c9-46b1-afbb-8b05363df559\") " Oct 30 16:48:11.675674 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:11.675327 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/597c97ef-00c9-46b1-afbb-8b05363df559-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/597c97ef-00c9-46b1-afbb-8b05363df559-istiod-ca-cert\") pod \"597c97ef-00c9-46b1-afbb-8b05363df559\" (UID: \"597c97ef-00c9-46b1-afbb-8b05363df559\") " Oct 30 16:48:11.675674 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:11.675354 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/597c97ef-00c9-46b1-afbb-8b05363df559-istio-token\" (UniqueName: \"kubernetes.io/projected/597c97ef-00c9-46b1-afbb-8b05363df559-istio-token\") pod \"597c97ef-00c9-46b1-afbb-8b05363df559\" (UID: \"597c97ef-00c9-46b1-afbb-8b05363df559\") " Oct 30 16:48:11.675674 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:11.675443 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/597c97ef-00c9-46b1-afbb-8b05363df559-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/597c97ef-00c9-46b1-afbb-8b05363df559-istio-podinfo\") pod \"597c97ef-00c9-46b1-afbb-8b05363df559\" (UID: \"597c97ef-00c9-46b1-afbb-8b05363df559\") " Oct 30 16:48:11.675674 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:11.675487 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/597c97ef-00c9-46b1-afbb-8b05363df559-workload-certs" pod "597c97ef-00c9-46b1-afbb-8b05363df559" (UID: "597c97ef-00c9-46b1-afbb-8b05363df559"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:48:11.676574 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:11.676552 5052 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/597c97ef-00c9-46b1-afbb-8b05363df559-workload-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:48:11.679394 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:11.679368 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/597c97ef-00c9-46b1-afbb-8b05363df559-credential-socket" pod "597c97ef-00c9-46b1-afbb-8b05363df559" (UID: "597c97ef-00c9-46b1-afbb-8b05363df559"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:48:11.679830 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:11.679806 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/597c97ef-00c9-46b1-afbb-8b05363df559-kube-api-access-zqhz2" pod "597c97ef-00c9-46b1-afbb-8b05363df559" (UID: "597c97ef-00c9-46b1-afbb-8b05363df559"). InnerVolumeSpecName "kube-api-access-zqhz2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:48:11.680134 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:11.680106 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/597c97ef-00c9-46b1-afbb-8b05363df559-istiod-ca-cert" pod "597c97ef-00c9-46b1-afbb-8b05363df559" (UID: "597c97ef-00c9-46b1-afbb-8b05363df559"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:48:11.680994 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:11.680970 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/597c97ef-00c9-46b1-afbb-8b05363df559-workload-socket" pod "597c97ef-00c9-46b1-afbb-8b05363df559" (UID: "597c97ef-00c9-46b1-afbb-8b05363df559"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:48:11.681377 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:11.681357 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/597c97ef-00c9-46b1-afbb-8b05363df559-istio-data" pod "597c97ef-00c9-46b1-afbb-8b05363df559" (UID: "597c97ef-00c9-46b1-afbb-8b05363df559"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:48:11.682782 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:11.682754 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/597c97ef-00c9-46b1-afbb-8b05363df559-istio-token" pod "597c97ef-00c9-46b1-afbb-8b05363df559" (UID: "597c97ef-00c9-46b1-afbb-8b05363df559"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:48:11.688074 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:11.688038 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/597c97ef-00c9-46b1-afbb-8b05363df559-istio-podinfo" pod "597c97ef-00c9-46b1-afbb-8b05363df559" (UID: "597c97ef-00c9-46b1-afbb-8b05363df559"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:48:11.694782 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:11.694757 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/597c97ef-00c9-46b1-afbb-8b05363df559-istio-envoy" pod "597c97ef-00c9-46b1-afbb-8b05363df559" (UID: "597c97ef-00c9-46b1-afbb-8b05363df559"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:48:11.729814 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:11.729786 5052 scope.go:122] "RemoveContainer" containerID="533bd54ab34a531074d3353f760a16a82f42ac44d0c44ba8d18b40240a57427f" Oct 30 16:48:11.730154 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:48:11.730132 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"533bd54ab34a531074d3353f760a16a82f42ac44d0c44ba8d18b40240a57427f\": container with ID starting with 533bd54ab34a531074d3353f760a16a82f42ac44d0c44ba8d18b40240a57427f not found: ID does not exist" containerID="533bd54ab34a531074d3353f760a16a82f42ac44d0c44ba8d18b40240a57427f" Oct 30 16:48:11.730222 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:11.730163 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"533bd54ab34a531074d3353f760a16a82f42ac44d0c44ba8d18b40240a57427f"} err="failed to get container status \"533bd54ab34a531074d3353f760a16a82f42ac44d0c44ba8d18b40240a57427f\": rpc error: code = NotFound desc = could not find container \"533bd54ab34a531074d3353f760a16a82f42ac44d0c44ba8d18b40240a57427f\": container with ID starting with 533bd54ab34a531074d3353f760a16a82f42ac44d0c44ba8d18b40240a57427f not found: ID does not exist" Oct 30 16:48:11.730222 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:11.730181 5052 scope.go:122] "RemoveContainer" containerID="3dcd2c17c569e4716e96bf386a1abf72d5329514b6ae4a6a3fde68a0fbce0037" Oct 30 16:48:11.774130 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:11.774092 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hkcvj"] Oct 30 16:48:11.778302 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:11.778276 5052 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/597c97ef-00c9-46b1-afbb-8b05363df559-istio-envoy\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:48:11.778504 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:11.778485 5052 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/597c97ef-00c9-46b1-afbb-8b05363df559-istiod-ca-cert\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:48:11.778504 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:11.778504 5052 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/597c97ef-00c9-46b1-afbb-8b05363df559-istio-token\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:48:11.778658 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:11.778512 5052 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/597c97ef-00c9-46b1-afbb-8b05363df559-istio-podinfo\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:48:11.778658 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:11.778521 5052 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/597c97ef-00c9-46b1-afbb-8b05363df559-credential-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:48:11.778658 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:11.778531 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqhz2\" (UniqueName: \"kubernetes.io/projected/597c97ef-00c9-46b1-afbb-8b05363df559-kube-api-access-zqhz2\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:48:11.778658 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:11.778540 5052 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/597c97ef-00c9-46b1-afbb-8b05363df559-workload-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:48:11.778658 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:11.778549 5052 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/597c97ef-00c9-46b1-afbb-8b05363df559-istio-data\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:48:11.781402 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:11.781381 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hkcvj"] Oct 30 16:48:12.600440 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:12.600401 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="545986bf-3dc2-4204-9fd1-ac7085df215b" path="/var/lib/kubelet/pods/545986bf-3dc2-4204-9fd1-ac7085df215b/volumes" Oct 30 16:48:12.616605 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:12.616570 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-bg8br" Oct 30 16:48:12.616719 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:12.616570 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-bg8br" event={"ID":"597c97ef-00c9-46b1-afbb-8b05363df559","Type":"ContainerDied","Data":"2d9d77889e9fcbeb3b6fc6479700c5142ed040f2d59de00cea38c3dfb968b0cc"} Oct 30 16:48:12.683864 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:12.683838 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-bg8br"] Oct 30 16:48:12.695756 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:12.695730 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-bg8br"] Oct 30 16:48:14.600158 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:14.600115 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="597c97ef-00c9-46b1-afbb-8b05363df559" path="/var/lib/kubelet/pods/597c97ef-00c9-46b1-afbb-8b05363df559/volumes" Oct 30 16:48:17.289767 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:17.289730 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ca5971-7d5f4777f7-cttvv"] Oct 30 16:48:17.290984 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:17.290961 5052 cpu_manager.go:414] "containerMap: removing container" podUID="545986bf-3dc2-4204-9fd1-ac7085df215b" containerName="istio-proxy" Oct 30 16:48:17.291044 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:17.290988 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="545986bf-3dc2-4204-9fd1-ac7085df215b" containerName="istio-proxy" Oct 30 16:48:17.291044 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:17.291006 5052 cpu_manager.go:414] "containerMap: removing container" podUID="597c97ef-00c9-46b1-afbb-8b05363df559" containerName="istio-proxy" Oct 30 16:48:17.291044 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:17.291013 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="597c97ef-00c9-46b1-afbb-8b05363df559" containerName="istio-proxy" Oct 30 16:48:17.291236 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:17.291214 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="545986bf-3dc2-4204-9fd1-ac7085df215b" containerName="istio-proxy" Oct 30 16:48:17.291236 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:17.291236 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="597c97ef-00c9-46b1-afbb-8b05363df559" containerName="istio-proxy" Oct 30 16:48:17.297359 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:17.297340 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ca5971-7d5f4777f7-cttvv" Oct 30 16:48:17.314731 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:17.314710 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-ca5971-serving-cert\"" Oct 30 16:48:17.319168 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:17.319147 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ca5971-7d5f4777f7-cttvv"] Oct 30 16:48:17.347742 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:17.347709 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/607c64a6-dc52-4862-b415-e25194a55649-metrics-certs\") pod \"maas-api-e2e-api-b-ca5971-7d5f4777f7-cttvv\" (UID: \"607c64a6-dc52-4862-b415-e25194a55649\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ca5971-7d5f4777f7-cttvv" Oct 30 16:48:17.347911 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:17.347845 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/607c64a6-dc52-4862-b415-e25194a55649-maas-api-tls\") pod \"maas-api-e2e-api-b-ca5971-7d5f4777f7-cttvv\" (UID: \"607c64a6-dc52-4862-b415-e25194a55649\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ca5971-7d5f4777f7-cttvv" Oct 30 16:48:17.347973 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:17.347945 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sz5f\" (UniqueName: \"kubernetes.io/projected/607c64a6-dc52-4862-b415-e25194a55649-kube-api-access-4sz5f\") pod \"maas-api-e2e-api-b-ca5971-7d5f4777f7-cttvv\" (UID: \"607c64a6-dc52-4862-b415-e25194a55649\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ca5971-7d5f4777f7-cttvv" Oct 30 16:48:17.348125 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:17.348099 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/607c64a6-dc52-4862-b415-e25194a55649-openshift-service-ca\") pod \"maas-api-e2e-api-b-ca5971-7d5f4777f7-cttvv\" (UID: \"607c64a6-dc52-4862-b415-e25194a55649\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ca5971-7d5f4777f7-cttvv" Oct 30 16:48:17.449851 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:17.449813 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/607c64a6-dc52-4862-b415-e25194a55649-metrics-certs\") pod \"maas-api-e2e-api-b-ca5971-7d5f4777f7-cttvv\" (UID: \"607c64a6-dc52-4862-b415-e25194a55649\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ca5971-7d5f4777f7-cttvv" Oct 30 16:48:17.450030 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:17.449928 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/607c64a6-dc52-4862-b415-e25194a55649-maas-api-tls\") pod \"maas-api-e2e-api-b-ca5971-7d5f4777f7-cttvv\" (UID: \"607c64a6-dc52-4862-b415-e25194a55649\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ca5971-7d5f4777f7-cttvv" Oct 30 16:48:17.450074 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:17.450044 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sz5f\" (UniqueName: \"kubernetes.io/projected/607c64a6-dc52-4862-b415-e25194a55649-kube-api-access-4sz5f\") pod \"maas-api-e2e-api-b-ca5971-7d5f4777f7-cttvv\" (UID: \"607c64a6-dc52-4862-b415-e25194a55649\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ca5971-7d5f4777f7-cttvv" Oct 30 16:48:17.450193 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:17.450175 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/607c64a6-dc52-4862-b415-e25194a55649-openshift-service-ca\") pod \"maas-api-e2e-api-b-ca5971-7d5f4777f7-cttvv\" (UID: \"607c64a6-dc52-4862-b415-e25194a55649\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ca5971-7d5f4777f7-cttvv" Oct 30 16:48:17.450848 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:17.450821 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/607c64a6-dc52-4862-b415-e25194a55649-openshift-service-ca\") pod \"maas-api-e2e-api-b-ca5971-7d5f4777f7-cttvv\" (UID: \"607c64a6-dc52-4862-b415-e25194a55649\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ca5971-7d5f4777f7-cttvv" Oct 30 16:48:17.452377 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:17.452354 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/607c64a6-dc52-4862-b415-e25194a55649-metrics-certs\") pod \"maas-api-e2e-api-b-ca5971-7d5f4777f7-cttvv\" (UID: \"607c64a6-dc52-4862-b415-e25194a55649\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ca5971-7d5f4777f7-cttvv" Oct 30 16:48:17.452476 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:17.452440 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/607c64a6-dc52-4862-b415-e25194a55649-maas-api-tls\") pod \"maas-api-e2e-api-b-ca5971-7d5f4777f7-cttvv\" (UID: \"607c64a6-dc52-4862-b415-e25194a55649\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ca5971-7d5f4777f7-cttvv" Oct 30 16:48:17.473527 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:17.473505 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sz5f\" (UniqueName: \"kubernetes.io/projected/607c64a6-dc52-4862-b415-e25194a55649-kube-api-access-4sz5f\") pod \"maas-api-e2e-api-b-ca5971-7d5f4777f7-cttvv\" (UID: \"607c64a6-dc52-4862-b415-e25194a55649\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ca5971-7d5f4777f7-cttvv" Oct 30 16:48:17.630044 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:17.629960 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ca5971-7d5f4777f7-cttvv" Oct 30 16:48:17.973246 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:17.973219 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ca5971-7d5f4777f7-cttvv"] Oct 30 16:48:17.973955 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:48:17.973927 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod607c64a6_dc52_4862_b415_e25194a55649.slice/crio-01c75b4dccacc4be8f8c99da8890784e2a40c782202747950abe583e3a9a3571 WatchSource:0}: Error finding container 01c75b4dccacc4be8f8c99da8890784e2a40c782202747950abe583e3a9a3571: Status 404 returned error can't find the container with id 01c75b4dccacc4be8f8c99da8890784e2a40c782202747950abe583e3a9a3571 Oct 30 16:48:18.840839 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:18.840807 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ca5971-7d5f4777f7-cttvv" event={"ID":"607c64a6-dc52-4862-b415-e25194a55649","Type":"ContainerStarted","Data":"265af70373f1c98c6c9a5de1a78df5071f059d30703664774f987636224fa085"} Oct 30 16:48:18.841201 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:18.840862 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ca5971-7d5f4777f7-cttvv" event={"ID":"607c64a6-dc52-4862-b415-e25194a55649","Type":"ContainerStarted","Data":"01c75b4dccacc4be8f8c99da8890784e2a40c782202747950abe583e3a9a3571"} Oct 30 16:48:18.841201 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:18.840919 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ca5971-7d5f4777f7-cttvv" Oct 30 16:48:18.858208 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:48:18.858168 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 16:48:18.885548 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:18.885499 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ca5971-7d5f4777f7-cttvv" podStartSLOduration=1.511722297 podStartE2EDuration="1.88548519s" totalImagesPullingTime="373.762893ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:48:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:48:18.882993125 +0000 UTC m=+1654.410328344" watchObservedRunningTime="2026-10-30 16:48:18.88548519 +0000 UTC m=+1654.412820450" Oct 30 16:48:24.853615 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:24.853561 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ca5971-7d5f4777f7-cttvv" Oct 30 16:48:31.947158 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:31.947122 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-529bc7-6bb85cdff-prc7w"] Oct 30 16:48:31.953227 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:31.953202 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-529bc7-6bb85cdff-prc7w" Oct 30 16:48:31.972879 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:31.972856 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-529bc7-6bb85cdff-prc7w"] Oct 30 16:48:31.979629 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:31.979595 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-529bc7-serving-cert\"" Oct 30 16:48:32.003764 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:32.003729 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdece98a-1500-46d3-96cb-ab50d9dd2068-metrics-certs\") pod \"maas-api-e2e-inf-a-529bc7-6bb85cdff-prc7w\" (UID: \"fdece98a-1500-46d3-96cb-ab50d9dd2068\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-529bc7-6bb85cdff-prc7w" Oct 30 16:48:32.003920 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:32.003901 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncwjk\" (UniqueName: \"kubernetes.io/projected/fdece98a-1500-46d3-96cb-ab50d9dd2068-kube-api-access-ncwjk\") pod \"maas-api-e2e-inf-a-529bc7-6bb85cdff-prc7w\" (UID: \"fdece98a-1500-46d3-96cb-ab50d9dd2068\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-529bc7-6bb85cdff-prc7w" Oct 30 16:48:32.003983 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:32.003952 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fdece98a-1500-46d3-96cb-ab50d9dd2068-maas-api-tls\") pod \"maas-api-e2e-inf-a-529bc7-6bb85cdff-prc7w\" (UID: \"fdece98a-1500-46d3-96cb-ab50d9dd2068\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-529bc7-6bb85cdff-prc7w" Oct 30 16:48:32.004080 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:32.003992 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdece98a-1500-46d3-96cb-ab50d9dd2068-openshift-service-ca\") pod \"maas-api-e2e-inf-a-529bc7-6bb85cdff-prc7w\" (UID: \"fdece98a-1500-46d3-96cb-ab50d9dd2068\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-529bc7-6bb85cdff-prc7w" Oct 30 16:48:32.105990 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:32.105959 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdece98a-1500-46d3-96cb-ab50d9dd2068-metrics-certs\") pod \"maas-api-e2e-inf-a-529bc7-6bb85cdff-prc7w\" (UID: \"fdece98a-1500-46d3-96cb-ab50d9dd2068\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-529bc7-6bb85cdff-prc7w" Oct 30 16:48:32.106166 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:32.106066 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncwjk\" (UniqueName: \"kubernetes.io/projected/fdece98a-1500-46d3-96cb-ab50d9dd2068-kube-api-access-ncwjk\") pod \"maas-api-e2e-inf-a-529bc7-6bb85cdff-prc7w\" (UID: \"fdece98a-1500-46d3-96cb-ab50d9dd2068\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-529bc7-6bb85cdff-prc7w" Oct 30 16:48:32.106166 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:32.106095 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fdece98a-1500-46d3-96cb-ab50d9dd2068-maas-api-tls\") pod \"maas-api-e2e-inf-a-529bc7-6bb85cdff-prc7w\" (UID: \"fdece98a-1500-46d3-96cb-ab50d9dd2068\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-529bc7-6bb85cdff-prc7w" Oct 30 16:48:32.106166 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:32.106128 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdece98a-1500-46d3-96cb-ab50d9dd2068-openshift-service-ca\") pod \"maas-api-e2e-inf-a-529bc7-6bb85cdff-prc7w\" (UID: \"fdece98a-1500-46d3-96cb-ab50d9dd2068\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-529bc7-6bb85cdff-prc7w" Oct 30 16:48:32.106855 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:32.106828 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdece98a-1500-46d3-96cb-ab50d9dd2068-openshift-service-ca\") pod \"maas-api-e2e-inf-a-529bc7-6bb85cdff-prc7w\" (UID: \"fdece98a-1500-46d3-96cb-ab50d9dd2068\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-529bc7-6bb85cdff-prc7w" Oct 30 16:48:32.108559 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:32.108519 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fdece98a-1500-46d3-96cb-ab50d9dd2068-maas-api-tls\") pod \"maas-api-e2e-inf-a-529bc7-6bb85cdff-prc7w\" (UID: \"fdece98a-1500-46d3-96cb-ab50d9dd2068\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-529bc7-6bb85cdff-prc7w" Oct 30 16:48:32.108559 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:32.108526 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdece98a-1500-46d3-96cb-ab50d9dd2068-metrics-certs\") pod \"maas-api-e2e-inf-a-529bc7-6bb85cdff-prc7w\" (UID: \"fdece98a-1500-46d3-96cb-ab50d9dd2068\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-529bc7-6bb85cdff-prc7w" Oct 30 16:48:32.131135 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:32.131109 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncwjk\" (UniqueName: \"kubernetes.io/projected/fdece98a-1500-46d3-96cb-ab50d9dd2068-kube-api-access-ncwjk\") pod \"maas-api-e2e-inf-a-529bc7-6bb85cdff-prc7w\" (UID: \"fdece98a-1500-46d3-96cb-ab50d9dd2068\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-529bc7-6bb85cdff-prc7w" Oct 30 16:48:32.273976 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:32.273889 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-529bc7-6bb85cdff-prc7w" Oct 30 16:48:32.604917 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:48:32.604889 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdece98a_1500_46d3_96cb_ab50d9dd2068.slice/crio-7e35e50bc6ce9913ba8be02f44e6970ec70f9bffc4f205d8f5406c99e327da53 WatchSource:0}: Error finding container 7e35e50bc6ce9913ba8be02f44e6970ec70f9bffc4f205d8f5406c99e327da53: Status 404 returned error can't find the container with id 7e35e50bc6ce9913ba8be02f44e6970ec70f9bffc4f205d8f5406c99e327da53 Oct 30 16:48:32.607100 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:32.607072 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-529bc7-6bb85cdff-prc7w"] Oct 30 16:48:33.032365 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:33.032331 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-529bc7-6bb85cdff-prc7w" event={"ID":"fdece98a-1500-46d3-96cb-ab50d9dd2068","Type":"ContainerStarted","Data":"7e35e50bc6ce9913ba8be02f44e6970ec70f9bffc4f205d8f5406c99e327da53"} Oct 30 16:48:34.041920 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:34.041878 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-529bc7-6bb85cdff-prc7w" event={"ID":"fdece98a-1500-46d3-96cb-ab50d9dd2068","Type":"ContainerStarted","Data":"60134c0f5fee1a88aea5f502ba106601381f01c6242ce1541cddc3247be3e0be"} Oct 30 16:48:34.042366 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:34.041949 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-529bc7-6bb85cdff-prc7w" Oct 30 16:48:34.094644 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:34.094534 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-529bc7-6bb85cdff-prc7w" podStartSLOduration=2.6688376160000002 podStartE2EDuration="3.094518368s" totalImagesPullingTime="425.680752ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:48:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:48:34.093010434 +0000 UTC m=+1669.620345678" watchObservedRunningTime="2026-10-30 16:48:34.094518368 +0000 UTC m=+1669.621853586" Oct 30 16:48:36.883618 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:36.880863 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cdwcj"] Oct 30 16:48:40.058460 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:40.058422 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-529bc7-6bb85cdff-prc7w" Oct 30 16:48:45.668994 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:45.668953 5052 scope.go:122] "RemoveContainer" containerID="92156f9a3149336337221add11aa03a36882c2a4ad2b74306bad6bfbb57ab416" Oct 30 16:48:48.286660 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:48.286630 5052 generic.go:356] "Generic (PLEG): container finished" podID="cbb6daa4-cde3-4e3f-8343-1b8ca62f8390" containerID="aed4f211ad9931452ee7d8b380b6e6031a24664f56d8843560dad937ceeb9356" exitCode=0 Oct 30 16:48:48.287023 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:48.286712 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ca5971-openshift-default-f54d56b47-qrn9w" event={"ID":"cbb6daa4-cde3-4e3f-8343-1b8ca62f8390","Type":"ContainerDied","Data":"aed4f211ad9931452ee7d8b380b6e6031a24664f56d8843560dad937ceeb9356"} Oct 30 16:48:48.287201 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:48.287187 5052 scope.go:122] "RemoveContainer" containerID="aed4f211ad9931452ee7d8b380b6e6031a24664f56d8843560dad937ceeb9356" Oct 30 16:48:48.289131 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:48.289102 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:48:48.289224 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:48.289175 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:48:48.289224 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:48.289210 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:48:48.846767 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:48:48.846725 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 16:48:49.297485 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:49.297448 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ca5971-openshift-default-f54d56b47-qrn9w" event={"ID":"cbb6daa4-cde3-4e3f-8343-1b8ca62f8390","Type":"ContainerStarted","Data":"5b98b79917c766e2f2ed7c3ee8d22e323fe8cc20d7b38e8161f6d39037082135"} Oct 30 16:48:50.053502 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:50.053020 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-ca5971-openshift-default-f54d56b47-qrn9w" Oct 30 16:48:50.053994 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:50.053955 5052 patch_prober.go:29] interesting pod/e2e-api-b-ca5971-openshift-default-f54d56b47-qrn9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Oct 30 16:48:50.054147 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:50.054016 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ca5971-openshift-default-f54d56b47-qrn9w" podUID="cbb6daa4-cde3-4e3f-8343-1b8ca62f8390" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Oct 30 16:48:51.053183 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:51.053132 5052 patch_prober.go:29] interesting pod/e2e-api-b-ca5971-openshift-default-f54d56b47-qrn9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Oct 30 16:48:51.053737 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:51.053206 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ca5971-openshift-default-f54d56b47-qrn9w" podUID="cbb6daa4-cde3-4e3f-8343-1b8ca62f8390" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Oct 30 16:48:52.053286 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:52.053127 5052 patch_prober.go:29] interesting pod/e2e-api-b-ca5971-openshift-default-f54d56b47-qrn9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Oct 30 16:48:52.053286 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:52.053188 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ca5971-openshift-default-f54d56b47-qrn9w" podUID="cbb6daa4-cde3-4e3f-8343-1b8ca62f8390" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Oct 30 16:48:52.619443 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:52.619414 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-ca5971-openshift-default-f54d56b47-qrn9w" Oct 30 16:48:52.980995 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:52.980961 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b8af660d-787d8764b-hq7zs"] Oct 30 16:48:52.985787 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:52.985765 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-b8af660d-787d8764b-hq7zs" Oct 30 16:48:53.004949 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:53.004923 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b8af660d-787d8764b-hq7zs"] Oct 30 16:48:53.013994 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:53.013970 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-b8af660d-dockercfg-p6rlm\"" Oct 30 16:48:53.013994 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:53.013969 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-b8af660d\"" Oct 30 16:48:53.059187 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:53.059097 5052 patch_prober.go:29] interesting pod/e2e-api-b-ca5971-openshift-default-f54d56b47-qrn9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Oct 30 16:48:53.059187 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:53.059159 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ca5971-openshift-default-f54d56b47-qrn9w" podUID="cbb6daa4-cde3-4e3f-8343-1b8ca62f8390" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Oct 30 16:48:53.115098 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:53.115018 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9e65258-2d9f-4341-abae-d786c09a6a56-plugins-config-volume\") pod \"payload-processing-e2e-mt-b8af660d-787d8764b-hq7zs\" (UID: \"b9e65258-2d9f-4341-abae-d786c09a6a56\") " pod="openshift-ingress/payload-processing-e2e-mt-b8af660d-787d8764b-hq7zs" Oct 30 16:48:53.143605 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:53.125912 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h29kl\" (UniqueName: \"kubernetes.io/projected/b9e65258-2d9f-4341-abae-d786c09a6a56-kube-api-access-h29kl\") pod \"payload-processing-e2e-mt-b8af660d-787d8764b-hq7zs\" (UID: \"b9e65258-2d9f-4341-abae-d786c09a6a56\") " pod="openshift-ingress/payload-processing-e2e-mt-b8af660d-787d8764b-hq7zs" Oct 30 16:48:53.267657 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:53.251152 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9e65258-2d9f-4341-abae-d786c09a6a56-plugins-config-volume\") pod \"payload-processing-e2e-mt-b8af660d-787d8764b-hq7zs\" (UID: \"b9e65258-2d9f-4341-abae-d786c09a6a56\") " pod="openshift-ingress/payload-processing-e2e-mt-b8af660d-787d8764b-hq7zs" Oct 30 16:48:53.280607 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:53.280378 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h29kl\" (UniqueName: \"kubernetes.io/projected/b9e65258-2d9f-4341-abae-d786c09a6a56-kube-api-access-h29kl\") pod \"payload-processing-e2e-mt-b8af660d-787d8764b-hq7zs\" (UID: \"b9e65258-2d9f-4341-abae-d786c09a6a56\") " pod="openshift-ingress/payload-processing-e2e-mt-b8af660d-787d8764b-hq7zs" Oct 30 16:48:53.280878 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:53.268781 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9e65258-2d9f-4341-abae-d786c09a6a56-plugins-config-volume\") pod \"payload-processing-e2e-mt-b8af660d-787d8764b-hq7zs\" (UID: \"b9e65258-2d9f-4341-abae-d786c09a6a56\") " pod="openshift-ingress/payload-processing-e2e-mt-b8af660d-787d8764b-hq7zs" Oct 30 16:48:53.417377 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:53.417341 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h29kl\" (UniqueName: \"kubernetes.io/projected/b9e65258-2d9f-4341-abae-d786c09a6a56-kube-api-access-h29kl\") pod \"payload-processing-e2e-mt-b8af660d-787d8764b-hq7zs\" (UID: \"b9e65258-2d9f-4341-abae-d786c09a6a56\") " pod="openshift-ingress/payload-processing-e2e-mt-b8af660d-787d8764b-hq7zs" Oct 30 16:48:53.600891 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:53.600758 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-b8af660d-787d8764b-hq7zs" Oct 30 16:48:53.957953 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:48:53.957921 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9e65258_2d9f_4341_abae_d786c09a6a56.slice/crio-7a4a3232481c37db391d23da9299abe24826a2ffa21a2255b762c535aaffdff3 WatchSource:0}: Error finding container 7a4a3232481c37db391d23da9299abe24826a2ffa21a2255b762c535aaffdff3: Status 404 returned error can't find the container with id 7a4a3232481c37db391d23da9299abe24826a2ffa21a2255b762c535aaffdff3 Oct 30 16:48:53.961172 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:53.961144 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b8af660d-787d8764b-hq7zs"] Oct 30 16:48:54.054706 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:54.053923 5052 patch_prober.go:29] interesting pod/e2e-api-b-ca5971-openshift-default-f54d56b47-qrn9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Oct 30 16:48:54.054706 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:54.053979 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ca5971-openshift-default-f54d56b47-qrn9w" podUID="cbb6daa4-cde3-4e3f-8343-1b8ca62f8390" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Oct 30 16:48:54.564497 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:54.564442 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-b8af660d-787d8764b-hq7zs" event={"ID":"b9e65258-2d9f-4341-abae-d786c09a6a56","Type":"ContainerStarted","Data":"4112f8da7f8b4120b5371d0eb443db95af671acdd97f308e429d186d604463c8"} Oct 30 16:48:54.564964 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:54.564506 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-b8af660d-787d8764b-hq7zs" event={"ID":"b9e65258-2d9f-4341-abae-d786c09a6a56","Type":"ContainerStarted","Data":"7a4a3232481c37db391d23da9299abe24826a2ffa21a2255b762c535aaffdff3"} Oct 30 16:48:54.564964 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:54.564533 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-b8af660d-787d8764b-hq7zs" Oct 30 16:48:54.612099 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:54.612052 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-b8af660d-787d8764b-hq7zs" podStartSLOduration=2.612037993 podStartE2EDuration="2.612037993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:48:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:48:54.609990115 +0000 UTC m=+1690.137325334" watchObservedRunningTime="2026-10-30 16:48:54.612037993 +0000 UTC m=+1690.139373212" Oct 30 16:48:55.051981 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:55.051940 5052 patch_prober.go:29] interesting pod/e2e-api-b-ca5971-openshift-default-f54d56b47-qrn9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Oct 30 16:48:55.052178 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:55.052013 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ca5971-openshift-default-f54d56b47-qrn9w" podUID="cbb6daa4-cde3-4e3f-8343-1b8ca62f8390" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Oct 30 16:48:56.058282 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:56.058250 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-ca5971-openshift-default-f54d56b47-qrn9w" Oct 30 16:48:56.059477 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:48:56.059454 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-ca5971-openshift-default-f54d56b47-qrn9w" Oct 30 16:49:05.576632 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:05.576569 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-b8af660d-787d8764b-hq7zs" Oct 30 16:49:05.898181 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:05.898146 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-v9r87"] Oct 30 16:49:05.898422 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:05.898371 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-v9r87" podUID="20129ef0-4c2c-498f-9427-985db412bb2e" containerName="istio-proxy" containerID="cri-o://95c15eab4330dec0ed5761be3ae5327a9fe786d4cb463872766b7d5204f8fd78" gracePeriod=30 Oct 30 16:49:05.911094 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:05.911055 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-92x47"] Oct 30 16:49:05.911345 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:05.911315 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-92x47" podUID="39ebd9d1-3580-45a0-84ca-de3126fe5224" containerName="istio-proxy" containerID="cri-o://fa96c67b575da8b32a91902eca0c2fdea9954fec81050b3dc444dec6c5bbc283" gracePeriod=30 Oct 30 16:49:05.915447 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:05.915423 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2slrc"] Oct 30 16:49:05.915624 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:05.915604 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2slrc" podUID="69a65b89-4d81-476b-b4b7-06736537fa4a" containerName="istio-proxy" containerID="cri-o://653b37f0a36ca625c0e710ae2c5299203f8da7df81f26a3b6513447f5ef2d81f" gracePeriod=30 Oct 30 16:49:05.920077 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:05.920053 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-98gfq"] Oct 30 16:49:05.920263 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:05.920243 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-98gfq" podUID="c22edc73-463a-4439-92aa-601b4cddd714" containerName="istio-proxy" containerID="cri-o://80c70291d36578c73cf3ef5ac29fca0a9b96ca339b43923582b9ecf045d4afdf" gracePeriod=30 Oct 30 16:49:11.249496 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.249459 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-v9r87" Oct 30 16:49:11.271736 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.271711 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-98gfq" Oct 30 16:49:11.319149 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.319112 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20129ef0-4c2c-498f-9427-985db412bb2e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20129ef0-4c2c-498f-9427-985db412bb2e-credential-socket\") pod \"20129ef0-4c2c-498f-9427-985db412bb2e\" (UID: \"20129ef0-4c2c-498f-9427-985db412bb2e\") " Oct 30 16:49:11.319340 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.319164 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20129ef0-4c2c-498f-9427-985db412bb2e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20129ef0-4c2c-498f-9427-985db412bb2e-workload-certs\") pod \"20129ef0-4c2c-498f-9427-985db412bb2e\" (UID: \"20129ef0-4c2c-498f-9427-985db412bb2e\") " Oct 30 16:49:11.319340 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.319262 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20129ef0-4c2c-498f-9427-985db412bb2e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20129ef0-4c2c-498f-9427-985db412bb2e-istiod-ca-cert\") pod \"20129ef0-4c2c-498f-9427-985db412bb2e\" (UID: \"20129ef0-4c2c-498f-9427-985db412bb2e\") " Oct 30 16:49:11.319502 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.319353 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20129ef0-4c2c-498f-9427-985db412bb2e-istio-token\" (UniqueName: \"kubernetes.io/projected/20129ef0-4c2c-498f-9427-985db412bb2e-istio-token\") pod \"20129ef0-4c2c-498f-9427-985db412bb2e\" (UID: \"20129ef0-4c2c-498f-9427-985db412bb2e\") " Oct 30 16:49:11.319502 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.319417 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20129ef0-4c2c-498f-9427-985db412bb2e-credential-socket" pod "20129ef0-4c2c-498f-9427-985db412bb2e" (UID: "20129ef0-4c2c-498f-9427-985db412bb2e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:49:11.319502 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.319489 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20129ef0-4c2c-498f-9427-985db412bb2e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20129ef0-4c2c-498f-9427-985db412bb2e-workload-socket\") pod \"20129ef0-4c2c-498f-9427-985db412bb2e\" (UID: \"20129ef0-4c2c-498f-9427-985db412bb2e\") " Oct 30 16:49:11.319718 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.319570 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20129ef0-4c2c-498f-9427-985db412bb2e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20129ef0-4c2c-498f-9427-985db412bb2e-istio-envoy\") pod \"20129ef0-4c2c-498f-9427-985db412bb2e\" (UID: \"20129ef0-4c2c-498f-9427-985db412bb2e\") " Oct 30 16:49:11.319718 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.319689 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/20129ef0-4c2c-498f-9427-985db412bb2e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20129ef0-4c2c-498f-9427-985db412bb2e-istio-podinfo\") pod \"20129ef0-4c2c-498f-9427-985db412bb2e\" (UID: \"20129ef0-4c2c-498f-9427-985db412bb2e\") " Oct 30 16:49:11.319859 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.319754 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20129ef0-4c2c-498f-9427-985db412bb2e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20129ef0-4c2c-498f-9427-985db412bb2e-istio-data\") pod \"20129ef0-4c2c-498f-9427-985db412bb2e\" (UID: \"20129ef0-4c2c-498f-9427-985db412bb2e\") " Oct 30 16:49:11.319956 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.319939 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20129ef0-4c2c-498f-9427-985db412bb2e-kube-api-access-mj6sd\" (UniqueName: \"kubernetes.io/projected/20129ef0-4c2c-498f-9427-985db412bb2e-kube-api-access-mj6sd\") pod \"20129ef0-4c2c-498f-9427-985db412bb2e\" (UID: \"20129ef0-4c2c-498f-9427-985db412bb2e\") " Oct 30 16:49:11.320017 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.319982 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20129ef0-4c2c-498f-9427-985db412bb2e-workload-certs" pod "20129ef0-4c2c-498f-9427-985db412bb2e" (UID: "20129ef0-4c2c-498f-9427-985db412bb2e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:49:11.320805 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.320652 5052 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20129ef0-4c2c-498f-9427-985db412bb2e-credential-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:49:11.320805 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.320679 5052 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20129ef0-4c2c-498f-9427-985db412bb2e-workload-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:49:11.321344 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.321318 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20129ef0-4c2c-498f-9427-985db412bb2e-istio-data" pod "20129ef0-4c2c-498f-9427-985db412bb2e" (UID: "20129ef0-4c2c-498f-9427-985db412bb2e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:49:11.321445 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.321377 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20129ef0-4c2c-498f-9427-985db412bb2e-workload-socket" pod "20129ef0-4c2c-498f-9427-985db412bb2e" (UID: "20129ef0-4c2c-498f-9427-985db412bb2e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:49:11.322721 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.322684 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20129ef0-4c2c-498f-9427-985db412bb2e-istiod-ca-cert" pod "20129ef0-4c2c-498f-9427-985db412bb2e" (UID: "20129ef0-4c2c-498f-9427-985db412bb2e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:49:11.338464 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.336559 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20129ef0-4c2c-498f-9427-985db412bb2e-istio-envoy" pod "20129ef0-4c2c-498f-9427-985db412bb2e" (UID: "20129ef0-4c2c-498f-9427-985db412bb2e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:49:11.338464 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.336809 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20129ef0-4c2c-498f-9427-985db412bb2e-istio-token" pod "20129ef0-4c2c-498f-9427-985db412bb2e" (UID: "20129ef0-4c2c-498f-9427-985db412bb2e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:49:11.338464 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.336890 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20129ef0-4c2c-498f-9427-985db412bb2e-kube-api-access-mj6sd" pod "20129ef0-4c2c-498f-9427-985db412bb2e" (UID: "20129ef0-4c2c-498f-9427-985db412bb2e"). InnerVolumeSpecName "kube-api-access-mj6sd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:49:11.357378 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.357297 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-92x47" Oct 30 16:49:11.367499 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.364078 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/20129ef0-4c2c-498f-9427-985db412bb2e-istio-podinfo" pod "20129ef0-4c2c-498f-9427-985db412bb2e" (UID: "20129ef0-4c2c-498f-9427-985db412bb2e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:49:11.384998 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.384972 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2slrc" Oct 30 16:49:11.421641 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.421604 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c22edc73-463a-4439-92aa-601b4cddd714-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c22edc73-463a-4439-92aa-601b4cddd714-istio-envoy\") pod \"c22edc73-463a-4439-92aa-601b4cddd714\" (UID: \"c22edc73-463a-4439-92aa-601b4cddd714\") " Oct 30 16:49:11.421833 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.421657 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c22edc73-463a-4439-92aa-601b4cddd714-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c22edc73-463a-4439-92aa-601b4cddd714-credential-socket\") pod \"c22edc73-463a-4439-92aa-601b4cddd714\" (UID: \"c22edc73-463a-4439-92aa-601b4cddd714\") " Oct 30 16:49:11.421833 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.421712 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39ebd9d1-3580-45a0-84ca-de3126fe5224-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39ebd9d1-3580-45a0-84ca-de3126fe5224-workload-certs\") pod \"39ebd9d1-3580-45a0-84ca-de3126fe5224\" (UID: \"39ebd9d1-3580-45a0-84ca-de3126fe5224\") " Oct 30 16:49:11.421833 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.421755 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/39ebd9d1-3580-45a0-84ca-de3126fe5224-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39ebd9d1-3580-45a0-84ca-de3126fe5224-istio-podinfo\") pod \"39ebd9d1-3580-45a0-84ca-de3126fe5224\" (UID: \"39ebd9d1-3580-45a0-84ca-de3126fe5224\") " Oct 30 16:49:11.422004 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.421873 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39ebd9d1-3580-45a0-84ca-de3126fe5224-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39ebd9d1-3580-45a0-84ca-de3126fe5224-workload-socket\") pod \"39ebd9d1-3580-45a0-84ca-de3126fe5224\" (UID: \"39ebd9d1-3580-45a0-84ca-de3126fe5224\") " Oct 30 16:49:11.422004 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.421942 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c22edc73-463a-4439-92aa-601b4cddd714-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c22edc73-463a-4439-92aa-601b4cddd714-istiod-ca-cert\") pod \"c22edc73-463a-4439-92aa-601b4cddd714\" (UID: \"c22edc73-463a-4439-92aa-601b4cddd714\") " Oct 30 16:49:11.422121 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.422019 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c22edc73-463a-4439-92aa-601b4cddd714-credential-socket" pod "c22edc73-463a-4439-92aa-601b4cddd714" (UID: "c22edc73-463a-4439-92aa-601b4cddd714"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:49:11.422121 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.422042 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c22edc73-463a-4439-92aa-601b4cddd714-istio-token\" (UniqueName: \"kubernetes.io/projected/c22edc73-463a-4439-92aa-601b4cddd714-istio-token\") pod \"c22edc73-463a-4439-92aa-601b4cddd714\" (UID: \"c22edc73-463a-4439-92aa-601b4cddd714\") " Oct 30 16:49:11.422121 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.422089 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c22edc73-463a-4439-92aa-601b4cddd714-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c22edc73-463a-4439-92aa-601b4cddd714-workload-certs\") pod \"c22edc73-463a-4439-92aa-601b4cddd714\" (UID: \"c22edc73-463a-4439-92aa-601b4cddd714\") " Oct 30 16:49:11.422272 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.422142 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39ebd9d1-3580-45a0-84ca-de3126fe5224-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39ebd9d1-3580-45a0-84ca-de3126fe5224-istiod-ca-cert\") pod \"39ebd9d1-3580-45a0-84ca-de3126fe5224\" (UID: \"39ebd9d1-3580-45a0-84ca-de3126fe5224\") " Oct 30 16:49:11.422272 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.422175 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39ebd9d1-3580-45a0-84ca-de3126fe5224-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39ebd9d1-3580-45a0-84ca-de3126fe5224-istio-data\") pod \"39ebd9d1-3580-45a0-84ca-de3126fe5224\" (UID: \"39ebd9d1-3580-45a0-84ca-de3126fe5224\") " Oct 30 16:49:11.422374 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.422287 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c22edc73-463a-4439-92aa-601b4cddd714-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c22edc73-463a-4439-92aa-601b4cddd714-workload-socket\") pod \"c22edc73-463a-4439-92aa-601b4cddd714\" (UID: \"c22edc73-463a-4439-92aa-601b4cddd714\") " Oct 30 16:49:11.422374 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.422328 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39ebd9d1-3580-45a0-84ca-de3126fe5224-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39ebd9d1-3580-45a0-84ca-de3126fe5224-credential-socket\") pod \"39ebd9d1-3580-45a0-84ca-de3126fe5224\" (UID: \"39ebd9d1-3580-45a0-84ca-de3126fe5224\") " Oct 30 16:49:11.422474 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.422372 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39ebd9d1-3580-45a0-84ca-de3126fe5224-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39ebd9d1-3580-45a0-84ca-de3126fe5224-istio-envoy\") pod \"39ebd9d1-3580-45a0-84ca-de3126fe5224\" (UID: \"39ebd9d1-3580-45a0-84ca-de3126fe5224\") " Oct 30 16:49:11.422474 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.422408 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c22edc73-463a-4439-92aa-601b4cddd714-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c22edc73-463a-4439-92aa-601b4cddd714-istio-data\") pod \"c22edc73-463a-4439-92aa-601b4cddd714\" (UID: \"c22edc73-463a-4439-92aa-601b4cddd714\") " Oct 30 16:49:11.422474 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.422445 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c22edc73-463a-4439-92aa-601b4cddd714-kube-api-access-nrkxn\" (UniqueName: \"kubernetes.io/projected/c22edc73-463a-4439-92aa-601b4cddd714-kube-api-access-nrkxn\") pod \"c22edc73-463a-4439-92aa-601b4cddd714\" (UID: \"c22edc73-463a-4439-92aa-601b4cddd714\") " Oct 30 16:49:11.422786 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.422485 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39ebd9d1-3580-45a0-84ca-de3126fe5224-istio-token\" (UniqueName: \"kubernetes.io/projected/39ebd9d1-3580-45a0-84ca-de3126fe5224-istio-token\") pod \"39ebd9d1-3580-45a0-84ca-de3126fe5224\" (UID: \"39ebd9d1-3580-45a0-84ca-de3126fe5224\") " Oct 30 16:49:11.422786 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.422574 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39ebd9d1-3580-45a0-84ca-de3126fe5224-kube-api-access-7d5mh\" (UniqueName: \"kubernetes.io/projected/39ebd9d1-3580-45a0-84ca-de3126fe5224-kube-api-access-7d5mh\") pod \"39ebd9d1-3580-45a0-84ca-de3126fe5224\" (UID: \"39ebd9d1-3580-45a0-84ca-de3126fe5224\") " Oct 30 16:49:11.422786 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.422717 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c22edc73-463a-4439-92aa-601b4cddd714-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c22edc73-463a-4439-92aa-601b4cddd714-istio-podinfo\") pod \"c22edc73-463a-4439-92aa-601b4cddd714\" (UID: \"c22edc73-463a-4439-92aa-601b4cddd714\") " Oct 30 16:49:11.423119 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.423091 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39ebd9d1-3580-45a0-84ca-de3126fe5224-workload-certs" pod "39ebd9d1-3580-45a0-84ca-de3126fe5224" (UID: "39ebd9d1-3580-45a0-84ca-de3126fe5224"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:49:11.423194 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.423126 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39ebd9d1-3580-45a0-84ca-de3126fe5224-istio-data" pod "39ebd9d1-3580-45a0-84ca-de3126fe5224" (UID: "39ebd9d1-3580-45a0-84ca-de3126fe5224"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:49:11.423639 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.423611 5052 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20129ef0-4c2c-498f-9427-985db412bb2e-istio-token\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:49:11.423639 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.423638 5052 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20129ef0-4c2c-498f-9427-985db412bb2e-workload-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:49:11.423803 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.423652 5052 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20129ef0-4c2c-498f-9427-985db412bb2e-istio-envoy\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:49:11.423803 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.423665 5052 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20129ef0-4c2c-498f-9427-985db412bb2e-istio-podinfo\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:49:11.423803 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.423675 5052 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20129ef0-4c2c-498f-9427-985db412bb2e-istio-data\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:49:11.423803 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.423687 5052 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39ebd9d1-3580-45a0-84ca-de3126fe5224-istio-data\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:49:11.423803 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.423698 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mj6sd\" (UniqueName: \"kubernetes.io/projected/20129ef0-4c2c-498f-9427-985db412bb2e-kube-api-access-mj6sd\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:49:11.423803 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.423712 5052 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20129ef0-4c2c-498f-9427-985db412bb2e-istiod-ca-cert\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:49:11.423803 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.423727 5052 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c22edc73-463a-4439-92aa-601b4cddd714-credential-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:49:11.423803 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.423740 5052 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39ebd9d1-3580-45a0-84ca-de3126fe5224-workload-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:49:11.424244 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.424193 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c22edc73-463a-4439-92aa-601b4cddd714-istio-data" pod "c22edc73-463a-4439-92aa-601b4cddd714" (UID: "c22edc73-463a-4439-92aa-601b4cddd714"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:49:11.424393 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.424371 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c22edc73-463a-4439-92aa-601b4cddd714-workload-socket" pod "c22edc73-463a-4439-92aa-601b4cddd714" (UID: "c22edc73-463a-4439-92aa-601b4cddd714"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:49:11.424574 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.424484 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c22edc73-463a-4439-92aa-601b4cddd714-workload-certs" pod "c22edc73-463a-4439-92aa-601b4cddd714" (UID: "c22edc73-463a-4439-92aa-601b4cddd714"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:49:11.424866 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.424842 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39ebd9d1-3580-45a0-84ca-de3126fe5224-istiod-ca-cert" pod "39ebd9d1-3580-45a0-84ca-de3126fe5224" (UID: "39ebd9d1-3580-45a0-84ca-de3126fe5224"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:49:11.425012 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.424991 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39ebd9d1-3580-45a0-84ca-de3126fe5224-credential-socket" pod "39ebd9d1-3580-45a0-84ca-de3126fe5224" (UID: "39ebd9d1-3580-45a0-84ca-de3126fe5224"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:49:11.433807 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.433500 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39ebd9d1-3580-45a0-84ca-de3126fe5224-workload-socket" pod "39ebd9d1-3580-45a0-84ca-de3126fe5224" (UID: "39ebd9d1-3580-45a0-84ca-de3126fe5224"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:49:11.433900 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.433811 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c22edc73-463a-4439-92aa-601b4cddd714-istiod-ca-cert" pod "c22edc73-463a-4439-92aa-601b4cddd714" (UID: "c22edc73-463a-4439-92aa-601b4cddd714"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:49:11.436902 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.436874 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c22edc73-463a-4439-92aa-601b4cddd714-kube-api-access-nrkxn" pod "c22edc73-463a-4439-92aa-601b4cddd714" (UID: "c22edc73-463a-4439-92aa-601b4cddd714"). InnerVolumeSpecName "kube-api-access-nrkxn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:49:11.438646 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.438609 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c22edc73-463a-4439-92aa-601b4cddd714-istio-envoy" pod "c22edc73-463a-4439-92aa-601b4cddd714" (UID: "c22edc73-463a-4439-92aa-601b4cddd714"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:49:11.442890 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.440359 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39ebd9d1-3580-45a0-84ca-de3126fe5224-kube-api-access-7d5mh" pod "39ebd9d1-3580-45a0-84ca-de3126fe5224" (UID: "39ebd9d1-3580-45a0-84ca-de3126fe5224"). InnerVolumeSpecName "kube-api-access-7d5mh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:49:11.444962 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.444932 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c22edc73-463a-4439-92aa-601b4cddd714-istio-podinfo" pod "c22edc73-463a-4439-92aa-601b4cddd714" (UID: "c22edc73-463a-4439-92aa-601b4cddd714"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:49:11.444962 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.444940 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/39ebd9d1-3580-45a0-84ca-de3126fe5224-istio-podinfo" pod "39ebd9d1-3580-45a0-84ca-de3126fe5224" (UID: "39ebd9d1-3580-45a0-84ca-de3126fe5224"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:49:11.445109 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.444963 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c22edc73-463a-4439-92aa-601b4cddd714-istio-token" pod "c22edc73-463a-4439-92aa-601b4cddd714" (UID: "c22edc73-463a-4439-92aa-601b4cddd714"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:49:11.445109 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.445027 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39ebd9d1-3580-45a0-84ca-de3126fe5224-istio-token" pod "39ebd9d1-3580-45a0-84ca-de3126fe5224" (UID: "39ebd9d1-3580-45a0-84ca-de3126fe5224"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:49:11.446353 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.446321 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39ebd9d1-3580-45a0-84ca-de3126fe5224-istio-envoy" pod "39ebd9d1-3580-45a0-84ca-de3126fe5224" (UID: "39ebd9d1-3580-45a0-84ca-de3126fe5224"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:49:11.524785 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.524700 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69a65b89-4d81-476b-b4b7-06736537fa4a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69a65b89-4d81-476b-b4b7-06736537fa4a-istio-data\") pod \"69a65b89-4d81-476b-b4b7-06736537fa4a\" (UID: \"69a65b89-4d81-476b-b4b7-06736537fa4a\") " Oct 30 16:49:11.524785 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.524743 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69a65b89-4d81-476b-b4b7-06736537fa4a-istio-token\" (UniqueName: \"kubernetes.io/projected/69a65b89-4d81-476b-b4b7-06736537fa4a-istio-token\") pod \"69a65b89-4d81-476b-b4b7-06736537fa4a\" (UID: \"69a65b89-4d81-476b-b4b7-06736537fa4a\") " Oct 30 16:49:11.525019 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.524865 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/69a65b89-4d81-476b-b4b7-06736537fa4a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69a65b89-4d81-476b-b4b7-06736537fa4a-istio-podinfo\") pod \"69a65b89-4d81-476b-b4b7-06736537fa4a\" (UID: \"69a65b89-4d81-476b-b4b7-06736537fa4a\") " Oct 30 16:49:11.525019 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.524887 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69a65b89-4d81-476b-b4b7-06736537fa4a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69a65b89-4d81-476b-b4b7-06736537fa4a-workload-socket\") pod \"69a65b89-4d81-476b-b4b7-06736537fa4a\" (UID: \"69a65b89-4d81-476b-b4b7-06736537fa4a\") " Oct 30 16:49:11.525130 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.525010 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69a65b89-4d81-476b-b4b7-06736537fa4a-istio-data" pod "69a65b89-4d81-476b-b4b7-06736537fa4a" (UID: "69a65b89-4d81-476b-b4b7-06736537fa4a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:49:11.525130 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.525053 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69a65b89-4d81-476b-b4b7-06736537fa4a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69a65b89-4d81-476b-b4b7-06736537fa4a-istio-envoy\") pod \"69a65b89-4d81-476b-b4b7-06736537fa4a\" (UID: \"69a65b89-4d81-476b-b4b7-06736537fa4a\") " Oct 30 16:49:11.525238 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.525132 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69a65b89-4d81-476b-b4b7-06736537fa4a-kube-api-access-m9pf8\" (UniqueName: \"kubernetes.io/projected/69a65b89-4d81-476b-b4b7-06736537fa4a-kube-api-access-m9pf8\") pod \"69a65b89-4d81-476b-b4b7-06736537fa4a\" (UID: \"69a65b89-4d81-476b-b4b7-06736537fa4a\") " Oct 30 16:49:11.525238 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.525188 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69a65b89-4d81-476b-b4b7-06736537fa4a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69a65b89-4d81-476b-b4b7-06736537fa4a-workload-certs\") pod \"69a65b89-4d81-476b-b4b7-06736537fa4a\" (UID: \"69a65b89-4d81-476b-b4b7-06736537fa4a\") " Oct 30 16:49:11.525238 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.525190 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69a65b89-4d81-476b-b4b7-06736537fa4a-workload-socket" pod "69a65b89-4d81-476b-b4b7-06736537fa4a" (UID: "69a65b89-4d81-476b-b4b7-06736537fa4a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:49:11.525455 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.525270 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69a65b89-4d81-476b-b4b7-06736537fa4a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69a65b89-4d81-476b-b4b7-06736537fa4a-credential-socket\") pod \"69a65b89-4d81-476b-b4b7-06736537fa4a\" (UID: \"69a65b89-4d81-476b-b4b7-06736537fa4a\") " Oct 30 16:49:11.525455 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.525348 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69a65b89-4d81-476b-b4b7-06736537fa4a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69a65b89-4d81-476b-b4b7-06736537fa4a-istiod-ca-cert\") pod \"69a65b89-4d81-476b-b4b7-06736537fa4a\" (UID: \"69a65b89-4d81-476b-b4b7-06736537fa4a\") " Oct 30 16:49:11.526035 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.525863 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69a65b89-4d81-476b-b4b7-06736537fa4a-workload-certs" pod "69a65b89-4d81-476b-b4b7-06736537fa4a" (UID: "69a65b89-4d81-476b-b4b7-06736537fa4a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:49:11.526035 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.525883 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69a65b89-4d81-476b-b4b7-06736537fa4a-credential-socket" pod "69a65b89-4d81-476b-b4b7-06736537fa4a" (UID: "69a65b89-4d81-476b-b4b7-06736537fa4a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:49:11.526278 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.526215 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69a65b89-4d81-476b-b4b7-06736537fa4a-istiod-ca-cert" pod "69a65b89-4d81-476b-b4b7-06736537fa4a" (UID: "69a65b89-4d81-476b-b4b7-06736537fa4a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:49:11.526384 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.526337 5052 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69a65b89-4d81-476b-b4b7-06736537fa4a-workload-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:49:11.526384 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.526360 5052 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c22edc73-463a-4439-92aa-601b4cddd714-workload-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:49:11.526384 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.526377 5052 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39ebd9d1-3580-45a0-84ca-de3126fe5224-credential-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:49:11.526572 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.526393 5052 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39ebd9d1-3580-45a0-84ca-de3126fe5224-istio-envoy\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:49:11.526572 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.526407 5052 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c22edc73-463a-4439-92aa-601b4cddd714-istio-data\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:49:11.526572 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.526421 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrkxn\" (UniqueName: \"kubernetes.io/projected/c22edc73-463a-4439-92aa-601b4cddd714-kube-api-access-nrkxn\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:49:11.526572 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.526435 5052 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39ebd9d1-3580-45a0-84ca-de3126fe5224-istio-token\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:49:11.526572 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.526448 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7d5mh\" (UniqueName: \"kubernetes.io/projected/39ebd9d1-3580-45a0-84ca-de3126fe5224-kube-api-access-7d5mh\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:49:11.526572 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.526461 5052 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c22edc73-463a-4439-92aa-601b4cddd714-istio-podinfo\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:49:11.526572 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.526474 5052 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c22edc73-463a-4439-92aa-601b4cddd714-istio-envoy\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:49:11.526572 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.526487 5052 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39ebd9d1-3580-45a0-84ca-de3126fe5224-istio-podinfo\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:49:11.526572 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.526498 5052 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69a65b89-4d81-476b-b4b7-06736537fa4a-workload-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:49:11.526572 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.526511 5052 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69a65b89-4d81-476b-b4b7-06736537fa4a-credential-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:49:11.526572 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.526525 5052 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39ebd9d1-3580-45a0-84ca-de3126fe5224-workload-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:49:11.526572 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.526538 5052 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c22edc73-463a-4439-92aa-601b4cddd714-istiod-ca-cert\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:49:11.526572 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.526551 5052 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69a65b89-4d81-476b-b4b7-06736537fa4a-istio-data\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:49:11.526572 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.526563 5052 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c22edc73-463a-4439-92aa-601b4cddd714-istio-token\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:49:11.526572 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.526595 5052 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c22edc73-463a-4439-92aa-601b4cddd714-workload-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:49:11.527398 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.526609 5052 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39ebd9d1-3580-45a0-84ca-de3126fe5224-istiod-ca-cert\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:49:11.527398 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.527245 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/69a65b89-4d81-476b-b4b7-06736537fa4a-istio-podinfo" pod "69a65b89-4d81-476b-b4b7-06736537fa4a" (UID: "69a65b89-4d81-476b-b4b7-06736537fa4a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:49:11.527398 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.527374 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69a65b89-4d81-476b-b4b7-06736537fa4a-istio-envoy" pod "69a65b89-4d81-476b-b4b7-06736537fa4a" (UID: "69a65b89-4d81-476b-b4b7-06736537fa4a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:49:11.527678 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.527561 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69a65b89-4d81-476b-b4b7-06736537fa4a-kube-api-access-m9pf8" pod "69a65b89-4d81-476b-b4b7-06736537fa4a" (UID: "69a65b89-4d81-476b-b4b7-06736537fa4a"). InnerVolumeSpecName "kube-api-access-m9pf8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:49:11.527678 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.527613 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69a65b89-4d81-476b-b4b7-06736537fa4a-istio-token" pod "69a65b89-4d81-476b-b4b7-06736537fa4a" (UID: "69a65b89-4d81-476b-b4b7-06736537fa4a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:49:11.628362 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.628320 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9pf8\" (UniqueName: \"kubernetes.io/projected/69a65b89-4d81-476b-b4b7-06736537fa4a-kube-api-access-m9pf8\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:49:11.628362 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.628366 5052 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69a65b89-4d81-476b-b4b7-06736537fa4a-istiod-ca-cert\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:49:11.628706 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.628383 5052 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69a65b89-4d81-476b-b4b7-06736537fa4a-istio-token\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:49:11.628706 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.628396 5052 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69a65b89-4d81-476b-b4b7-06736537fa4a-istio-podinfo\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:49:11.628706 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.628408 5052 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69a65b89-4d81-476b-b4b7-06736537fa4a-istio-envoy\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:49:11.760642 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.759305 5052 generic.go:356] "Generic (PLEG): container finished" podID="c22edc73-463a-4439-92aa-601b4cddd714" containerID="80c70291d36578c73cf3ef5ac29fca0a9b96ca339b43923582b9ecf045d4afdf" exitCode=0 Oct 30 16:49:11.760642 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.759372 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-98gfq" event={"ID":"c22edc73-463a-4439-92aa-601b4cddd714","Type":"ContainerDied","Data":"80c70291d36578c73cf3ef5ac29fca0a9b96ca339b43923582b9ecf045d4afdf"} Oct 30 16:49:11.760642 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.759441 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-98gfq" event={"ID":"c22edc73-463a-4439-92aa-601b4cddd714","Type":"ContainerDied","Data":"538324dedebe43c2b4c2867c742b92eefb8d3ce338a99020628d9bd5014820b6"} Oct 30 16:49:11.760642 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.759473 5052 scope.go:122] "RemoveContainer" containerID="80c70291d36578c73cf3ef5ac29fca0a9b96ca339b43923582b9ecf045d4afdf" Oct 30 16:49:11.760642 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.759650 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-98gfq" Oct 30 16:49:11.761128 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.761103 5052 generic.go:356] "Generic (PLEG): container finished" podID="39ebd9d1-3580-45a0-84ca-de3126fe5224" containerID="fa96c67b575da8b32a91902eca0c2fdea9954fec81050b3dc444dec6c5bbc283" exitCode=0 Oct 30 16:49:11.761262 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.761161 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-92x47" Oct 30 16:49:11.761321 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.761299 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-92x47" event={"ID":"39ebd9d1-3580-45a0-84ca-de3126fe5224","Type":"ContainerDied","Data":"fa96c67b575da8b32a91902eca0c2fdea9954fec81050b3dc444dec6c5bbc283"} Oct 30 16:49:11.761360 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.761346 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-92x47" event={"ID":"39ebd9d1-3580-45a0-84ca-de3126fe5224","Type":"ContainerDied","Data":"c20cc8bc540fa007e4580e5c42cf64b41a2989a081fa07f301e209ac0d53012e"} Oct 30 16:49:11.762950 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.762928 5052 generic.go:356] "Generic (PLEG): container finished" podID="20129ef0-4c2c-498f-9427-985db412bb2e" containerID="95c15eab4330dec0ed5761be3ae5327a9fe786d4cb463872766b7d5204f8fd78" exitCode=0 Oct 30 16:49:11.763186 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.762977 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-v9r87" Oct 30 16:49:11.763275 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.763015 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-v9r87" event={"ID":"20129ef0-4c2c-498f-9427-985db412bb2e","Type":"ContainerDied","Data":"95c15eab4330dec0ed5761be3ae5327a9fe786d4cb463872766b7d5204f8fd78"} Oct 30 16:49:11.763343 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.763315 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-v9r87" event={"ID":"20129ef0-4c2c-498f-9427-985db412bb2e","Type":"ContainerDied","Data":"05c597d856e44a702ec3f1b620c1e2ab3c3d078b7fb639e4684d275efb2ad377"} Oct 30 16:49:11.765874 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.764963 5052 generic.go:356] "Generic (PLEG): container finished" podID="69a65b89-4d81-476b-b4b7-06736537fa4a" containerID="653b37f0a36ca625c0e710ae2c5299203f8da7df81f26a3b6513447f5ef2d81f" exitCode=0 Oct 30 16:49:11.765874 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.765007 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2slrc" Oct 30 16:49:11.765874 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.765011 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2slrc" event={"ID":"69a65b89-4d81-476b-b4b7-06736537fa4a","Type":"ContainerDied","Data":"653b37f0a36ca625c0e710ae2c5299203f8da7df81f26a3b6513447f5ef2d81f"} Oct 30 16:49:11.765874 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.765085 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2slrc" event={"ID":"69a65b89-4d81-476b-b4b7-06736537fa4a","Type":"ContainerDied","Data":"08c518ba2b3f5eec81978e237b3b1210114de3d3f2803e177207f817235d04a8"} Oct 30 16:49:11.778420 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.777102 5052 scope.go:122] "RemoveContainer" containerID="80c70291d36578c73cf3ef5ac29fca0a9b96ca339b43923582b9ecf045d4afdf" Oct 30 16:49:11.778420 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:49:11.777464 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80c70291d36578c73cf3ef5ac29fca0a9b96ca339b43923582b9ecf045d4afdf\": container with ID starting with 80c70291d36578c73cf3ef5ac29fca0a9b96ca339b43923582b9ecf045d4afdf not found: ID does not exist" containerID="80c70291d36578c73cf3ef5ac29fca0a9b96ca339b43923582b9ecf045d4afdf" Oct 30 16:49:11.778420 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.777495 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80c70291d36578c73cf3ef5ac29fca0a9b96ca339b43923582b9ecf045d4afdf"} err="failed to get container status \"80c70291d36578c73cf3ef5ac29fca0a9b96ca339b43923582b9ecf045d4afdf\": rpc error: code = NotFound desc = could not find container \"80c70291d36578c73cf3ef5ac29fca0a9b96ca339b43923582b9ecf045d4afdf\": container with ID starting with 80c70291d36578c73cf3ef5ac29fca0a9b96ca339b43923582b9ecf045d4afdf not found: ID does not exist" Oct 30 16:49:11.778420 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.777518 5052 scope.go:122] "RemoveContainer" containerID="fa96c67b575da8b32a91902eca0c2fdea9954fec81050b3dc444dec6c5bbc283" Oct 30 16:49:11.793036 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.793013 5052 scope.go:122] "RemoveContainer" containerID="fa96c67b575da8b32a91902eca0c2fdea9954fec81050b3dc444dec6c5bbc283" Oct 30 16:49:11.793632 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:49:11.793565 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa96c67b575da8b32a91902eca0c2fdea9954fec81050b3dc444dec6c5bbc283\": container with ID starting with fa96c67b575da8b32a91902eca0c2fdea9954fec81050b3dc444dec6c5bbc283 not found: ID does not exist" containerID="fa96c67b575da8b32a91902eca0c2fdea9954fec81050b3dc444dec6c5bbc283" Oct 30 16:49:11.793731 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.793643 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa96c67b575da8b32a91902eca0c2fdea9954fec81050b3dc444dec6c5bbc283"} err="failed to get container status \"fa96c67b575da8b32a91902eca0c2fdea9954fec81050b3dc444dec6c5bbc283\": rpc error: code = NotFound desc = could not find container \"fa96c67b575da8b32a91902eca0c2fdea9954fec81050b3dc444dec6c5bbc283\": container with ID starting with fa96c67b575da8b32a91902eca0c2fdea9954fec81050b3dc444dec6c5bbc283 not found: ID does not exist" Oct 30 16:49:11.793731 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.793664 5052 scope.go:122] "RemoveContainer" containerID="95c15eab4330dec0ed5761be3ae5327a9fe786d4cb463872766b7d5204f8fd78" Oct 30 16:49:11.803929 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.803910 5052 scope.go:122] "RemoveContainer" containerID="95c15eab4330dec0ed5761be3ae5327a9fe786d4cb463872766b7d5204f8fd78" Oct 30 16:49:11.804195 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:49:11.804177 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95c15eab4330dec0ed5761be3ae5327a9fe786d4cb463872766b7d5204f8fd78\": container with ID starting with 95c15eab4330dec0ed5761be3ae5327a9fe786d4cb463872766b7d5204f8fd78 not found: ID does not exist" containerID="95c15eab4330dec0ed5761be3ae5327a9fe786d4cb463872766b7d5204f8fd78" Oct 30 16:49:11.804245 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.804204 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95c15eab4330dec0ed5761be3ae5327a9fe786d4cb463872766b7d5204f8fd78"} err="failed to get container status \"95c15eab4330dec0ed5761be3ae5327a9fe786d4cb463872766b7d5204f8fd78\": rpc error: code = NotFound desc = could not find container \"95c15eab4330dec0ed5761be3ae5327a9fe786d4cb463872766b7d5204f8fd78\": container with ID starting with 95c15eab4330dec0ed5761be3ae5327a9fe786d4cb463872766b7d5204f8fd78 not found: ID does not exist" Oct 30 16:49:11.804245 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.804221 5052 scope.go:122] "RemoveContainer" containerID="653b37f0a36ca625c0e710ae2c5299203f8da7df81f26a3b6513447f5ef2d81f" Oct 30 16:49:11.814501 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.814479 5052 scope.go:122] "RemoveContainer" containerID="653b37f0a36ca625c0e710ae2c5299203f8da7df81f26a3b6513447f5ef2d81f" Oct 30 16:49:11.814806 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:49:11.814781 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"653b37f0a36ca625c0e710ae2c5299203f8da7df81f26a3b6513447f5ef2d81f\": container with ID starting with 653b37f0a36ca625c0e710ae2c5299203f8da7df81f26a3b6513447f5ef2d81f not found: ID does not exist" containerID="653b37f0a36ca625c0e710ae2c5299203f8da7df81f26a3b6513447f5ef2d81f" Oct 30 16:49:11.814880 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.814806 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"653b37f0a36ca625c0e710ae2c5299203f8da7df81f26a3b6513447f5ef2d81f"} err="failed to get container status \"653b37f0a36ca625c0e710ae2c5299203f8da7df81f26a3b6513447f5ef2d81f\": rpc error: code = NotFound desc = could not find container \"653b37f0a36ca625c0e710ae2c5299203f8da7df81f26a3b6513447f5ef2d81f\": container with ID starting with 653b37f0a36ca625c0e710ae2c5299203f8da7df81f26a3b6513447f5ef2d81f not found: ID does not exist" Oct 30 16:49:11.869481 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.869445 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-98gfq"] Oct 30 16:49:11.875143 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.875120 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-98gfq"] Oct 30 16:49:11.910003 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.909977 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-92x47"] Oct 30 16:49:11.915472 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.915443 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-92x47"] Oct 30 16:49:11.956649 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.956625 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-v9r87"] Oct 30 16:49:11.965092 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:11.965069 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-v9r87"] Oct 30 16:49:12.027799 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:12.027712 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2slrc"] Oct 30 16:49:12.031546 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:12.031519 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2slrc"] Oct 30 16:49:12.599597 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:12.599541 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20129ef0-4c2c-498f-9427-985db412bb2e" path="/var/lib/kubelet/pods/20129ef0-4c2c-498f-9427-985db412bb2e/volumes" Oct 30 16:49:12.600316 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:12.600265 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39ebd9d1-3580-45a0-84ca-de3126fe5224" path="/var/lib/kubelet/pods/39ebd9d1-3580-45a0-84ca-de3126fe5224/volumes" Oct 30 16:49:12.600850 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:12.600829 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69a65b89-4d81-476b-b4b7-06736537fa4a" path="/var/lib/kubelet/pods/69a65b89-4d81-476b-b4b7-06736537fa4a/volumes" Oct 30 16:49:12.601673 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:12.601652 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c22edc73-463a-4439-92aa-601b4cddd714" path="/var/lib/kubelet/pods/c22edc73-463a-4439-92aa-601b4cddd714/volumes" Oct 30 16:49:12.627467 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:12.627440 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b315d4-84cb4fb687-9466n"] Oct 30 16:49:13.126522 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:13.126491 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-407eab-86959ffcb8-dwfh8"] Oct 30 16:49:13.127333 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:13.127316 5052 cpu_manager.go:414] "containerMap: removing container" podUID="20129ef0-4c2c-498f-9427-985db412bb2e" containerName="istio-proxy" Oct 30 16:49:13.127401 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:13.127335 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20129ef0-4c2c-498f-9427-985db412bb2e" containerName="istio-proxy" Oct 30 16:49:13.127401 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:13.127365 5052 cpu_manager.go:414] "containerMap: removing container" podUID="c22edc73-463a-4439-92aa-601b4cddd714" containerName="istio-proxy" Oct 30 16:49:13.127401 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:13.127371 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c22edc73-463a-4439-92aa-601b4cddd714" containerName="istio-proxy" Oct 30 16:49:13.127401 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:13.127390 5052 cpu_manager.go:414] "containerMap: removing container" podUID="69a65b89-4d81-476b-b4b7-06736537fa4a" containerName="istio-proxy" Oct 30 16:49:13.127401 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:13.127394 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69a65b89-4d81-476b-b4b7-06736537fa4a" containerName="istio-proxy" Oct 30 16:49:13.127555 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:13.127408 5052 cpu_manager.go:414] "containerMap: removing container" podUID="39ebd9d1-3580-45a0-84ca-de3126fe5224" containerName="istio-proxy" Oct 30 16:49:13.127555 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:13.127413 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39ebd9d1-3580-45a0-84ca-de3126fe5224" containerName="istio-proxy" Oct 30 16:49:13.127555 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:13.127526 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="20129ef0-4c2c-498f-9427-985db412bb2e" containerName="istio-proxy" Oct 30 16:49:13.127555 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:13.127539 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="c22edc73-463a-4439-92aa-601b4cddd714" containerName="istio-proxy" Oct 30 16:49:13.127555 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:13.127547 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="69a65b89-4d81-476b-b4b7-06736537fa4a" containerName="istio-proxy" Oct 30 16:49:13.127555 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:13.127554 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="39ebd9d1-3580-45a0-84ca-de3126fe5224" containerName="istio-proxy" Oct 30 16:49:13.133222 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:13.133202 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-407eab-86959ffcb8-dwfh8" Oct 30 16:49:13.151598 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:13.151557 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-407eab-serving-cert\"" Oct 30 16:49:13.154895 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:13.154865 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-407eab-86959ffcb8-dwfh8"] Oct 30 16:49:13.273036 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:13.272997 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf584334-297c-4316-a308-fdbd217e5011-maas-api-tls\") pod \"maas-api-e2e-api-b-407eab-86959ffcb8-dwfh8\" (UID: \"bf584334-297c-4316-a308-fdbd217e5011\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-407eab-86959ffcb8-dwfh8" Oct 30 16:49:13.273226 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:13.273053 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf584334-297c-4316-a308-fdbd217e5011-openshift-service-ca\") pod \"maas-api-e2e-api-b-407eab-86959ffcb8-dwfh8\" (UID: \"bf584334-297c-4316-a308-fdbd217e5011\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-407eab-86959ffcb8-dwfh8" Oct 30 16:49:13.273283 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:13.273238 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf584334-297c-4316-a308-fdbd217e5011-metrics-certs\") pod \"maas-api-e2e-api-b-407eab-86959ffcb8-dwfh8\" (UID: \"bf584334-297c-4316-a308-fdbd217e5011\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-407eab-86959ffcb8-dwfh8" Oct 30 16:49:13.273364 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:13.273350 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-js9fp\" (UniqueName: \"kubernetes.io/projected/bf584334-297c-4316-a308-fdbd217e5011-kube-api-access-js9fp\") pod \"maas-api-e2e-api-b-407eab-86959ffcb8-dwfh8\" (UID: \"bf584334-297c-4316-a308-fdbd217e5011\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-407eab-86959ffcb8-dwfh8" Oct 30 16:49:13.381660 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:13.380000 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf584334-297c-4316-a308-fdbd217e5011-maas-api-tls\") pod \"maas-api-e2e-api-b-407eab-86959ffcb8-dwfh8\" (UID: \"bf584334-297c-4316-a308-fdbd217e5011\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-407eab-86959ffcb8-dwfh8" Oct 30 16:49:13.381660 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:13.380060 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf584334-297c-4316-a308-fdbd217e5011-openshift-service-ca\") pod \"maas-api-e2e-api-b-407eab-86959ffcb8-dwfh8\" (UID: \"bf584334-297c-4316-a308-fdbd217e5011\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-407eab-86959ffcb8-dwfh8" Oct 30 16:49:13.381660 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:13.380213 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf584334-297c-4316-a308-fdbd217e5011-metrics-certs\") pod \"maas-api-e2e-api-b-407eab-86959ffcb8-dwfh8\" (UID: \"bf584334-297c-4316-a308-fdbd217e5011\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-407eab-86959ffcb8-dwfh8" Oct 30 16:49:13.381660 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:13.380319 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-js9fp\" (UniqueName: \"kubernetes.io/projected/bf584334-297c-4316-a308-fdbd217e5011-kube-api-access-js9fp\") pod \"maas-api-e2e-api-b-407eab-86959ffcb8-dwfh8\" (UID: \"bf584334-297c-4316-a308-fdbd217e5011\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-407eab-86959ffcb8-dwfh8" Oct 30 16:49:13.382108 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:13.381856 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf584334-297c-4316-a308-fdbd217e5011-openshift-service-ca\") pod \"maas-api-e2e-api-b-407eab-86959ffcb8-dwfh8\" (UID: \"bf584334-297c-4316-a308-fdbd217e5011\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-407eab-86959ffcb8-dwfh8" Oct 30 16:49:13.393873 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:13.391160 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf584334-297c-4316-a308-fdbd217e5011-maas-api-tls\") pod \"maas-api-e2e-api-b-407eab-86959ffcb8-dwfh8\" (UID: \"bf584334-297c-4316-a308-fdbd217e5011\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-407eab-86959ffcb8-dwfh8" Oct 30 16:49:13.404635 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:13.402634 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf584334-297c-4316-a308-fdbd217e5011-metrics-certs\") pod \"maas-api-e2e-api-b-407eab-86959ffcb8-dwfh8\" (UID: \"bf584334-297c-4316-a308-fdbd217e5011\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-407eab-86959ffcb8-dwfh8" Oct 30 16:49:13.412083 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:13.412052 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-js9fp\" (UniqueName: \"kubernetes.io/projected/bf584334-297c-4316-a308-fdbd217e5011-kube-api-access-js9fp\") pod \"maas-api-e2e-api-b-407eab-86959ffcb8-dwfh8\" (UID: \"bf584334-297c-4316-a308-fdbd217e5011\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-407eab-86959ffcb8-dwfh8" Oct 30 16:49:13.455966 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:13.455687 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-407eab-86959ffcb8-dwfh8" Oct 30 16:49:13.921756 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:13.921727 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-407eab-86959ffcb8-dwfh8"] Oct 30 16:49:13.922713 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:49:13.922682 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf584334_297c_4316_a308_fdbd217e5011.slice/crio-2b33847dc0e8d96a5b38a44e1c903b9298d72a477dd57244401ccead6b3bf59c WatchSource:0}: Error finding container 2b33847dc0e8d96a5b38a44e1c903b9298d72a477dd57244401ccead6b3bf59c: Status 404 returned error can't find the container with id 2b33847dc0e8d96a5b38a44e1c903b9298d72a477dd57244401ccead6b3bf59c Oct 30 16:49:14.105604 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:14.104138 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cdwcj"] Oct 30 16:49:14.796141 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:14.796099 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-407eab-86959ffcb8-dwfh8" event={"ID":"bf584334-297c-4316-a308-fdbd217e5011","Type":"ContainerStarted","Data":"a0fe8427ce43831b4cf8763a54c1a543266c101e8907d3659baa17426dbab54f"} Oct 30 16:49:14.796325 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:14.796153 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-407eab-86959ffcb8-dwfh8" event={"ID":"bf584334-297c-4316-a308-fdbd217e5011","Type":"ContainerStarted","Data":"2b33847dc0e8d96a5b38a44e1c903b9298d72a477dd57244401ccead6b3bf59c"} Oct 30 16:49:14.796325 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:14.796298 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-407eab-86959ffcb8-dwfh8" Oct 30 16:49:14.918445 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:14.918376 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-407eab-86959ffcb8-dwfh8" podStartSLOduration=1.564935237 podStartE2EDuration="1.918348958s" totalImagesPullingTime="353.413721ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:49:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:49:14.857544901 +0000 UTC m=+1710.384880145" watchObservedRunningTime="2026-10-30 16:49:14.918348958 +0000 UTC m=+1710.445684192" Oct 30 16:49:17.627769 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:17.627715 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b315d4-84cb4fb687-9466n" podUID="a626d0b1-3ca3-49a1-ab32-dde876466afe" containerName="payload-pre-processing" containerID="cri-o://7b70434e3d66c791346ec315a70ca9e411d9ffc4c8c0a71955a03eae2604d7da" gracePeriod=25 Oct 30 16:49:17.836032 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:17.835992 5052 generic.go:356] "Generic (PLEG): container finished" podID="a626d0b1-3ca3-49a1-ab32-dde876466afe" containerID="7b70434e3d66c791346ec315a70ca9e411d9ffc4c8c0a71955a03eae2604d7da" exitCode=0 Oct 30 16:49:17.836311 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:17.836289 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b315d4-84cb4fb687-9466n" event={"ID":"a626d0b1-3ca3-49a1-ab32-dde876466afe","Type":"ContainerDied","Data":"7b70434e3d66c791346ec315a70ca9e411d9ffc4c8c0a71955a03eae2604d7da"} Oct 30 16:49:17.934003 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:17.933974 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b315d4-84cb4fb687-9466n" Oct 30 16:49:17.974462 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:17.974427 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a626d0b1-3ca3-49a1-ab32-dde876466afe-kube-api-access-27c2v\" (UniqueName: \"kubernetes.io/projected/a626d0b1-3ca3-49a1-ab32-dde876466afe-kube-api-access-27c2v\") pod \"a626d0b1-3ca3-49a1-ab32-dde876466afe\" (UID: \"a626d0b1-3ca3-49a1-ab32-dde876466afe\") " Oct 30 16:49:17.974625 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:17.974561 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a626d0b1-3ca3-49a1-ab32-dde876466afe-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a626d0b1-3ca3-49a1-ab32-dde876466afe-plugins-config-volume\") pod \"a626d0b1-3ca3-49a1-ab32-dde876466afe\" (UID: \"a626d0b1-3ca3-49a1-ab32-dde876466afe\") " Oct 30 16:49:17.974916 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:17.974895 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a626d0b1-3ca3-49a1-ab32-dde876466afe-plugins-config-volume" pod "a626d0b1-3ca3-49a1-ab32-dde876466afe" (UID: "a626d0b1-3ca3-49a1-ab32-dde876466afe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:49:17.975334 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:17.975315 5052 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a626d0b1-3ca3-49a1-ab32-dde876466afe-plugins-config-volume\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:49:17.976824 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:17.976700 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a626d0b1-3ca3-49a1-ab32-dde876466afe-kube-api-access-27c2v" pod "a626d0b1-3ca3-49a1-ab32-dde876466afe" (UID: "a626d0b1-3ca3-49a1-ab32-dde876466afe"). InnerVolumeSpecName "kube-api-access-27c2v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:49:18.027497 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:18.027461 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-3ee682-openshift-default-5f7887cb79-6tsdx"] Oct 30 16:49:18.028794 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:18.028771 5052 cpu_manager.go:414] "containerMap: removing container" podUID="a626d0b1-3ca3-49a1-ab32-dde876466afe" containerName="payload-pre-processing" Oct 30 16:49:18.028901 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:18.028796 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a626d0b1-3ca3-49a1-ab32-dde876466afe" containerName="payload-pre-processing" Oct 30 16:49:18.029157 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:18.029138 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="a626d0b1-3ca3-49a1-ab32-dde876466afe" containerName="payload-pre-processing" Oct 30 16:49:18.033565 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:18.033544 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3ee682-openshift-default-5f7887cb79-6tsdx" Oct 30 16:49:18.059441 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:18.053414 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-3ee682-openshift-default-dockercfg-7mxxr\"" Oct 30 16:49:18.063392 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:18.063218 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-3ee682-openshift-default-5f7887cb79-6tsdx"] Oct 30 16:49:18.076648 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:18.076613 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe851cb7-f6e1-4baf-b75e-69018da63e35-istio-token\") pod \"e2e-inf-b-3ee682-openshift-default-5f7887cb79-6tsdx\" (UID: \"fe851cb7-f6e1-4baf-b75e-69018da63e35\") " pod="openshift-ingress/e2e-inf-b-3ee682-openshift-default-5f7887cb79-6tsdx" Oct 30 16:49:18.076767 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:18.076679 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe851cb7-f6e1-4baf-b75e-69018da63e35-istio-data\") pod \"e2e-inf-b-3ee682-openshift-default-5f7887cb79-6tsdx\" (UID: \"fe851cb7-f6e1-4baf-b75e-69018da63e35\") " pod="openshift-ingress/e2e-inf-b-3ee682-openshift-default-5f7887cb79-6tsdx" Oct 30 16:49:18.076767 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:18.076709 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe851cb7-f6e1-4baf-b75e-69018da63e35-workload-certs\") pod \"e2e-inf-b-3ee682-openshift-default-5f7887cb79-6tsdx\" (UID: \"fe851cb7-f6e1-4baf-b75e-69018da63e35\") " pod="openshift-ingress/e2e-inf-b-3ee682-openshift-default-5f7887cb79-6tsdx" Oct 30 16:49:18.076767 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:18.076736 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe851cb7-f6e1-4baf-b75e-69018da63e35-istiod-ca-cert\") pod \"e2e-inf-b-3ee682-openshift-default-5f7887cb79-6tsdx\" (UID: \"fe851cb7-f6e1-4baf-b75e-69018da63e35\") " pod="openshift-ingress/e2e-inf-b-3ee682-openshift-default-5f7887cb79-6tsdx" Oct 30 16:49:18.077329 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:18.076957 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe851cb7-f6e1-4baf-b75e-69018da63e35-credential-socket\") pod \"e2e-inf-b-3ee682-openshift-default-5f7887cb79-6tsdx\" (UID: \"fe851cb7-f6e1-4baf-b75e-69018da63e35\") " pod="openshift-ingress/e2e-inf-b-3ee682-openshift-default-5f7887cb79-6tsdx" Oct 30 16:49:18.077329 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:18.077056 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8hgk\" (UniqueName: \"kubernetes.io/projected/fe851cb7-f6e1-4baf-b75e-69018da63e35-kube-api-access-t8hgk\") pod \"e2e-inf-b-3ee682-openshift-default-5f7887cb79-6tsdx\" (UID: \"fe851cb7-f6e1-4baf-b75e-69018da63e35\") " pod="openshift-ingress/e2e-inf-b-3ee682-openshift-default-5f7887cb79-6tsdx" Oct 30 16:49:18.077329 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:18.077102 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe851cb7-f6e1-4baf-b75e-69018da63e35-istio-envoy\") pod \"e2e-inf-b-3ee682-openshift-default-5f7887cb79-6tsdx\" (UID: \"fe851cb7-f6e1-4baf-b75e-69018da63e35\") " pod="openshift-ingress/e2e-inf-b-3ee682-openshift-default-5f7887cb79-6tsdx" Oct 30 16:49:18.077329 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:18.077126 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe851cb7-f6e1-4baf-b75e-69018da63e35-istio-podinfo\") pod \"e2e-inf-b-3ee682-openshift-default-5f7887cb79-6tsdx\" (UID: \"fe851cb7-f6e1-4baf-b75e-69018da63e35\") " pod="openshift-ingress/e2e-inf-b-3ee682-openshift-default-5f7887cb79-6tsdx" Oct 30 16:49:18.077329 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:18.077262 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe851cb7-f6e1-4baf-b75e-69018da63e35-workload-socket\") pod \"e2e-inf-b-3ee682-openshift-default-5f7887cb79-6tsdx\" (UID: \"fe851cb7-f6e1-4baf-b75e-69018da63e35\") " pod="openshift-ingress/e2e-inf-b-3ee682-openshift-default-5f7887cb79-6tsdx" Oct 30 16:49:18.077564 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:18.077521 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27c2v\" (UniqueName: \"kubernetes.io/projected/a626d0b1-3ca3-49a1-ab32-dde876466afe-kube-api-access-27c2v\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:49:18.199604 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:18.178878 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe851cb7-f6e1-4baf-b75e-69018da63e35-credential-socket\") pod \"e2e-inf-b-3ee682-openshift-default-5f7887cb79-6tsdx\" (UID: \"fe851cb7-f6e1-4baf-b75e-69018da63e35\") " pod="openshift-ingress/e2e-inf-b-3ee682-openshift-default-5f7887cb79-6tsdx" Oct 30 16:49:18.200056 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:18.200019 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8hgk\" (UniqueName: \"kubernetes.io/projected/fe851cb7-f6e1-4baf-b75e-69018da63e35-kube-api-access-t8hgk\") pod \"e2e-inf-b-3ee682-openshift-default-5f7887cb79-6tsdx\" (UID: \"fe851cb7-f6e1-4baf-b75e-69018da63e35\") " pod="openshift-ingress/e2e-inf-b-3ee682-openshift-default-5f7887cb79-6tsdx" Oct 30 16:49:18.200294 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:18.200277 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe851cb7-f6e1-4baf-b75e-69018da63e35-istio-envoy\") pod \"e2e-inf-b-3ee682-openshift-default-5f7887cb79-6tsdx\" (UID: \"fe851cb7-f6e1-4baf-b75e-69018da63e35\") " pod="openshift-ingress/e2e-inf-b-3ee682-openshift-default-5f7887cb79-6tsdx" Oct 30 16:49:18.200428 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:18.200410 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe851cb7-f6e1-4baf-b75e-69018da63e35-istio-podinfo\") pod \"e2e-inf-b-3ee682-openshift-default-5f7887cb79-6tsdx\" (UID: \"fe851cb7-f6e1-4baf-b75e-69018da63e35\") " pod="openshift-ingress/e2e-inf-b-3ee682-openshift-default-5f7887cb79-6tsdx" Oct 30 16:49:18.200566 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:18.200548 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe851cb7-f6e1-4baf-b75e-69018da63e35-workload-socket\") pod \"e2e-inf-b-3ee682-openshift-default-5f7887cb79-6tsdx\" (UID: \"fe851cb7-f6e1-4baf-b75e-69018da63e35\") " pod="openshift-ingress/e2e-inf-b-3ee682-openshift-default-5f7887cb79-6tsdx" Oct 30 16:49:18.204215 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:18.204183 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe851cb7-f6e1-4baf-b75e-69018da63e35-istio-token\") pod \"e2e-inf-b-3ee682-openshift-default-5f7887cb79-6tsdx\" (UID: \"fe851cb7-f6e1-4baf-b75e-69018da63e35\") " pod="openshift-ingress/e2e-inf-b-3ee682-openshift-default-5f7887cb79-6tsdx" Oct 30 16:49:18.204425 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:18.204408 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe851cb7-f6e1-4baf-b75e-69018da63e35-istio-data\") pod \"e2e-inf-b-3ee682-openshift-default-5f7887cb79-6tsdx\" (UID: \"fe851cb7-f6e1-4baf-b75e-69018da63e35\") " pod="openshift-ingress/e2e-inf-b-3ee682-openshift-default-5f7887cb79-6tsdx" Oct 30 16:49:18.218695 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:18.218659 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe851cb7-f6e1-4baf-b75e-69018da63e35-workload-certs\") pod \"e2e-inf-b-3ee682-openshift-default-5f7887cb79-6tsdx\" (UID: \"fe851cb7-f6e1-4baf-b75e-69018da63e35\") " pod="openshift-ingress/e2e-inf-b-3ee682-openshift-default-5f7887cb79-6tsdx" Oct 30 16:49:18.218913 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:18.218895 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe851cb7-f6e1-4baf-b75e-69018da63e35-istiod-ca-cert\") pod \"e2e-inf-b-3ee682-openshift-default-5f7887cb79-6tsdx\" (UID: \"fe851cb7-f6e1-4baf-b75e-69018da63e35\") " pod="openshift-ingress/e2e-inf-b-3ee682-openshift-default-5f7887cb79-6tsdx" Oct 30 16:49:18.220020 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:18.219993 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe851cb7-f6e1-4baf-b75e-69018da63e35-istiod-ca-cert\") pod \"e2e-inf-b-3ee682-openshift-default-5f7887cb79-6tsdx\" (UID: \"fe851cb7-f6e1-4baf-b75e-69018da63e35\") " pod="openshift-ingress/e2e-inf-b-3ee682-openshift-default-5f7887cb79-6tsdx" Oct 30 16:49:18.226608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:18.179543 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe851cb7-f6e1-4baf-b75e-69018da63e35-credential-socket\") pod \"e2e-inf-b-3ee682-openshift-default-5f7887cb79-6tsdx\" (UID: \"fe851cb7-f6e1-4baf-b75e-69018da63e35\") " pod="openshift-ingress/e2e-inf-b-3ee682-openshift-default-5f7887cb79-6tsdx" Oct 30 16:49:18.227077 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:18.227054 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe851cb7-f6e1-4baf-b75e-69018da63e35-istio-data\") pod \"e2e-inf-b-3ee682-openshift-default-5f7887cb79-6tsdx\" (UID: \"fe851cb7-f6e1-4baf-b75e-69018da63e35\") " pod="openshift-ingress/e2e-inf-b-3ee682-openshift-default-5f7887cb79-6tsdx" Oct 30 16:49:18.232055 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:18.227426 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe851cb7-f6e1-4baf-b75e-69018da63e35-workload-certs\") pod \"e2e-inf-b-3ee682-openshift-default-5f7887cb79-6tsdx\" (UID: \"fe851cb7-f6e1-4baf-b75e-69018da63e35\") " pod="openshift-ingress/e2e-inf-b-3ee682-openshift-default-5f7887cb79-6tsdx" Oct 30 16:49:18.232509 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:18.232483 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe851cb7-f6e1-4baf-b75e-69018da63e35-workload-socket\") pod \"e2e-inf-b-3ee682-openshift-default-5f7887cb79-6tsdx\" (UID: \"fe851cb7-f6e1-4baf-b75e-69018da63e35\") " pod="openshift-ingress/e2e-inf-b-3ee682-openshift-default-5f7887cb79-6tsdx" Oct 30 16:49:18.242654 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:18.236126 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe851cb7-f6e1-4baf-b75e-69018da63e35-istio-podinfo\") pod \"e2e-inf-b-3ee682-openshift-default-5f7887cb79-6tsdx\" (UID: \"fe851cb7-f6e1-4baf-b75e-69018da63e35\") " pod="openshift-ingress/e2e-inf-b-3ee682-openshift-default-5f7887cb79-6tsdx" Oct 30 16:49:18.316937 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:18.314904 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe851cb7-f6e1-4baf-b75e-69018da63e35-istio-envoy\") pod \"e2e-inf-b-3ee682-openshift-default-5f7887cb79-6tsdx\" (UID: \"fe851cb7-f6e1-4baf-b75e-69018da63e35\") " pod="openshift-ingress/e2e-inf-b-3ee682-openshift-default-5f7887cb79-6tsdx" Oct 30 16:49:18.319702 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:18.319668 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe851cb7-f6e1-4baf-b75e-69018da63e35-istio-token\") pod \"e2e-inf-b-3ee682-openshift-default-5f7887cb79-6tsdx\" (UID: \"fe851cb7-f6e1-4baf-b75e-69018da63e35\") " pod="openshift-ingress/e2e-inf-b-3ee682-openshift-default-5f7887cb79-6tsdx" Oct 30 16:49:18.343567 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:18.343532 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8hgk\" (UniqueName: \"kubernetes.io/projected/fe851cb7-f6e1-4baf-b75e-69018da63e35-kube-api-access-t8hgk\") pod \"e2e-inf-b-3ee682-openshift-default-5f7887cb79-6tsdx\" (UID: \"fe851cb7-f6e1-4baf-b75e-69018da63e35\") " pod="openshift-ingress/e2e-inf-b-3ee682-openshift-default-5f7887cb79-6tsdx" Oct 30 16:49:18.366976 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:18.366941 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3ee682-openshift-default-5f7887cb79-6tsdx" Oct 30 16:49:18.845726 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:49:18.845645 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 16:49:18.848165 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:18.848142 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b315d4-84cb4fb687-9466n" Oct 30 16:49:18.848313 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:18.848167 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b315d4-84cb4fb687-9466n" event={"ID":"a626d0b1-3ca3-49a1-ab32-dde876466afe","Type":"ContainerDied","Data":"d5082b56c17936408caf2205792229686844f2df1adfc55e6ca28fd0af7cad84"} Oct 30 16:49:18.848313 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:18.848246 5052 scope.go:122] "RemoveContainer" containerID="7b70434e3d66c791346ec315a70ca9e411d9ffc4c8c0a71955a03eae2604d7da" Oct 30 16:49:18.892787 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:49:18.892756 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe851cb7_f6e1_4baf_b75e_69018da63e35.slice/crio-cb4e5c5f06e0cd8d4a7c1809a3fd94a4a6f954b11430107aedbdaa11d7963ddf WatchSource:0}: Error finding container cb4e5c5f06e0cd8d4a7c1809a3fd94a4a6f954b11430107aedbdaa11d7963ddf: Status 404 returned error can't find the container with id cb4e5c5f06e0cd8d4a7c1809a3fd94a4a6f954b11430107aedbdaa11d7963ddf Oct 30 16:49:18.896400 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:18.896366 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:49:18.896507 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:18.896490 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:49:18.896547 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:18.896535 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:49:18.898314 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:18.898291 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-3ee682-openshift-default-5f7887cb79-6tsdx"] Oct 30 16:49:18.905622 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:18.905599 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b315d4-84cb4fb687-9466n"] Oct 30 16:49:18.910288 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:18.910266 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b315d4-84cb4fb687-9466n"] Oct 30 16:49:19.859362 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:19.859330 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3ee682-openshift-default-5f7887cb79-6tsdx" event={"ID":"fe851cb7-f6e1-4baf-b75e-69018da63e35","Type":"ContainerStarted","Data":"2e130b1db7c8d1085dd61c563edc847dfa62b1e77499d4cdd0050cc8604b1682"} Oct 30 16:49:19.859777 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:19.859411 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3ee682-openshift-default-5f7887cb79-6tsdx" event={"ID":"fe851cb7-f6e1-4baf-b75e-69018da63e35","Type":"ContainerStarted","Data":"cb4e5c5f06e0cd8d4a7c1809a3fd94a4a6f954b11430107aedbdaa11d7963ddf"} Oct 30 16:49:19.925059 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:19.925002 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-3ee682-openshift-default-5f7887cb79-6tsdx" podStartSLOduration=2.924984566 podStartE2EDuration="2.924984566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:49:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:49:19.921925346 +0000 UTC m=+1715.449260565" watchObservedRunningTime="2026-10-30 16:49:19.924984566 +0000 UTC m=+1715.452319785" Oct 30 16:49:20.367963 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:20.367927 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-3ee682-openshift-default-5f7887cb79-6tsdx" Oct 30 16:49:20.374035 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:20.374012 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-3ee682-openshift-default-5f7887cb79-6tsdx" Oct 30 16:49:20.601308 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:20.601276 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a626d0b1-3ca3-49a1-ab32-dde876466afe" path="/var/lib/kubelet/pods/a626d0b1-3ca3-49a1-ab32-dde876466afe/volumes" Oct 30 16:49:20.870609 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:20.868800 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-407eab-86959ffcb8-dwfh8" Oct 30 16:49:20.907983 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:20.907909 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-3ee682-openshift-default-5f7887cb79-6tsdx" Oct 30 16:49:20.908681 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:20.908627 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-3ee682-openshift-default-5f7887cb79-6tsdx" Oct 30 16:49:29.518383 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:29.518350 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b8af660d-6697cd894d-nlcxt"] Oct 30 16:49:29.523897 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:29.523875 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b8af660d-6697cd894d-nlcxt" Oct 30 16:49:29.547720 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:29.547696 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-b8af660d-serving-cert\"" Oct 30 16:49:29.552335 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:29.552309 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b8af660d-6697cd894d-nlcxt"] Oct 30 16:49:29.698300 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:29.698262 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xbnr\" (UniqueName: \"kubernetes.io/projected/2c10ebe7-ea42-47e1-b384-03dedbe51daa-kube-api-access-8xbnr\") pod \"maas-api-e2e-mt-b8af660d-6697cd894d-nlcxt\" (UID: \"2c10ebe7-ea42-47e1-b384-03dedbe51daa\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b8af660d-6697cd894d-nlcxt" Oct 30 16:49:29.698536 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:29.698511 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c10ebe7-ea42-47e1-b384-03dedbe51daa-openshift-service-ca\") pod \"maas-api-e2e-mt-b8af660d-6697cd894d-nlcxt\" (UID: \"2c10ebe7-ea42-47e1-b384-03dedbe51daa\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b8af660d-6697cd894d-nlcxt" Oct 30 16:49:29.698623 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:29.698556 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c10ebe7-ea42-47e1-b384-03dedbe51daa-metrics-certs\") pod \"maas-api-e2e-mt-b8af660d-6697cd894d-nlcxt\" (UID: \"2c10ebe7-ea42-47e1-b384-03dedbe51daa\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b8af660d-6697cd894d-nlcxt" Oct 30 16:49:29.698940 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:29.698919 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c10ebe7-ea42-47e1-b384-03dedbe51daa-maas-api-tls\") pod \"maas-api-e2e-mt-b8af660d-6697cd894d-nlcxt\" (UID: \"2c10ebe7-ea42-47e1-b384-03dedbe51daa\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b8af660d-6697cd894d-nlcxt" Oct 30 16:49:29.802127 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:29.802024 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c10ebe7-ea42-47e1-b384-03dedbe51daa-openshift-service-ca\") pod \"maas-api-e2e-mt-b8af660d-6697cd894d-nlcxt\" (UID: \"2c10ebe7-ea42-47e1-b384-03dedbe51daa\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b8af660d-6697cd894d-nlcxt" Oct 30 16:49:29.802386 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:29.802364 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c10ebe7-ea42-47e1-b384-03dedbe51daa-metrics-certs\") pod \"maas-api-e2e-mt-b8af660d-6697cd894d-nlcxt\" (UID: \"2c10ebe7-ea42-47e1-b384-03dedbe51daa\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b8af660d-6697cd894d-nlcxt" Oct 30 16:49:29.802780 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:29.802753 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c10ebe7-ea42-47e1-b384-03dedbe51daa-maas-api-tls\") pod \"maas-api-e2e-mt-b8af660d-6697cd894d-nlcxt\" (UID: \"2c10ebe7-ea42-47e1-b384-03dedbe51daa\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b8af660d-6697cd894d-nlcxt" Oct 30 16:49:29.803043 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:29.803022 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xbnr\" (UniqueName: \"kubernetes.io/projected/2c10ebe7-ea42-47e1-b384-03dedbe51daa-kube-api-access-8xbnr\") pod \"maas-api-e2e-mt-b8af660d-6697cd894d-nlcxt\" (UID: \"2c10ebe7-ea42-47e1-b384-03dedbe51daa\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b8af660d-6697cd894d-nlcxt" Oct 30 16:49:29.804981 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:29.804957 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c10ebe7-ea42-47e1-b384-03dedbe51daa-openshift-service-ca\") pod \"maas-api-e2e-mt-b8af660d-6697cd894d-nlcxt\" (UID: \"2c10ebe7-ea42-47e1-b384-03dedbe51daa\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b8af660d-6697cd894d-nlcxt" Oct 30 16:49:29.833432 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:29.833394 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c10ebe7-ea42-47e1-b384-03dedbe51daa-maas-api-tls\") pod \"maas-api-e2e-mt-b8af660d-6697cd894d-nlcxt\" (UID: \"2c10ebe7-ea42-47e1-b384-03dedbe51daa\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b8af660d-6697cd894d-nlcxt" Oct 30 16:49:29.833820 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:29.833790 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c10ebe7-ea42-47e1-b384-03dedbe51daa-metrics-certs\") pod \"maas-api-e2e-mt-b8af660d-6697cd894d-nlcxt\" (UID: \"2c10ebe7-ea42-47e1-b384-03dedbe51daa\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b8af660d-6697cd894d-nlcxt" Oct 30 16:49:29.865096 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:29.861173 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xbnr\" (UniqueName: \"kubernetes.io/projected/2c10ebe7-ea42-47e1-b384-03dedbe51daa-kube-api-access-8xbnr\") pod \"maas-api-e2e-mt-b8af660d-6697cd894d-nlcxt\" (UID: \"2c10ebe7-ea42-47e1-b384-03dedbe51daa\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b8af660d-6697cd894d-nlcxt" Oct 30 16:49:30.139717 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:30.139635 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b8af660d-6697cd894d-nlcxt" Oct 30 16:49:30.473396 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:49:30.473365 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c10ebe7_ea42_47e1_b384_03dedbe51daa.slice/crio-53d4fcd0df277579806f97c17383b86e6270aec0f07094688062b6422922be4b WatchSource:0}: Error finding container 53d4fcd0df277579806f97c17383b86e6270aec0f07094688062b6422922be4b: Status 404 returned error can't find the container with id 53d4fcd0df277579806f97c17383b86e6270aec0f07094688062b6422922be4b Oct 30 16:49:30.473529 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:30.473506 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b8af660d-6697cd894d-nlcxt"] Oct 30 16:49:30.997037 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:30.996940 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b8af660d-6697cd894d-nlcxt" event={"ID":"2c10ebe7-ea42-47e1-b384-03dedbe51daa","Type":"ContainerStarted","Data":"53d4fcd0df277579806f97c17383b86e6270aec0f07094688062b6422922be4b"} Oct 30 16:49:32.007018 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:32.006979 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b8af660d-6697cd894d-nlcxt" event={"ID":"2c10ebe7-ea42-47e1-b384-03dedbe51daa","Type":"ContainerStarted","Data":"ee24d3dd37eb61d18960faa6453e3a9e8ec90d657e7c0438418d4e1f1c2535ef"} Oct 30 16:49:32.007419 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:32.007118 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b8af660d-6697cd894d-nlcxt" Oct 30 16:49:32.056298 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:32.056250 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b8af660d-6697cd894d-nlcxt" podStartSLOduration=2.567472033 podStartE2EDuration="3.056237063s" totalImagesPullingTime="488.76503ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:49:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:49:32.054052117 +0000 UTC m=+1727.581387337" watchObservedRunningTime="2026-10-30 16:49:32.056237063 +0000 UTC m=+1727.583572283" Oct 30 16:49:38.033999 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:38.033970 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b8af660d-6697cd894d-nlcxt" Oct 30 16:49:41.135604 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:41.134550 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d693d0/distinct-2-w6-d693d0-kserve-6498fdc474-dpbjj" podUID="4b98b8ff-72cd-4ddb-a4b8-67078162e611" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:49:41.223603 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:41.218189 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-750dcb/distinct-w2-750dcb-kserve-86bbdfcbb-bpn77" podUID="c45190af-d795-4e13-8f2f-670b2c597658" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded" Oct 30 16:49:41.235606 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:41.234688 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/distinct-2-w4-263f00-kserve-8464897cf5-q8ck8" podUID="15ed43ef-5cdd-489a-a70e-063239822ff3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:49:41.279610 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:41.278663 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx" podUID="bd662f3a-659f-477c-b8d5-94c64288c0c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 16:49:41.662812 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:41.662761 5052 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-t8b5p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:49:41.663134 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:41.663102 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-t8b5p" podUID="fcc7c9e1-99ea-4b97-9e95-2e23a00ac096" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:49:41.825009 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:41.824889 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/distinct-w5-967fac-kserve-69cbcccd8b-b85bn" podUID="8f6b9354-378f-4451-9ff7-be7f7fec12f2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:49:42.136040 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:42.135930 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d693d0/distinct-2-w6-d693d0-kserve-6498fdc474-dpbjj" podUID="4b98b8ff-72cd-4ddb-a4b8-67078162e611" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:49:42.137404 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:42.137377 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d693d0/distinct-2-w6-d693d0-kserve-6498fdc474-dpbjj" Oct 30 16:49:42.223760 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:42.223714 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-750dcb/distinct-w2-750dcb-kserve-86bbdfcbb-bpn77" podUID="c45190af-d795-4e13-8f2f-670b2c597658" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:49:42.247606 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:42.224641 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-750dcb/distinct-w2-750dcb-kserve-86bbdfcbb-bpn77" Oct 30 16:49:42.291265 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:42.280330 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx" podUID="bd662f3a-659f-477c-b8d5-94c64288c0c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:49:42.291265 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:42.289481 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-967fac/distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx" Oct 30 16:49:42.797257 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:42.797225 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-750dcb/distinct-w2-750dcb-kserve-86bbdfcbb-bpn77" Oct 30 16:49:42.797460 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:42.797300 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d693d0/distinct-2-w6-d693d0-kserve-6498fdc474-dpbjj" Oct 30 16:49:42.797460 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:42.797333 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-967fac/distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx" Oct 30 16:49:46.067503 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:46.067405 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-ff1189-openshift-default-7d9f9f5c65-jjdbw"] Oct 30 16:49:46.080799 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:46.080771 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-ff1189-openshift-default-7d9f9f5c65-jjdbw"] Oct 30 16:49:46.080943 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:46.080900 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ff1189-openshift-default-7d9f9f5c65-jjdbw" Oct 30 16:49:46.097440 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:46.097398 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-ff1189-openshift-default-dockercfg-vqjsh\"" Oct 30 16:49:46.172095 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:46.172049 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba6262c9-b3ea-4c61-bb84-4f60954d02a0-istio-token\") pod \"e2e-inf-a-ff1189-openshift-default-7d9f9f5c65-jjdbw\" (UID: \"ba6262c9-b3ea-4c61-bb84-4f60954d02a0\") " pod="openshift-ingress/e2e-inf-a-ff1189-openshift-default-7d9f9f5c65-jjdbw" Oct 30 16:49:46.172272 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:46.172209 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba6262c9-b3ea-4c61-bb84-4f60954d02a0-istio-envoy\") pod \"e2e-inf-a-ff1189-openshift-default-7d9f9f5c65-jjdbw\" (UID: \"ba6262c9-b3ea-4c61-bb84-4f60954d02a0\") " pod="openshift-ingress/e2e-inf-a-ff1189-openshift-default-7d9f9f5c65-jjdbw" Oct 30 16:49:46.172272 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:46.172228 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba6262c9-b3ea-4c61-bb84-4f60954d02a0-istio-podinfo\") pod \"e2e-inf-a-ff1189-openshift-default-7d9f9f5c65-jjdbw\" (UID: \"ba6262c9-b3ea-4c61-bb84-4f60954d02a0\") " pod="openshift-ingress/e2e-inf-a-ff1189-openshift-default-7d9f9f5c65-jjdbw" Oct 30 16:49:46.172369 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:46.172333 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba6262c9-b3ea-4c61-bb84-4f60954d02a0-workload-certs\") pod \"e2e-inf-a-ff1189-openshift-default-7d9f9f5c65-jjdbw\" (UID: \"ba6262c9-b3ea-4c61-bb84-4f60954d02a0\") " pod="openshift-ingress/e2e-inf-a-ff1189-openshift-default-7d9f9f5c65-jjdbw" Oct 30 16:49:46.172369 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:46.172363 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba6262c9-b3ea-4c61-bb84-4f60954d02a0-workload-socket\") pod \"e2e-inf-a-ff1189-openshift-default-7d9f9f5c65-jjdbw\" (UID: \"ba6262c9-b3ea-4c61-bb84-4f60954d02a0\") " pod="openshift-ingress/e2e-inf-a-ff1189-openshift-default-7d9f9f5c65-jjdbw" Oct 30 16:49:46.172470 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:46.172389 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba6262c9-b3ea-4c61-bb84-4f60954d02a0-istio-data\") pod \"e2e-inf-a-ff1189-openshift-default-7d9f9f5c65-jjdbw\" (UID: \"ba6262c9-b3ea-4c61-bb84-4f60954d02a0\") " pod="openshift-ingress/e2e-inf-a-ff1189-openshift-default-7d9f9f5c65-jjdbw" Oct 30 16:49:46.172605 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:46.172566 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba6262c9-b3ea-4c61-bb84-4f60954d02a0-istiod-ca-cert\") pod \"e2e-inf-a-ff1189-openshift-default-7d9f9f5c65-jjdbw\" (UID: \"ba6262c9-b3ea-4c61-bb84-4f60954d02a0\") " pod="openshift-ingress/e2e-inf-a-ff1189-openshift-default-7d9f9f5c65-jjdbw" Oct 30 16:49:46.172688 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:46.172621 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba6262c9-b3ea-4c61-bb84-4f60954d02a0-credential-socket\") pod \"e2e-inf-a-ff1189-openshift-default-7d9f9f5c65-jjdbw\" (UID: \"ba6262c9-b3ea-4c61-bb84-4f60954d02a0\") " pod="openshift-ingress/e2e-inf-a-ff1189-openshift-default-7d9f9f5c65-jjdbw" Oct 30 16:49:46.172750 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:46.172727 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cq9nr\" (UniqueName: \"kubernetes.io/projected/ba6262c9-b3ea-4c61-bb84-4f60954d02a0-kube-api-access-cq9nr\") pod \"e2e-inf-a-ff1189-openshift-default-7d9f9f5c65-jjdbw\" (UID: \"ba6262c9-b3ea-4c61-bb84-4f60954d02a0\") " pod="openshift-ingress/e2e-inf-a-ff1189-openshift-default-7d9f9f5c65-jjdbw" Oct 30 16:49:46.283606 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:46.283526 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba6262c9-b3ea-4c61-bb84-4f60954d02a0-istio-envoy\") pod \"e2e-inf-a-ff1189-openshift-default-7d9f9f5c65-jjdbw\" (UID: \"ba6262c9-b3ea-4c61-bb84-4f60954d02a0\") " pod="openshift-ingress/e2e-inf-a-ff1189-openshift-default-7d9f9f5c65-jjdbw" Oct 30 16:49:46.283606 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:46.283596 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba6262c9-b3ea-4c61-bb84-4f60954d02a0-istio-podinfo\") pod \"e2e-inf-a-ff1189-openshift-default-7d9f9f5c65-jjdbw\" (UID: \"ba6262c9-b3ea-4c61-bb84-4f60954d02a0\") " pod="openshift-ingress/e2e-inf-a-ff1189-openshift-default-7d9f9f5c65-jjdbw" Oct 30 16:49:46.283868 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:46.283711 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba6262c9-b3ea-4c61-bb84-4f60954d02a0-workload-certs\") pod \"e2e-inf-a-ff1189-openshift-default-7d9f9f5c65-jjdbw\" (UID: \"ba6262c9-b3ea-4c61-bb84-4f60954d02a0\") " pod="openshift-ingress/e2e-inf-a-ff1189-openshift-default-7d9f9f5c65-jjdbw" Oct 30 16:49:46.283868 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:46.283765 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba6262c9-b3ea-4c61-bb84-4f60954d02a0-workload-socket\") pod \"e2e-inf-a-ff1189-openshift-default-7d9f9f5c65-jjdbw\" (UID: \"ba6262c9-b3ea-4c61-bb84-4f60954d02a0\") " pod="openshift-ingress/e2e-inf-a-ff1189-openshift-default-7d9f9f5c65-jjdbw" Oct 30 16:49:46.283868 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:46.283808 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba6262c9-b3ea-4c61-bb84-4f60954d02a0-istio-data\") pod \"e2e-inf-a-ff1189-openshift-default-7d9f9f5c65-jjdbw\" (UID: \"ba6262c9-b3ea-4c61-bb84-4f60954d02a0\") " pod="openshift-ingress/e2e-inf-a-ff1189-openshift-default-7d9f9f5c65-jjdbw" Oct 30 16:49:46.284039 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:46.283898 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba6262c9-b3ea-4c61-bb84-4f60954d02a0-istiod-ca-cert\") pod \"e2e-inf-a-ff1189-openshift-default-7d9f9f5c65-jjdbw\" (UID: \"ba6262c9-b3ea-4c61-bb84-4f60954d02a0\") " pod="openshift-ingress/e2e-inf-a-ff1189-openshift-default-7d9f9f5c65-jjdbw" Oct 30 16:49:46.284039 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:46.283926 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba6262c9-b3ea-4c61-bb84-4f60954d02a0-credential-socket\") pod \"e2e-inf-a-ff1189-openshift-default-7d9f9f5c65-jjdbw\" (UID: \"ba6262c9-b3ea-4c61-bb84-4f60954d02a0\") " pod="openshift-ingress/e2e-inf-a-ff1189-openshift-default-7d9f9f5c65-jjdbw" Oct 30 16:49:46.284039 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:46.284000 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq9nr\" (UniqueName: \"kubernetes.io/projected/ba6262c9-b3ea-4c61-bb84-4f60954d02a0-kube-api-access-cq9nr\") pod \"e2e-inf-a-ff1189-openshift-default-7d9f9f5c65-jjdbw\" (UID: \"ba6262c9-b3ea-4c61-bb84-4f60954d02a0\") " pod="openshift-ingress/e2e-inf-a-ff1189-openshift-default-7d9f9f5c65-jjdbw" Oct 30 16:49:46.284202 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:46.284086 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba6262c9-b3ea-4c61-bb84-4f60954d02a0-istio-token\") pod \"e2e-inf-a-ff1189-openshift-default-7d9f9f5c65-jjdbw\" (UID: \"ba6262c9-b3ea-4c61-bb84-4f60954d02a0\") " pod="openshift-ingress/e2e-inf-a-ff1189-openshift-default-7d9f9f5c65-jjdbw" Oct 30 16:49:46.306861 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:46.285176 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba6262c9-b3ea-4c61-bb84-4f60954d02a0-credential-socket\") pod \"e2e-inf-a-ff1189-openshift-default-7d9f9f5c65-jjdbw\" (UID: \"ba6262c9-b3ea-4c61-bb84-4f60954d02a0\") " pod="openshift-ingress/e2e-inf-a-ff1189-openshift-default-7d9f9f5c65-jjdbw" Oct 30 16:49:46.306861 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:46.286320 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba6262c9-b3ea-4c61-bb84-4f60954d02a0-workload-certs\") pod \"e2e-inf-a-ff1189-openshift-default-7d9f9f5c65-jjdbw\" (UID: \"ba6262c9-b3ea-4c61-bb84-4f60954d02a0\") " pod="openshift-ingress/e2e-inf-a-ff1189-openshift-default-7d9f9f5c65-jjdbw" Oct 30 16:49:46.306861 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:46.286960 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba6262c9-b3ea-4c61-bb84-4f60954d02a0-workload-socket\") pod \"e2e-inf-a-ff1189-openshift-default-7d9f9f5c65-jjdbw\" (UID: \"ba6262c9-b3ea-4c61-bb84-4f60954d02a0\") " pod="openshift-ingress/e2e-inf-a-ff1189-openshift-default-7d9f9f5c65-jjdbw" Oct 30 16:49:46.306861 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:46.295215 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba6262c9-b3ea-4c61-bb84-4f60954d02a0-istiod-ca-cert\") pod \"e2e-inf-a-ff1189-openshift-default-7d9f9f5c65-jjdbw\" (UID: \"ba6262c9-b3ea-4c61-bb84-4f60954d02a0\") " pod="openshift-ingress/e2e-inf-a-ff1189-openshift-default-7d9f9f5c65-jjdbw" Oct 30 16:49:46.306861 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:46.295550 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba6262c9-b3ea-4c61-bb84-4f60954d02a0-istio-data\") pod \"e2e-inf-a-ff1189-openshift-default-7d9f9f5c65-jjdbw\" (UID: \"ba6262c9-b3ea-4c61-bb84-4f60954d02a0\") " pod="openshift-ingress/e2e-inf-a-ff1189-openshift-default-7d9f9f5c65-jjdbw" Oct 30 16:49:46.315767 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:46.313816 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba6262c9-b3ea-4c61-bb84-4f60954d02a0-istio-envoy\") pod \"e2e-inf-a-ff1189-openshift-default-7d9f9f5c65-jjdbw\" (UID: \"ba6262c9-b3ea-4c61-bb84-4f60954d02a0\") " pod="openshift-ingress/e2e-inf-a-ff1189-openshift-default-7d9f9f5c65-jjdbw" Oct 30 16:49:46.315767 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:46.314748 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba6262c9-b3ea-4c61-bb84-4f60954d02a0-istio-podinfo\") pod \"e2e-inf-a-ff1189-openshift-default-7d9f9f5c65-jjdbw\" (UID: \"ba6262c9-b3ea-4c61-bb84-4f60954d02a0\") " pod="openshift-ingress/e2e-inf-a-ff1189-openshift-default-7d9f9f5c65-jjdbw" Oct 30 16:49:46.324219 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:46.323641 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba6262c9-b3ea-4c61-bb84-4f60954d02a0-istio-token\") pod \"e2e-inf-a-ff1189-openshift-default-7d9f9f5c65-jjdbw\" (UID: \"ba6262c9-b3ea-4c61-bb84-4f60954d02a0\") " pod="openshift-ingress/e2e-inf-a-ff1189-openshift-default-7d9f9f5c65-jjdbw" Oct 30 16:49:46.345606 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:46.334457 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cq9nr\" (UniqueName: \"kubernetes.io/projected/ba6262c9-b3ea-4c61-bb84-4f60954d02a0-kube-api-access-cq9nr\") pod \"e2e-inf-a-ff1189-openshift-default-7d9f9f5c65-jjdbw\" (UID: \"ba6262c9-b3ea-4c61-bb84-4f60954d02a0\") " pod="openshift-ingress/e2e-inf-a-ff1189-openshift-default-7d9f9f5c65-jjdbw" Oct 30 16:49:46.419552 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:46.418912 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ff1189-openshift-default-7d9f9f5c65-jjdbw" Oct 30 16:49:46.991373 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:46.991336 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-ff1189-openshift-default-7d9f9f5c65-jjdbw"] Oct 30 16:49:46.994856 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:46.994813 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:49:46.994974 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:46.994902 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:49:46.994974 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:46.994944 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:49:47.466570 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:47.466527 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ff1189-openshift-default-7d9f9f5c65-jjdbw" event={"ID":"ba6262c9-b3ea-4c61-bb84-4f60954d02a0","Type":"ContainerStarted","Data":"32ccfb376ada51e1c34cb823a8a1cc25e29e84c2523661656fa4b3b14e66ec53"} Oct 30 16:49:47.466957 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:47.466613 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ff1189-openshift-default-7d9f9f5c65-jjdbw" event={"ID":"ba6262c9-b3ea-4c61-bb84-4f60954d02a0","Type":"ContainerStarted","Data":"21622c1501bf04f45dfb293e19e3c8ceb1d47685a3a3d9768d2ea38d70de5447"} Oct 30 16:49:47.523324 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:47.523261 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-ff1189-openshift-default-7d9f9f5c65-jjdbw" podStartSLOduration=1.523243393 podStartE2EDuration="1.523243393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:49:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:49:47.515664815 +0000 UTC m=+1743.043000038" watchObservedRunningTime="2026-10-30 16:49:47.523243393 +0000 UTC m=+1743.050578612" Oct 30 16:49:48.332747 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:48.332701 5052 patch_prober.go:29] interesting pod/e2e-api-b-407eab-openshift-default-85f774ff9c-nldmk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:49:48.333072 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:48.332764 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-407eab-openshift-default-85f774ff9c-nldmk" podUID="218cbb21-a450-4352-b4b3-f7e8558688bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:49:48.420423 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:48.420387 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-ff1189-openshift-default-7d9f9f5c65-jjdbw" Oct 30 16:49:48.428259 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:48.428227 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-ff1189-openshift-default-7d9f9f5c65-jjdbw" Oct 30 16:49:48.478309 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:48.476956 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-ff1189-openshift-default-7d9f9f5c65-jjdbw" Oct 30 16:49:48.478309 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:48.478107 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-ff1189-openshift-default-7d9f9f5c65-jjdbw" Oct 30 16:49:48.847924 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:49:48.847881 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 16:49:51.572080 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:51.567927 5052 generic.go:356] "Generic (PLEG): container finished" podID="218cbb21-a450-4352-b4b3-f7e8558688bb" containerID="5533cf1b92cd792dfcb6a1953f72d0abedaee4cb23f48905192a29a1642e75f3" exitCode=0 Oct 30 16:49:51.572080 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:51.569896 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-407eab-openshift-default-85f774ff9c-nldmk" event={"ID":"218cbb21-a450-4352-b4b3-f7e8558688bb","Type":"ContainerDied","Data":"5533cf1b92cd792dfcb6a1953f72d0abedaee4cb23f48905192a29a1642e75f3"} Oct 30 16:49:51.572080 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:51.570528 5052 scope.go:122] "RemoveContainer" containerID="5533cf1b92cd792dfcb6a1953f72d0abedaee4cb23f48905192a29a1642e75f3" Oct 30 16:49:51.573159 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:51.573123 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:49:51.573294 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:51.573269 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:49:51.573387 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:51.573339 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:49:51.841152 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:51.841115 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ff1189-759ff754b7-gl6mj"] Oct 30 16:49:51.847497 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:51.847458 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ff1189-759ff754b7-gl6mj" Oct 30 16:49:51.866977 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:51.866949 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ff1189-759ff754b7-gl6mj"] Oct 30 16:49:51.994682 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:51.982147 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-ff1189\"" Oct 30 16:49:51.994682 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:51.982310 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-ff1189-dockercfg-d2rk2\"" Oct 30 16:49:52.111603 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:52.108601 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4vfj\" (UniqueName: \"kubernetes.io/projected/5cbbc24b-bf64-455e-9dc2-c34487840dfb-kube-api-access-n4vfj\") pod \"payload-pre-processing-e2e-inf-a-ff1189-759ff754b7-gl6mj\" (UID: \"5cbbc24b-bf64-455e-9dc2-c34487840dfb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ff1189-759ff754b7-gl6mj" Oct 30 16:49:52.111603 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:52.108755 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cbbc24b-bf64-455e-9dc2-c34487840dfb-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-ff1189-759ff754b7-gl6mj\" (UID: \"5cbbc24b-bf64-455e-9dc2-c34487840dfb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ff1189-759ff754b7-gl6mj" Oct 30 16:49:52.266766 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:52.224126 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4vfj\" (UniqueName: \"kubernetes.io/projected/5cbbc24b-bf64-455e-9dc2-c34487840dfb-kube-api-access-n4vfj\") pod \"payload-pre-processing-e2e-inf-a-ff1189-759ff754b7-gl6mj\" (UID: \"5cbbc24b-bf64-455e-9dc2-c34487840dfb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ff1189-759ff754b7-gl6mj" Oct 30 16:49:52.266766 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:52.224277 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cbbc24b-bf64-455e-9dc2-c34487840dfb-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-ff1189-759ff754b7-gl6mj\" (UID: \"5cbbc24b-bf64-455e-9dc2-c34487840dfb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ff1189-759ff754b7-gl6mj" Oct 30 16:49:52.266766 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:52.225453 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cbbc24b-bf64-455e-9dc2-c34487840dfb-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-ff1189-759ff754b7-gl6mj\" (UID: \"5cbbc24b-bf64-455e-9dc2-c34487840dfb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ff1189-759ff754b7-gl6mj" Oct 30 16:49:52.301368 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:52.300812 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4vfj\" (UniqueName: \"kubernetes.io/projected/5cbbc24b-bf64-455e-9dc2-c34487840dfb-kube-api-access-n4vfj\") pod \"payload-pre-processing-e2e-inf-a-ff1189-759ff754b7-gl6mj\" (UID: \"5cbbc24b-bf64-455e-9dc2-c34487840dfb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ff1189-759ff754b7-gl6mj" Oct 30 16:49:52.467432 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:52.467401 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ff1189-759ff754b7-gl6mj" Oct 30 16:49:52.581553 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:52.581521 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-407eab-openshift-default-85f774ff9c-nldmk" event={"ID":"218cbb21-a450-4352-b4b3-f7e8558688bb","Type":"ContainerStarted","Data":"cf98a641a60e69324dd36c184f61b43dc9e0ef04ff6c274019a05a026b4dab99"} Oct 30 16:49:52.786002 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:52.785969 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ff1189-759ff754b7-gl6mj"] Oct 30 16:49:52.786410 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:49:52.786385 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cbbc24b_bf64_455e_9dc2_c34487840dfb.slice/crio-c3696052e4802b084352abf0e87dc286a0bc458cef025c67f0b72ffe6804f9b2 WatchSource:0}: Error finding container c3696052e4802b084352abf0e87dc286a0bc458cef025c67f0b72ffe6804f9b2: Status 404 returned error can't find the container with id c3696052e4802b084352abf0e87dc286a0bc458cef025c67f0b72ffe6804f9b2 Oct 30 16:49:53.412560 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:53.412529 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-407eab-openshift-default-85f774ff9c-nldmk" Oct 30 16:49:53.412747 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:53.412698 5052 patch_prober.go:29] interesting pod/e2e-api-b-407eab-openshift-default-85f774ff9c-nldmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Oct 30 16:49:53.412792 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:53.412740 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-407eab-openshift-default-85f774ff9c-nldmk" podUID="218cbb21-a450-4352-b4b3-f7e8558688bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Oct 30 16:49:53.688638 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:53.688546 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ff1189-759ff754b7-gl6mj" event={"ID":"5cbbc24b-bf64-455e-9dc2-c34487840dfb","Type":"ContainerStarted","Data":"321c963f3ddeb9e3048bc683bc1bd9dde745780526526d048f87684b21d0146f"} Oct 30 16:49:53.689000 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:53.688644 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ff1189-759ff754b7-gl6mj" Oct 30 16:49:53.689000 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:53.688665 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ff1189-759ff754b7-gl6mj" event={"ID":"5cbbc24b-bf64-455e-9dc2-c34487840dfb","Type":"ContainerStarted","Data":"c3696052e4802b084352abf0e87dc286a0bc458cef025c67f0b72ffe6804f9b2"} Oct 30 16:49:53.751607 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:53.742088 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ff1189-759ff754b7-gl6mj" podStartSLOduration=2.742066816 podStartE2EDuration="2.742066816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:49:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:49:53.740256149 +0000 UTC m=+1749.267591369" watchObservedRunningTime="2026-10-30 16:49:53.742066816 +0000 UTC m=+1749.269402036" Oct 30 16:49:54.412754 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:54.412721 5052 patch_prober.go:29] interesting pod/e2e-api-b-407eab-openshift-default-85f774ff9c-nldmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Oct 30 16:49:54.412927 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:54.412772 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-407eab-openshift-default-85f774ff9c-nldmk" podUID="218cbb21-a450-4352-b4b3-f7e8558688bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Oct 30 16:49:55.412711 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:55.412679 5052 patch_prober.go:29] interesting pod/e2e-api-b-407eab-openshift-default-85f774ff9c-nldmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Oct 30 16:49:55.413122 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:55.412734 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-407eab-openshift-default-85f774ff9c-nldmk" podUID="218cbb21-a450-4352-b4b3-f7e8558688bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Oct 30 16:49:56.412171 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:56.412131 5052 patch_prober.go:29] interesting pod/e2e-api-b-407eab-openshift-default-85f774ff9c-nldmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Oct 30 16:49:56.412346 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:56.412195 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-407eab-openshift-default-85f774ff9c-nldmk" podUID="218cbb21-a450-4352-b4b3-f7e8558688bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Oct 30 16:49:57.412834 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:57.412803 5052 patch_prober.go:29] interesting pod/e2e-api-b-407eab-openshift-default-85f774ff9c-nldmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Oct 30 16:49:57.413345 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:57.413311 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-407eab-openshift-default-85f774ff9c-nldmk" podUID="218cbb21-a450-4352-b4b3-f7e8558688bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Oct 30 16:49:58.692027 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:58.691994 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-407eab-openshift-default-85f774ff9c-nldmk" Oct 30 16:49:58.693188 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:58.693083 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-407eab-openshift-default-85f774ff9c-nldmk" Oct 30 16:49:58.697332 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:49:58.697298 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-407eab-openshift-default-85f774ff9c-nldmk" Oct 30 16:50:02.026154 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:02.026119 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-3ee682-6ddc5f6db7-wlnkt"] Oct 30 16:50:02.032011 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:02.031990 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3ee682-6ddc5f6db7-wlnkt" Oct 30 16:50:02.053849 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:02.053820 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-3ee682-serving-cert\"" Oct 30 16:50:02.053989 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:02.053880 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-3ee682-6ddc5f6db7-wlnkt"] Oct 30 16:50:02.110207 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:02.110169 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d254a10-0b28-40b3-8240-e278e39c2f9b-openshift-service-ca\") pod \"maas-api-e2e-inf-b-3ee682-6ddc5f6db7-wlnkt\" (UID: \"6d254a10-0b28-40b3-8240-e278e39c2f9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3ee682-6ddc5f6db7-wlnkt" Oct 30 16:50:02.110432 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:02.110225 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d254a10-0b28-40b3-8240-e278e39c2f9b-metrics-certs\") pod \"maas-api-e2e-inf-b-3ee682-6ddc5f6db7-wlnkt\" (UID: \"6d254a10-0b28-40b3-8240-e278e39c2f9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3ee682-6ddc5f6db7-wlnkt" Oct 30 16:50:02.110938 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:02.110911 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d254a10-0b28-40b3-8240-e278e39c2f9b-maas-api-tls\") pod \"maas-api-e2e-inf-b-3ee682-6ddc5f6db7-wlnkt\" (UID: \"6d254a10-0b28-40b3-8240-e278e39c2f9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3ee682-6ddc5f6db7-wlnkt" Oct 30 16:50:02.111054 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:02.111004 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vf9z4\" (UniqueName: \"kubernetes.io/projected/6d254a10-0b28-40b3-8240-e278e39c2f9b-kube-api-access-vf9z4\") pod \"maas-api-e2e-inf-b-3ee682-6ddc5f6db7-wlnkt\" (UID: \"6d254a10-0b28-40b3-8240-e278e39c2f9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3ee682-6ddc5f6db7-wlnkt" Oct 30 16:50:02.259601 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:02.228123 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d254a10-0b28-40b3-8240-e278e39c2f9b-maas-api-tls\") pod \"maas-api-e2e-inf-b-3ee682-6ddc5f6db7-wlnkt\" (UID: \"6d254a10-0b28-40b3-8240-e278e39c2f9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3ee682-6ddc5f6db7-wlnkt" Oct 30 16:50:02.265761 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:02.265712 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vf9z4\" (UniqueName: \"kubernetes.io/projected/6d254a10-0b28-40b3-8240-e278e39c2f9b-kube-api-access-vf9z4\") pod \"maas-api-e2e-inf-b-3ee682-6ddc5f6db7-wlnkt\" (UID: \"6d254a10-0b28-40b3-8240-e278e39c2f9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3ee682-6ddc5f6db7-wlnkt" Oct 30 16:50:02.266419 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:02.266389 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d254a10-0b28-40b3-8240-e278e39c2f9b-openshift-service-ca\") pod \"maas-api-e2e-inf-b-3ee682-6ddc5f6db7-wlnkt\" (UID: \"6d254a10-0b28-40b3-8240-e278e39c2f9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3ee682-6ddc5f6db7-wlnkt" Oct 30 16:50:02.267985 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:02.267950 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d254a10-0b28-40b3-8240-e278e39c2f9b-metrics-certs\") pod \"maas-api-e2e-inf-b-3ee682-6ddc5f6db7-wlnkt\" (UID: \"6d254a10-0b28-40b3-8240-e278e39c2f9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3ee682-6ddc5f6db7-wlnkt" Oct 30 16:50:02.269046 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:02.267867 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d254a10-0b28-40b3-8240-e278e39c2f9b-openshift-service-ca\") pod \"maas-api-e2e-inf-b-3ee682-6ddc5f6db7-wlnkt\" (UID: \"6d254a10-0b28-40b3-8240-e278e39c2f9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3ee682-6ddc5f6db7-wlnkt" Oct 30 16:50:02.276652 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:02.273082 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d254a10-0b28-40b3-8240-e278e39c2f9b-metrics-certs\") pod \"maas-api-e2e-inf-b-3ee682-6ddc5f6db7-wlnkt\" (UID: \"6d254a10-0b28-40b3-8240-e278e39c2f9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3ee682-6ddc5f6db7-wlnkt" Oct 30 16:50:02.293602 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:02.288232 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d254a10-0b28-40b3-8240-e278e39c2f9b-maas-api-tls\") pod \"maas-api-e2e-inf-b-3ee682-6ddc5f6db7-wlnkt\" (UID: \"6d254a10-0b28-40b3-8240-e278e39c2f9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3ee682-6ddc5f6db7-wlnkt" Oct 30 16:50:02.340820 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:02.329823 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vf9z4\" (UniqueName: \"kubernetes.io/projected/6d254a10-0b28-40b3-8240-e278e39c2f9b-kube-api-access-vf9z4\") pod \"maas-api-e2e-inf-b-3ee682-6ddc5f6db7-wlnkt\" (UID: \"6d254a10-0b28-40b3-8240-e278e39c2f9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3ee682-6ddc5f6db7-wlnkt" Oct 30 16:50:02.354631 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:02.348674 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3ee682-6ddc5f6db7-wlnkt" Oct 30 16:50:02.691189 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:02.691162 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-3ee682-6ddc5f6db7-wlnkt"] Oct 30 16:50:02.691555 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:50:02.691536 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d254a10_0b28_40b3_8240_e278e39c2f9b.slice/crio-e2ccd3c3a48c9460aed0444aa83d905b4b1846f0cb47b7e84224d638f762e424 WatchSource:0}: Error finding container e2ccd3c3a48c9460aed0444aa83d905b4b1846f0cb47b7e84224d638f762e424: Status 404 returned error can't find the container with id e2ccd3c3a48c9460aed0444aa83d905b4b1846f0cb47b7e84224d638f762e424 Oct 30 16:50:02.821419 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:02.821384 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3ee682-6ddc5f6db7-wlnkt" event={"ID":"6d254a10-0b28-40b3-8240-e278e39c2f9b","Type":"ContainerStarted","Data":"e2ccd3c3a48c9460aed0444aa83d905b4b1846f0cb47b7e84224d638f762e424"} Oct 30 16:50:03.832170 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:03.832129 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3ee682-6ddc5f6db7-wlnkt" event={"ID":"6d254a10-0b28-40b3-8240-e278e39c2f9b","Type":"ContainerStarted","Data":"ab703084114e4f6ff8dc76f954a96eed87c72473273e649bad82249a528f3c2e"} Oct 30 16:50:03.832793 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:03.832262 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3ee682-6ddc5f6db7-wlnkt" Oct 30 16:50:03.927954 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:03.927866 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3ee682-6ddc5f6db7-wlnkt" podStartSLOduration=2.520708892 podStartE2EDuration="2.927851447s" totalImagesPullingTime="407.142555ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:50:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:50:03.923829808 +0000 UTC m=+1759.451165026" watchObservedRunningTime="2026-10-30 16:50:03.927851447 +0000 UTC m=+1759.455186665" Oct 30 16:50:03.987459 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:03.987418 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cdwcj"] Oct 30 16:50:04.787800 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:04.786680 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ff1189-759ff754b7-gl6mj" Oct 30 16:50:09.934507 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:09.934479 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3ee682-6ddc5f6db7-wlnkt" Oct 30 16:50:13.374619 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:13.374074 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-00fd1a-86b946bb7f-dtk29"] Oct 30 16:50:18.375279 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:18.375218 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-00fd1a-86b946bb7f-dtk29" podUID="a1e2767a-41a6-453f-8ac7-4bc93b77b143" containerName="payload-pre-processing" containerID="cri-o://6882094be201b85d43522fe5d438a0306cf2236c06b32757a497439be3f82349" gracePeriod=25 Oct 30 16:50:18.805250 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:18.805219 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-00fd1a-86b946bb7f-dtk29" Oct 30 16:50:18.845476 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:50:18.845432 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 16:50:18.918258 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:18.918217 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1e2767a-41a6-453f-8ac7-4bc93b77b143-kube-api-access-68zcp\" (UniqueName: \"kubernetes.io/projected/a1e2767a-41a6-453f-8ac7-4bc93b77b143-kube-api-access-68zcp\") pod \"a1e2767a-41a6-453f-8ac7-4bc93b77b143\" (UID: \"a1e2767a-41a6-453f-8ac7-4bc93b77b143\") " Oct 30 16:50:18.918428 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:18.918304 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1e2767a-41a6-453f-8ac7-4bc93b77b143-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1e2767a-41a6-453f-8ac7-4bc93b77b143-plugins-config-volume\") pod \"a1e2767a-41a6-453f-8ac7-4bc93b77b143\" (UID: \"a1e2767a-41a6-453f-8ac7-4bc93b77b143\") " Oct 30 16:50:18.918697 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:18.918669 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1e2767a-41a6-453f-8ac7-4bc93b77b143-plugins-config-volume" pod "a1e2767a-41a6-453f-8ac7-4bc93b77b143" (UID: "a1e2767a-41a6-453f-8ac7-4bc93b77b143"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:50:18.919218 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:18.919194 5052 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1e2767a-41a6-453f-8ac7-4bc93b77b143-plugins-config-volume\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:50:18.920469 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:18.920448 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1e2767a-41a6-453f-8ac7-4bc93b77b143-kube-api-access-68zcp" pod "a1e2767a-41a6-453f-8ac7-4bc93b77b143" (UID: "a1e2767a-41a6-453f-8ac7-4bc93b77b143"). InnerVolumeSpecName "kube-api-access-68zcp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:50:19.020538 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:19.020471 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68zcp\" (UniqueName: \"kubernetes.io/projected/a1e2767a-41a6-453f-8ac7-4bc93b77b143-kube-api-access-68zcp\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:50:19.423124 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:19.423088 5052 generic.go:356] "Generic (PLEG): container finished" podID="a1e2767a-41a6-453f-8ac7-4bc93b77b143" containerID="6882094be201b85d43522fe5d438a0306cf2236c06b32757a497439be3f82349" exitCode=0 Oct 30 16:50:19.423566 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:19.423144 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-00fd1a-86b946bb7f-dtk29" Oct 30 16:50:19.423566 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:19.423152 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-00fd1a-86b946bb7f-dtk29" event={"ID":"a1e2767a-41a6-453f-8ac7-4bc93b77b143","Type":"ContainerDied","Data":"6882094be201b85d43522fe5d438a0306cf2236c06b32757a497439be3f82349"} Oct 30 16:50:19.423566 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:19.423205 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-00fd1a-86b946bb7f-dtk29" event={"ID":"a1e2767a-41a6-453f-8ac7-4bc93b77b143","Type":"ContainerDied","Data":"a3530ffdb87ad7a34b69a0f3a9044a4949600d2ef621bfe3c467c4e2437c297a"} Oct 30 16:50:19.423566 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:19.423227 5052 scope.go:122] "RemoveContainer" containerID="6882094be201b85d43522fe5d438a0306cf2236c06b32757a497439be3f82349" Oct 30 16:50:19.434442 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:19.434426 5052 scope.go:122] "RemoveContainer" containerID="6882094be201b85d43522fe5d438a0306cf2236c06b32757a497439be3f82349" Oct 30 16:50:19.434743 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:50:19.434720 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6882094be201b85d43522fe5d438a0306cf2236c06b32757a497439be3f82349\": container with ID starting with 6882094be201b85d43522fe5d438a0306cf2236c06b32757a497439be3f82349 not found: ID does not exist" containerID="6882094be201b85d43522fe5d438a0306cf2236c06b32757a497439be3f82349" Oct 30 16:50:19.434826 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:19.434753 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6882094be201b85d43522fe5d438a0306cf2236c06b32757a497439be3f82349"} err="failed to get container status \"6882094be201b85d43522fe5d438a0306cf2236c06b32757a497439be3f82349\": rpc error: code = NotFound desc = could not find container \"6882094be201b85d43522fe5d438a0306cf2236c06b32757a497439be3f82349\": container with ID starting with 6882094be201b85d43522fe5d438a0306cf2236c06b32757a497439be3f82349 not found: ID does not exist" Oct 30 16:50:19.481616 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:19.481574 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-00fd1a-86b946bb7f-dtk29"] Oct 30 16:50:19.494846 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:19.494817 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-00fd1a-86b946bb7f-dtk29"] Oct 30 16:50:20.750941 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:20.750742 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1e2767a-41a6-453f-8ac7-4bc93b77b143" path="/var/lib/kubelet/pods/a1e2767a-41a6-453f-8ac7-4bc93b77b143/volumes" Oct 30 16:50:21.371408 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:21.371373 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-hff9w"] Oct 30 16:50:21.371921 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:21.371624 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-hff9w" podUID="559db955-2645-4b32-9070-e87d7bcb485e" containerName="istio-proxy" containerID="cri-o://cf59e7c517f3ab79b6bb2d3c5016ce9ada14cc4a77b33e89e5c0c5271f581a54" gracePeriod=30 Oct 30 16:50:21.386401 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:21.386367 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-v2jw7"] Oct 30 16:50:21.386624 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:21.386596 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-v2jw7" podUID="a40b021f-16e2-4821-ad8b-9a7fb6a4a58f" containerName="istio-proxy" containerID="cri-o://c23ab4acd7d104fb93b7a9ff046abf01b183265f52ce398769d1d413d399f552" gracePeriod=30 Oct 30 16:50:21.505758 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:21.505725 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-f7blz"] Oct 30 16:50:21.505946 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:21.505926 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-f7blz" podUID="1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1" containerName="istio-proxy" containerID="cri-o://2d72c42e5bc4e3f1dab3f51c29d6ffad1ad86844ed26aeace4fd738a49c2bfd7" gracePeriod=30 Oct 30 16:50:26.327038 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:26.326998 5052 patch_prober.go:29] interesting pod/e2e-worker-w4-263f00-openshift-default-b969d55bc-f7blz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 16:50:26.327496 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:26.327080 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-f7blz" podUID="1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 16:50:26.818772 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:26.818047 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-hff9w" Oct 30 16:50:26.823022 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:26.822447 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-v2jw7" Oct 30 16:50:26.860885 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:26.854136 5052 generic.go:356] "Generic (PLEG): container finished" podID="559db955-2645-4b32-9070-e87d7bcb485e" containerID="cf59e7c517f3ab79b6bb2d3c5016ce9ada14cc4a77b33e89e5c0c5271f581a54" exitCode=0 Oct 30 16:50:26.860885 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:26.854315 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-hff9w" event={"ID":"559db955-2645-4b32-9070-e87d7bcb485e","Type":"ContainerDied","Data":"cf59e7c517f3ab79b6bb2d3c5016ce9ada14cc4a77b33e89e5c0c5271f581a54"} Oct 30 16:50:26.860885 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:26.854375 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-hff9w" event={"ID":"559db955-2645-4b32-9070-e87d7bcb485e","Type":"ContainerDied","Data":"5fa9cfd7dc3091f4bdec10986926c81c7fe12ae7ccd7b56dd71ac53d9de06d30"} Oct 30 16:50:26.860885 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:26.854406 5052 scope.go:122] "RemoveContainer" containerID="cf59e7c517f3ab79b6bb2d3c5016ce9ada14cc4a77b33e89e5c0c5271f581a54" Oct 30 16:50:26.860885 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:26.854567 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-hff9w" Oct 30 16:50:26.865325 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:26.864990 5052 generic.go:356] "Generic (PLEG): container finished" podID="a40b021f-16e2-4821-ad8b-9a7fb6a4a58f" containerID="c23ab4acd7d104fb93b7a9ff046abf01b183265f52ce398769d1d413d399f552" exitCode=0 Oct 30 16:50:26.865325 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:26.865079 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-v2jw7" Oct 30 16:50:26.865325 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:26.865142 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-v2jw7" event={"ID":"a40b021f-16e2-4821-ad8b-9a7fb6a4a58f","Type":"ContainerDied","Data":"c23ab4acd7d104fb93b7a9ff046abf01b183265f52ce398769d1d413d399f552"} Oct 30 16:50:26.865325 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:26.865196 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-v2jw7" event={"ID":"a40b021f-16e2-4821-ad8b-9a7fb6a4a58f","Type":"ContainerDied","Data":"29dda3bf19c718bd1846e00f562ac38729bb8ee10adf7718a8856492979cd8c0"} Oct 30 16:50:26.868061 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:26.868037 5052 generic.go:356] "Generic (PLEG): container finished" podID="1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1" containerID="2d72c42e5bc4e3f1dab3f51c29d6ffad1ad86844ed26aeace4fd738a49c2bfd7" exitCode=0 Oct 30 16:50:26.868195 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:26.868098 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-f7blz" event={"ID":"1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1","Type":"ContainerDied","Data":"2d72c42e5bc4e3f1dab3f51c29d6ffad1ad86844ed26aeace4fd738a49c2bfd7"} Oct 30 16:50:26.876161 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:26.876125 5052 scope.go:122] "RemoveContainer" containerID="1d5a748362b69bcf0ccd10cfb10f69389cbb92e9c484daf9086340a58d426295" Oct 30 16:50:26.887807 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:26.887753 5052 scope.go:122] "RemoveContainer" containerID="cf59e7c517f3ab79b6bb2d3c5016ce9ada14cc4a77b33e89e5c0c5271f581a54" Oct 30 16:50:26.888113 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:50:26.888074 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf59e7c517f3ab79b6bb2d3c5016ce9ada14cc4a77b33e89e5c0c5271f581a54\": container with ID starting with cf59e7c517f3ab79b6bb2d3c5016ce9ada14cc4a77b33e89e5c0c5271f581a54 not found: ID does not exist" containerID="cf59e7c517f3ab79b6bb2d3c5016ce9ada14cc4a77b33e89e5c0c5271f581a54" Oct 30 16:50:26.888184 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:26.888124 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf59e7c517f3ab79b6bb2d3c5016ce9ada14cc4a77b33e89e5c0c5271f581a54"} err="failed to get container status \"cf59e7c517f3ab79b6bb2d3c5016ce9ada14cc4a77b33e89e5c0c5271f581a54\": rpc error: code = NotFound desc = could not find container \"cf59e7c517f3ab79b6bb2d3c5016ce9ada14cc4a77b33e89e5c0c5271f581a54\": container with ID starting with cf59e7c517f3ab79b6bb2d3c5016ce9ada14cc4a77b33e89e5c0c5271f581a54 not found: ID does not exist" Oct 30 16:50:26.888184 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:26.888141 5052 scope.go:122] "RemoveContainer" containerID="1d5a748362b69bcf0ccd10cfb10f69389cbb92e9c484daf9086340a58d426295" Oct 30 16:50:26.888777 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:50:26.888666 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d5a748362b69bcf0ccd10cfb10f69389cbb92e9c484daf9086340a58d426295\": container with ID starting with 1d5a748362b69bcf0ccd10cfb10f69389cbb92e9c484daf9086340a58d426295 not found: ID does not exist" containerID="1d5a748362b69bcf0ccd10cfb10f69389cbb92e9c484daf9086340a58d426295" Oct 30 16:50:26.888777 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:26.888692 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d5a748362b69bcf0ccd10cfb10f69389cbb92e9c484daf9086340a58d426295"} err="failed to get container status \"1d5a748362b69bcf0ccd10cfb10f69389cbb92e9c484daf9086340a58d426295\": rpc error: code = NotFound desc = could not find container \"1d5a748362b69bcf0ccd10cfb10f69389cbb92e9c484daf9086340a58d426295\": container with ID starting with 1d5a748362b69bcf0ccd10cfb10f69389cbb92e9c484daf9086340a58d426295 not found: ID does not exist" Oct 30 16:50:26.888777 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:26.888708 5052 scope.go:122] "RemoveContainer" containerID="c23ab4acd7d104fb93b7a9ff046abf01b183265f52ce398769d1d413d399f552" Oct 30 16:50:26.911984 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:26.911918 5052 scope.go:122] "RemoveContainer" containerID="c23ab4acd7d104fb93b7a9ff046abf01b183265f52ce398769d1d413d399f552" Oct 30 16:50:26.912334 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:50:26.912304 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c23ab4acd7d104fb93b7a9ff046abf01b183265f52ce398769d1d413d399f552\": container with ID starting with c23ab4acd7d104fb93b7a9ff046abf01b183265f52ce398769d1d413d399f552 not found: ID does not exist" containerID="c23ab4acd7d104fb93b7a9ff046abf01b183265f52ce398769d1d413d399f552" Oct 30 16:50:26.912449 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:26.912336 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c23ab4acd7d104fb93b7a9ff046abf01b183265f52ce398769d1d413d399f552"} err="failed to get container status \"c23ab4acd7d104fb93b7a9ff046abf01b183265f52ce398769d1d413d399f552\": rpc error: code = NotFound desc = could not find container \"c23ab4acd7d104fb93b7a9ff046abf01b183265f52ce398769d1d413d399f552\": container with ID starting with c23ab4acd7d104fb93b7a9ff046abf01b183265f52ce398769d1d413d399f552 not found: ID does not exist" Oct 30 16:50:26.963698 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:26.963669 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-f7blz" Oct 30 16:50:27.005741 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:27.005699 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a40b021f-16e2-4821-ad8b-9a7fb6a4a58f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a40b021f-16e2-4821-ad8b-9a7fb6a4a58f-istio-data\") pod \"a40b021f-16e2-4821-ad8b-9a7fb6a4a58f\" (UID: \"a40b021f-16e2-4821-ad8b-9a7fb6a4a58f\") " Oct 30 16:50:27.005938 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:27.005757 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a40b021f-16e2-4821-ad8b-9a7fb6a4a58f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a40b021f-16e2-4821-ad8b-9a7fb6a4a58f-credential-socket\") pod \"a40b021f-16e2-4821-ad8b-9a7fb6a4a58f\" (UID: \"a40b021f-16e2-4821-ad8b-9a7fb6a4a58f\") " Oct 30 16:50:27.005938 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:27.005802 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/559db955-2645-4b32-9070-e87d7bcb485e-istio-token\" (UniqueName: \"kubernetes.io/projected/559db955-2645-4b32-9070-e87d7bcb485e-istio-token\") pod \"559db955-2645-4b32-9070-e87d7bcb485e\" (UID: \"559db955-2645-4b32-9070-e87d7bcb485e\") " Oct 30 16:50:27.005938 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:27.005843 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a40b021f-16e2-4821-ad8b-9a7fb6a4a58f-kube-api-access-gvwvd\" (UniqueName: \"kubernetes.io/projected/a40b021f-16e2-4821-ad8b-9a7fb6a4a58f-kube-api-access-gvwvd\") pod \"a40b021f-16e2-4821-ad8b-9a7fb6a4a58f\" (UID: \"a40b021f-16e2-4821-ad8b-9a7fb6a4a58f\") " Oct 30 16:50:27.006132 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:27.005968 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a40b021f-16e2-4821-ad8b-9a7fb6a4a58f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a40b021f-16e2-4821-ad8b-9a7fb6a4a58f-istio-podinfo\") pod \"a40b021f-16e2-4821-ad8b-9a7fb6a4a58f\" (UID: \"a40b021f-16e2-4821-ad8b-9a7fb6a4a58f\") " Oct 30 16:50:27.006132 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:27.006013 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a40b021f-16e2-4821-ad8b-9a7fb6a4a58f-istio-data" pod "a40b021f-16e2-4821-ad8b-9a7fb6a4a58f" (UID: "a40b021f-16e2-4821-ad8b-9a7fb6a4a58f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:50:27.006132 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:27.006086 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a40b021f-16e2-4821-ad8b-9a7fb6a4a58f-istio-token\" (UniqueName: \"kubernetes.io/projected/a40b021f-16e2-4821-ad8b-9a7fb6a4a58f-istio-token\") pod \"a40b021f-16e2-4821-ad8b-9a7fb6a4a58f\" (UID: \"a40b021f-16e2-4821-ad8b-9a7fb6a4a58f\") " Oct 30 16:50:27.006132 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:27.006086 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a40b021f-16e2-4821-ad8b-9a7fb6a4a58f-credential-socket" pod "a40b021f-16e2-4821-ad8b-9a7fb6a4a58f" (UID: "a40b021f-16e2-4821-ad8b-9a7fb6a4a58f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:50:27.006303 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:27.006196 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a40b021f-16e2-4821-ad8b-9a7fb6a4a58f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a40b021f-16e2-4821-ad8b-9a7fb6a4a58f-workload-socket\") pod \"a40b021f-16e2-4821-ad8b-9a7fb6a4a58f\" (UID: \"a40b021f-16e2-4821-ad8b-9a7fb6a4a58f\") " Oct 30 16:50:27.006303 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:27.006218 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/559db955-2645-4b32-9070-e87d7bcb485e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/559db955-2645-4b32-9070-e87d7bcb485e-workload-socket\") pod \"559db955-2645-4b32-9070-e87d7bcb485e\" (UID: \"559db955-2645-4b32-9070-e87d7bcb485e\") " Oct 30 16:50:27.006303 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:27.006262 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/559db955-2645-4b32-9070-e87d7bcb485e-kube-api-access-dzs6s\" (UniqueName: \"kubernetes.io/projected/559db955-2645-4b32-9070-e87d7bcb485e-kube-api-access-dzs6s\") pod \"559db955-2645-4b32-9070-e87d7bcb485e\" (UID: \"559db955-2645-4b32-9070-e87d7bcb485e\") " Oct 30 16:50:27.006452 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:27.006315 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/559db955-2645-4b32-9070-e87d7bcb485e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/559db955-2645-4b32-9070-e87d7bcb485e-workload-certs\") pod \"559db955-2645-4b32-9070-e87d7bcb485e\" (UID: \"559db955-2645-4b32-9070-e87d7bcb485e\") " Oct 30 16:50:27.006530 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:27.006468 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a40b021f-16e2-4821-ad8b-9a7fb6a4a58f-workload-socket" pod "a40b021f-16e2-4821-ad8b-9a7fb6a4a58f" (UID: "a40b021f-16e2-4821-ad8b-9a7fb6a4a58f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:50:27.006608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:27.006543 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/559db955-2645-4b32-9070-e87d7bcb485e-workload-socket" pod "559db955-2645-4b32-9070-e87d7bcb485e" (UID: "559db955-2645-4b32-9070-e87d7bcb485e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:50:27.006608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:27.006557 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/559db955-2645-4b32-9070-e87d7bcb485e-workload-certs" pod "559db955-2645-4b32-9070-e87d7bcb485e" (UID: "559db955-2645-4b32-9070-e87d7bcb485e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:50:27.006608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:27.006562 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/559db955-2645-4b32-9070-e87d7bcb485e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/559db955-2645-4b32-9070-e87d7bcb485e-credential-socket\") pod \"559db955-2645-4b32-9070-e87d7bcb485e\" (UID: \"559db955-2645-4b32-9070-e87d7bcb485e\") " Oct 30 16:50:27.006769 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:27.006647 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/559db955-2645-4b32-9070-e87d7bcb485e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/559db955-2645-4b32-9070-e87d7bcb485e-istio-data\") pod \"559db955-2645-4b32-9070-e87d7bcb485e\" (UID: \"559db955-2645-4b32-9070-e87d7bcb485e\") " Oct 30 16:50:27.006769 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:27.006683 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/559db955-2645-4b32-9070-e87d7bcb485e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/559db955-2645-4b32-9070-e87d7bcb485e-istiod-ca-cert\") pod \"559db955-2645-4b32-9070-e87d7bcb485e\" (UID: \"559db955-2645-4b32-9070-e87d7bcb485e\") " Oct 30 16:50:27.006769 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:27.006731 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a40b021f-16e2-4821-ad8b-9a7fb6a4a58f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a40b021f-16e2-4821-ad8b-9a7fb6a4a58f-istio-envoy\") pod \"a40b021f-16e2-4821-ad8b-9a7fb6a4a58f\" (UID: \"a40b021f-16e2-4821-ad8b-9a7fb6a4a58f\") " Oct 30 16:50:27.006769 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:27.006758 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a40b021f-16e2-4821-ad8b-9a7fb6a4a58f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a40b021f-16e2-4821-ad8b-9a7fb6a4a58f-istiod-ca-cert\") pod \"a40b021f-16e2-4821-ad8b-9a7fb6a4a58f\" (UID: \"a40b021f-16e2-4821-ad8b-9a7fb6a4a58f\") " Oct 30 16:50:27.006978 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:27.006788 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/559db955-2645-4b32-9070-e87d7bcb485e-credential-socket" pod "559db955-2645-4b32-9070-e87d7bcb485e" (UID: "559db955-2645-4b32-9070-e87d7bcb485e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:50:27.006978 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:27.006841 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/559db955-2645-4b32-9070-e87d7bcb485e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/559db955-2645-4b32-9070-e87d7bcb485e-istio-envoy\") pod \"559db955-2645-4b32-9070-e87d7bcb485e\" (UID: \"559db955-2645-4b32-9070-e87d7bcb485e\") " Oct 30 16:50:27.006978 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:27.006895 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/559db955-2645-4b32-9070-e87d7bcb485e-istio-data" pod "559db955-2645-4b32-9070-e87d7bcb485e" (UID: "559db955-2645-4b32-9070-e87d7bcb485e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:50:27.006978 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:27.006940 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a40b021f-16e2-4821-ad8b-9a7fb6a4a58f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a40b021f-16e2-4821-ad8b-9a7fb6a4a58f-workload-certs\") pod \"a40b021f-16e2-4821-ad8b-9a7fb6a4a58f\" (UID: \"a40b021f-16e2-4821-ad8b-9a7fb6a4a58f\") " Oct 30 16:50:27.007203 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:27.007042 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/559db955-2645-4b32-9070-e87d7bcb485e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/559db955-2645-4b32-9070-e87d7bcb485e-istio-podinfo\") pod \"559db955-2645-4b32-9070-e87d7bcb485e\" (UID: \"559db955-2645-4b32-9070-e87d7bcb485e\") " Oct 30 16:50:27.007203 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:27.007084 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/559db955-2645-4b32-9070-e87d7bcb485e-istiod-ca-cert" pod "559db955-2645-4b32-9070-e87d7bcb485e" (UID: "559db955-2645-4b32-9070-e87d7bcb485e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:50:27.007395 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:27.007356 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a40b021f-16e2-4821-ad8b-9a7fb6a4a58f-workload-certs" pod "a40b021f-16e2-4821-ad8b-9a7fb6a4a58f" (UID: "a40b021f-16e2-4821-ad8b-9a7fb6a4a58f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:50:27.008138 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:27.007490 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a40b021f-16e2-4821-ad8b-9a7fb6a4a58f-istiod-ca-cert" pod "a40b021f-16e2-4821-ad8b-9a7fb6a4a58f" (UID: "a40b021f-16e2-4821-ad8b-9a7fb6a4a58f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:50:27.008138 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:27.007959 5052 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a40b021f-16e2-4821-ad8b-9a7fb6a4a58f-istio-data\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:50:27.008138 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:27.007982 5052 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a40b021f-16e2-4821-ad8b-9a7fb6a4a58f-credential-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:50:27.008138 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:27.008000 5052 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a40b021f-16e2-4821-ad8b-9a7fb6a4a58f-workload-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:50:27.008138 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:27.008016 5052 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/559db955-2645-4b32-9070-e87d7bcb485e-workload-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:50:27.008138 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:27.008031 5052 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/559db955-2645-4b32-9070-e87d7bcb485e-workload-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:50:27.008138 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:27.008046 5052 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/559db955-2645-4b32-9070-e87d7bcb485e-credential-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:50:27.008138 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:27.008060 5052 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/559db955-2645-4b32-9070-e87d7bcb485e-istio-data\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:50:27.008138 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:27.008074 5052 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/559db955-2645-4b32-9070-e87d7bcb485e-istiod-ca-cert\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:50:27.008138 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:27.008087 5052 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a40b021f-16e2-4821-ad8b-9a7fb6a4a58f-istiod-ca-cert\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:50:27.008138 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:27.008102 5052 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a40b021f-16e2-4821-ad8b-9a7fb6a4a58f-workload-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:50:27.009486 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:27.009281 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a40b021f-16e2-4821-ad8b-9a7fb6a4a58f-istio-token" pod "a40b021f-16e2-4821-ad8b-9a7fb6a4a58f" (UID: "a40b021f-16e2-4821-ad8b-9a7fb6a4a58f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:50:27.009486 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:27.009317 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/559db955-2645-4b32-9070-e87d7bcb485e-kube-api-access-dzs6s" pod "559db955-2645-4b32-9070-e87d7bcb485e" (UID: "559db955-2645-4b32-9070-e87d7bcb485e"). InnerVolumeSpecName "kube-api-access-dzs6s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:50:27.009607 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:27.009561 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a40b021f-16e2-4821-ad8b-9a7fb6a4a58f-istio-podinfo" pod "a40b021f-16e2-4821-ad8b-9a7fb6a4a58f" (UID: "a40b021f-16e2-4821-ad8b-9a7fb6a4a58f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:50:27.009664 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:27.009642 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/559db955-2645-4b32-9070-e87d7bcb485e-istio-envoy" pod "559db955-2645-4b32-9070-e87d7bcb485e" (UID: "559db955-2645-4b32-9070-e87d7bcb485e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:50:27.009711 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:27.009662 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a40b021f-16e2-4821-ad8b-9a7fb6a4a58f-kube-api-access-gvwvd" pod "a40b021f-16e2-4821-ad8b-9a7fb6a4a58f" (UID: "a40b021f-16e2-4821-ad8b-9a7fb6a4a58f"). InnerVolumeSpecName "kube-api-access-gvwvd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:50:27.009711 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:27.009650 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/559db955-2645-4b32-9070-e87d7bcb485e-istio-token" pod "559db955-2645-4b32-9070-e87d7bcb485e" (UID: "559db955-2645-4b32-9070-e87d7bcb485e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:50:27.009791 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:27.009779 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/559db955-2645-4b32-9070-e87d7bcb485e-istio-podinfo" pod "559db955-2645-4b32-9070-e87d7bcb485e" (UID: "559db955-2645-4b32-9070-e87d7bcb485e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:50:27.011241 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:27.011219 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a40b021f-16e2-4821-ad8b-9a7fb6a4a58f-istio-envoy" pod "a40b021f-16e2-4821-ad8b-9a7fb6a4a58f" (UID: "a40b021f-16e2-4821-ad8b-9a7fb6a4a58f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:50:27.108634 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:27.108564 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1-credential-socket\") pod \"1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1\" (UID: \"1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1\") " Oct 30 16:50:27.108804 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:27.108714 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1-istio-podinfo\") pod \"1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1\" (UID: \"1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1\") " Oct 30 16:50:27.108804 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:27.108766 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1-istiod-ca-cert\") pod \"1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1\" (UID: \"1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1\") " Oct 30 16:50:27.108890 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:27.108846 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1-istio-token\" (UniqueName: \"kubernetes.io/projected/1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1-istio-token\") pod \"1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1\" (UID: \"1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1\") " Oct 30 16:50:27.108890 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:27.108858 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1-credential-socket" pod "1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1" (UID: "1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:50:27.108978 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:27.108912 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1-istio-envoy\") pod \"1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1\" (UID: \"1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1\") " Oct 30 16:50:27.108978 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:27.108951 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1-workload-certs\") pod \"1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1\" (UID: \"1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1\") " Oct 30 16:50:27.109089 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:27.108981 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1-istio-data\") pod \"1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1\" (UID: \"1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1\") " Oct 30 16:50:27.109149 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:27.109097 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1-workload-socket\") pod \"1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1\" (UID: \"1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1\") " Oct 30 16:50:27.109211 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:27.109169 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1-kube-api-access-m54j8\" (UniqueName: \"kubernetes.io/projected/1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1-kube-api-access-m54j8\") pod \"1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1\" (UID: \"1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1\") " Oct 30 16:50:27.109276 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:27.109253 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1-istiod-ca-cert" pod "1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1" (UID: "1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:50:27.109467 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:27.109394 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1-workload-certs" pod "1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1" (UID: "1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:50:27.109467 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:27.109430 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1-istio-data" pod "1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1" (UID: "1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:50:27.110127 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:27.110087 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1-workload-socket" pod "1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1" (UID: "1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:50:27.110127 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:27.110103 5052 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/559db955-2645-4b32-9070-e87d7bcb485e-istio-podinfo\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:50:27.110301 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:27.110148 5052 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1-istiod-ca-cert\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:50:27.110301 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:27.110165 5052 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/559db955-2645-4b32-9070-e87d7bcb485e-istio-token\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:50:27.110301 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:27.110180 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvwvd\" (UniqueName: \"kubernetes.io/projected/a40b021f-16e2-4821-ad8b-9a7fb6a4a58f-kube-api-access-gvwvd\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:50:27.110301 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:27.110198 5052 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a40b021f-16e2-4821-ad8b-9a7fb6a4a58f-istio-podinfo\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:50:27.110301 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:27.110214 5052 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a40b021f-16e2-4821-ad8b-9a7fb6a4a58f-istio-token\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:50:27.110301 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:27.110228 5052 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1-workload-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:50:27.110301 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:27.110241 5052 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1-istio-data\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:50:27.110301 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:27.110253 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzs6s\" (UniqueName: \"kubernetes.io/projected/559db955-2645-4b32-9070-e87d7bcb485e-kube-api-access-dzs6s\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:50:27.110301 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:27.110267 5052 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a40b021f-16e2-4821-ad8b-9a7fb6a4a58f-istio-envoy\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:50:27.110301 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:27.110280 5052 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/559db955-2645-4b32-9070-e87d7bcb485e-istio-envoy\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:50:27.110301 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:27.110294 5052 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1-credential-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:50:27.111267 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:27.111216 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1-istio-envoy" pod "1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1" (UID: "1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:50:27.111346 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:27.111276 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1-istio-token" pod "1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1" (UID: "1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:50:27.111346 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:27.111280 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1-istio-podinfo" pod "1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1" (UID: "1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:50:27.111569 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:27.111550 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1-kube-api-access-m54j8" pod "1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1" (UID: "1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1"). InnerVolumeSpecName "kube-api-access-m54j8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:50:27.210886 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:27.210852 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-hff9w"] Oct 30 16:50:27.211971 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:27.211854 5052 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1-istio-podinfo\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:50:27.211971 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:27.211885 5052 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1-istio-token\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:50:27.211971 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:27.211898 5052 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1-istio-envoy\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:50:27.211971 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:27.211929 5052 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1-workload-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:50:27.211971 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:27.211946 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m54j8\" (UniqueName: \"kubernetes.io/projected/1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1-kube-api-access-m54j8\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:50:27.234778 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:27.228468 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-hff9w"] Oct 30 16:50:27.297407 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:27.294320 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-v2jw7"] Oct 30 16:50:27.310727 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:27.309644 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-v2jw7"] Oct 30 16:50:27.882565 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:27.882482 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-f7blz" Oct 30 16:50:27.882565 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:27.882516 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-f7blz" event={"ID":"1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1","Type":"ContainerDied","Data":"e4aea41d7b4555220ff6a2eb543f644dc67a5504e4e2f88d8ee48ce83615b16d"} Oct 30 16:50:27.883136 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:27.882616 5052 scope.go:122] "RemoveContainer" containerID="2d72c42e5bc4e3f1dab3f51c29d6ffad1ad86844ed26aeace4fd738a49c2bfd7" Oct 30 16:50:27.941214 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:27.941188 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-f7blz"] Oct 30 16:50:27.954113 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:27.954080 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-f7blz"] Oct 30 16:50:28.600912 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:28.600876 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1" path="/var/lib/kubelet/pods/1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1/volumes" Oct 30 16:50:28.601411 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:28.601398 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="559db955-2645-4b32-9070-e87d7bcb485e" path="/var/lib/kubelet/pods/559db955-2645-4b32-9070-e87d7bcb485e/volumes" Oct 30 16:50:28.601914 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:28.601901 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a40b021f-16e2-4821-ad8b-9a7fb6a4a58f" path="/var/lib/kubelet/pods/a40b021f-16e2-4821-ad8b-9a7fb6a4a58f/volumes" Oct 30 16:50:31.683224 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:31.683189 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-zhqln"] Oct 30 16:50:31.684003 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:31.683987 5052 cpu_manager.go:414] "containerMap: removing container" podUID="a1e2767a-41a6-453f-8ac7-4bc93b77b143" containerName="payload-pre-processing" Oct 30 16:50:31.684056 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:31.684004 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1e2767a-41a6-453f-8ac7-4bc93b77b143" containerName="payload-pre-processing" Oct 30 16:50:31.684056 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:31.684025 5052 cpu_manager.go:414] "containerMap: removing container" podUID="559db955-2645-4b32-9070-e87d7bcb485e" containerName="istio-proxy" Oct 30 16:50:31.684056 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:31.684031 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="559db955-2645-4b32-9070-e87d7bcb485e" containerName="istio-proxy" Oct 30 16:50:31.684056 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:31.684038 5052 cpu_manager.go:414] "containerMap: removing container" podUID="1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1" containerName="istio-proxy" Oct 30 16:50:31.684056 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:31.684043 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1" containerName="istio-proxy" Oct 30 16:50:31.684222 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:31.684077 5052 cpu_manager.go:414] "containerMap: removing container" podUID="a40b021f-16e2-4821-ad8b-9a7fb6a4a58f" containerName="istio-proxy" Oct 30 16:50:31.684222 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:31.684082 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a40b021f-16e2-4821-ad8b-9a7fb6a4a58f" containerName="istio-proxy" Oct 30 16:50:31.684222 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:31.684185 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="559db955-2645-4b32-9070-e87d7bcb485e" containerName="istio-proxy" Oct 30 16:50:31.684222 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:31.684196 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="a40b021f-16e2-4821-ad8b-9a7fb6a4a58f" containerName="istio-proxy" Oct 30 16:50:31.684222 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:31.684206 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1e2767a-41a6-453f-8ac7-4bc93b77b143" containerName="payload-pre-processing" Oct 30 16:50:31.684222 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:31.684215 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="559db955-2645-4b32-9070-e87d7bcb485e" containerName="istio-proxy" Oct 30 16:50:31.684222 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:31.684221 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a6bcabe-663d-46c8-8c70-9b7ab0bd5db1" containerName="istio-proxy" Oct 30 16:50:31.689963 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:31.689941 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-zhqln" Oct 30 16:50:31.711776 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:31.711751 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-zhqln"] Oct 30 16:50:31.719489 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:31.719450 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-b8af660d-openshift-default-dockercfg-vhvv6\"" Oct 30 16:50:31.792299 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:31.792269 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d9e35b9-801d-4b2d-aba2-c34409d041ad-credential-socket\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-zhqln\" (UID: \"9d9e35b9-801d-4b2d-aba2-c34409d041ad\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-zhqln" Oct 30 16:50:31.792456 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:31.792310 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8n4m\" (UniqueName: \"kubernetes.io/projected/9d9e35b9-801d-4b2d-aba2-c34409d041ad-kube-api-access-t8n4m\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-zhqln\" (UID: \"9d9e35b9-801d-4b2d-aba2-c34409d041ad\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-zhqln" Oct 30 16:50:31.792456 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:31.792405 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d9e35b9-801d-4b2d-aba2-c34409d041ad-istio-token\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-zhqln\" (UID: \"9d9e35b9-801d-4b2d-aba2-c34409d041ad\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-zhqln" Oct 30 16:50:31.792563 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:31.792460 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d9e35b9-801d-4b2d-aba2-c34409d041ad-workload-certs\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-zhqln\" (UID: \"9d9e35b9-801d-4b2d-aba2-c34409d041ad\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-zhqln" Oct 30 16:50:31.792662 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:31.792562 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d9e35b9-801d-4b2d-aba2-c34409d041ad-istio-envoy\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-zhqln\" (UID: \"9d9e35b9-801d-4b2d-aba2-c34409d041ad\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-zhqln" Oct 30 16:50:31.792662 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:31.792643 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d9e35b9-801d-4b2d-aba2-c34409d041ad-istio-data\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-zhqln\" (UID: \"9d9e35b9-801d-4b2d-aba2-c34409d041ad\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-zhqln" Oct 30 16:50:31.792793 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:31.792775 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d9e35b9-801d-4b2d-aba2-c34409d041ad-workload-socket\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-zhqln\" (UID: \"9d9e35b9-801d-4b2d-aba2-c34409d041ad\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-zhqln" Oct 30 16:50:31.792829 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:31.792801 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d9e35b9-801d-4b2d-aba2-c34409d041ad-istio-podinfo\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-zhqln\" (UID: \"9d9e35b9-801d-4b2d-aba2-c34409d041ad\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-zhqln" Oct 30 16:50:31.792886 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:31.792871 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d9e35b9-801d-4b2d-aba2-c34409d041ad-istiod-ca-cert\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-zhqln\" (UID: \"9d9e35b9-801d-4b2d-aba2-c34409d041ad\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-zhqln" Oct 30 16:50:31.894353 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:31.894318 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d9e35b9-801d-4b2d-aba2-c34409d041ad-workload-socket\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-zhqln\" (UID: \"9d9e35b9-801d-4b2d-aba2-c34409d041ad\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-zhqln" Oct 30 16:50:31.894509 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:31.894369 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d9e35b9-801d-4b2d-aba2-c34409d041ad-istio-podinfo\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-zhqln\" (UID: \"9d9e35b9-801d-4b2d-aba2-c34409d041ad\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-zhqln" Oct 30 16:50:31.894509 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:31.894425 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d9e35b9-801d-4b2d-aba2-c34409d041ad-istiod-ca-cert\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-zhqln\" (UID: \"9d9e35b9-801d-4b2d-aba2-c34409d041ad\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-zhqln" Oct 30 16:50:31.894717 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:31.894642 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d9e35b9-801d-4b2d-aba2-c34409d041ad-credential-socket\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-zhqln\" (UID: \"9d9e35b9-801d-4b2d-aba2-c34409d041ad\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-zhqln" Oct 30 16:50:31.894717 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:31.894681 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8n4m\" (UniqueName: \"kubernetes.io/projected/9d9e35b9-801d-4b2d-aba2-c34409d041ad-kube-api-access-t8n4m\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-zhqln\" (UID: \"9d9e35b9-801d-4b2d-aba2-c34409d041ad\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-zhqln" Oct 30 16:50:31.894843 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:31.894717 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d9e35b9-801d-4b2d-aba2-c34409d041ad-istio-token\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-zhqln\" (UID: \"9d9e35b9-801d-4b2d-aba2-c34409d041ad\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-zhqln" Oct 30 16:50:31.894843 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:31.894751 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d9e35b9-801d-4b2d-aba2-c34409d041ad-workload-socket\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-zhqln\" (UID: \"9d9e35b9-801d-4b2d-aba2-c34409d041ad\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-zhqln" Oct 30 16:50:31.894843 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:31.894758 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d9e35b9-801d-4b2d-aba2-c34409d041ad-workload-certs\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-zhqln\" (UID: \"9d9e35b9-801d-4b2d-aba2-c34409d041ad\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-zhqln" Oct 30 16:50:31.894843 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:31.894829 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d9e35b9-801d-4b2d-aba2-c34409d041ad-istio-envoy\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-zhqln\" (UID: \"9d9e35b9-801d-4b2d-aba2-c34409d041ad\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-zhqln" Oct 30 16:50:31.895056 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:31.894874 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d9e35b9-801d-4b2d-aba2-c34409d041ad-istio-data\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-zhqln\" (UID: \"9d9e35b9-801d-4b2d-aba2-c34409d041ad\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-zhqln" Oct 30 16:50:31.895056 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:31.895002 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d9e35b9-801d-4b2d-aba2-c34409d041ad-credential-socket\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-zhqln\" (UID: \"9d9e35b9-801d-4b2d-aba2-c34409d041ad\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-zhqln" Oct 30 16:50:31.895171 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:31.895145 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d9e35b9-801d-4b2d-aba2-c34409d041ad-workload-certs\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-zhqln\" (UID: \"9d9e35b9-801d-4b2d-aba2-c34409d041ad\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-zhqln" Oct 30 16:50:31.895275 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:31.895256 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d9e35b9-801d-4b2d-aba2-c34409d041ad-istio-data\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-zhqln\" (UID: \"9d9e35b9-801d-4b2d-aba2-c34409d041ad\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-zhqln" Oct 30 16:50:31.895338 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:31.895280 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d9e35b9-801d-4b2d-aba2-c34409d041ad-istiod-ca-cert\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-zhqln\" (UID: \"9d9e35b9-801d-4b2d-aba2-c34409d041ad\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-zhqln" Oct 30 16:50:31.897086 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:31.897061 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d9e35b9-801d-4b2d-aba2-c34409d041ad-istio-envoy\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-zhqln\" (UID: \"9d9e35b9-801d-4b2d-aba2-c34409d041ad\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-zhqln" Oct 30 16:50:31.897186 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:31.897067 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d9e35b9-801d-4b2d-aba2-c34409d041ad-istio-podinfo\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-zhqln\" (UID: \"9d9e35b9-801d-4b2d-aba2-c34409d041ad\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-zhqln" Oct 30 16:50:31.922134 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:31.922109 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d9e35b9-801d-4b2d-aba2-c34409d041ad-istio-token\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-zhqln\" (UID: \"9d9e35b9-801d-4b2d-aba2-c34409d041ad\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-zhqln" Oct 30 16:50:31.922671 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:31.922647 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8n4m\" (UniqueName: \"kubernetes.io/projected/9d9e35b9-801d-4b2d-aba2-c34409d041ad-kube-api-access-t8n4m\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-zhqln\" (UID: \"9d9e35b9-801d-4b2d-aba2-c34409d041ad\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-zhqln" Oct 30 16:50:32.005092 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:32.005017 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-zhqln" Oct 30 16:50:32.356653 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:32.356617 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-zhqln"] Oct 30 16:50:32.358135 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:50:32.358108 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d9e35b9_801d_4b2d_aba2_c34409d041ad.slice/crio-d116f570c3f5cdb4c272c568220eb1b154deb483dd1574597626a583361f0981 WatchSource:0}: Error finding container d116f570c3f5cdb4c272c568220eb1b154deb483dd1574597626a583361f0981: Status 404 returned error can't find the container with id d116f570c3f5cdb4c272c568220eb1b154deb483dd1574597626a583361f0981 Oct 30 16:50:32.361356 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:32.361321 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:50:32.361452 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:32.361392 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:50:32.361452 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:32.361427 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:50:32.956566 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:32.952174 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-zhqln" event={"ID":"9d9e35b9-801d-4b2d-aba2-c34409d041ad","Type":"ContainerStarted","Data":"546d7652e7eba58d0c7db12381af0885040996537309f07c9e71cd75e0b85a5e"} Oct 30 16:50:32.956566 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:32.952246 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-zhqln" event={"ID":"9d9e35b9-801d-4b2d-aba2-c34409d041ad","Type":"ContainerStarted","Data":"d116f570c3f5cdb4c272c568220eb1b154deb483dd1574597626a583361f0981"} Oct 30 16:50:33.006173 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:33.006145 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-zhqln" Oct 30 16:50:33.007514 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:33.007482 5052 patch_prober.go:29] interesting pod/e2e-mt-b8af660d-openshift-default-69fbc6b79c-zhqln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Oct 30 16:50:33.007666 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:33.007539 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-zhqln" podUID="9d9e35b9-801d-4b2d-aba2-c34409d041ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Oct 30 16:50:33.017533 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:33.017481 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-zhqln" podStartSLOduration=2.017468521 podStartE2EDuration="2.017468521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:50:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:50:33.015978792 +0000 UTC m=+1788.543314028" watchObservedRunningTime="2026-10-30 16:50:33.017468521 +0000 UTC m=+1788.544803740" Oct 30 16:50:34.005602 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:34.005544 5052 patch_prober.go:29] interesting pod/e2e-mt-b8af660d-openshift-default-69fbc6b79c-zhqln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Oct 30 16:50:34.006000 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:34.005627 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-zhqln" podUID="9d9e35b9-801d-4b2d-aba2-c34409d041ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Oct 30 16:50:35.005975 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:35.005940 5052 patch_prober.go:29] interesting pod/e2e-mt-b8af660d-openshift-default-69fbc6b79c-zhqln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Oct 30 16:50:35.006920 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:35.006874 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-zhqln" podUID="9d9e35b9-801d-4b2d-aba2-c34409d041ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Oct 30 16:50:36.006543 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:36.006511 5052 patch_prober.go:29] interesting pod/e2e-mt-b8af660d-openshift-default-69fbc6b79c-zhqln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Oct 30 16:50:36.006966 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:36.006561 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-zhqln" podUID="9d9e35b9-801d-4b2d-aba2-c34409d041ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Oct 30 16:50:37.022607 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:37.016789 5052 patch_prober.go:29] interesting pod/e2e-mt-b8af660d-openshift-default-69fbc6b79c-zhqln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Oct 30 16:50:37.022607 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:37.016867 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-zhqln" podUID="9d9e35b9-801d-4b2d-aba2-c34409d041ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Oct 30 16:50:38.005766 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:38.005734 5052 patch_prober.go:29] interesting pod/e2e-mt-b8af660d-openshift-default-69fbc6b79c-zhqln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Oct 30 16:50:38.005961 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:38.005792 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-zhqln" podUID="9d9e35b9-801d-4b2d-aba2-c34409d041ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Oct 30 16:50:39.005834 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:39.005794 5052 patch_prober.go:29] interesting pod/e2e-mt-b8af660d-openshift-default-69fbc6b79c-zhqln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Oct 30 16:50:39.006221 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:39.005847 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-zhqln" podUID="9d9e35b9-801d-4b2d-aba2-c34409d041ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Oct 30 16:50:40.010719 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:40.010686 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-zhqln" Oct 30 16:50:40.011197 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:40.011174 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-zhqln" Oct 30 16:50:40.011893 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:40.011877 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-zhqln" Oct 30 16:50:45.401923 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:45.401893 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4wrf8_4f0f46f4-5cda-4d67-b6f3-47df06eb844c/console-operator/2.log" Oct 30 16:50:45.406309 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:45.406284 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4wrf8_4f0f46f4-5cda-4d67-b6f3-47df06eb844c/console-operator/2.log" Oct 30 16:50:46.148026 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:46.147999 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b8af660d-787d8764b-hq7zs"] Oct 30 16:50:46.764233 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:46.764180 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-5552t"] Oct 30 16:50:46.765943 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:46.765916 5052 cpu_manager.go:414] "containerMap: removing container" podUID="559db955-2645-4b32-9070-e87d7bcb485e" containerName="istio-proxy" Oct 30 16:50:46.765943 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:46.765942 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="559db955-2645-4b32-9070-e87d7bcb485e" containerName="istio-proxy" Oct 30 16:50:46.791088 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:46.791064 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-5552t" Oct 30 16:50:46.806474 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:46.806448 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-5552t"] Oct 30 16:50:46.824120 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:46.824094 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-st7mw"] Oct 30 16:50:46.829296 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:46.829275 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-st7mw" Oct 30 16:50:46.954603 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:46.938255 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd769390-6903-4ed3-9ee9-814fb84762e9-istio-envoy\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-st7mw\" (UID: \"dd769390-6903-4ed3-9ee9-814fb84762e9\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-st7mw" Oct 30 16:50:46.954603 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:46.938346 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd769390-6903-4ed3-9ee9-814fb84762e9-istio-token\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-st7mw\" (UID: \"dd769390-6903-4ed3-9ee9-814fb84762e9\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-st7mw" Oct 30 16:50:46.954603 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:46.938415 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a85b25b-991b-4f0d-9ef4-baab2b59d7b9-workload-socket\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-5552t\" (UID: \"0a85b25b-991b-4f0d-9ef4-baab2b59d7b9\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-5552t" Oct 30 16:50:46.954603 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:46.938463 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd769390-6903-4ed3-9ee9-814fb84762e9-workload-certs\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-st7mw\" (UID: \"dd769390-6903-4ed3-9ee9-814fb84762e9\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-st7mw" Oct 30 16:50:46.954603 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:46.938490 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a85b25b-991b-4f0d-9ef4-baab2b59d7b9-istio-token\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-5552t\" (UID: \"0a85b25b-991b-4f0d-9ef4-baab2b59d7b9\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-5552t" Oct 30 16:50:46.954603 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:46.938520 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcthm\" (UniqueName: \"kubernetes.io/projected/0a85b25b-991b-4f0d-9ef4-baab2b59d7b9-kube-api-access-lcthm\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-5552t\" (UID: \"0a85b25b-991b-4f0d-9ef4-baab2b59d7b9\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-5552t" Oct 30 16:50:46.954603 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:46.938563 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd769390-6903-4ed3-9ee9-814fb84762e9-istiod-ca-cert\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-st7mw\" (UID: \"dd769390-6903-4ed3-9ee9-814fb84762e9\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-st7mw" Oct 30 16:50:46.954603 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:46.938683 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a85b25b-991b-4f0d-9ef4-baab2b59d7b9-istio-podinfo\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-5552t\" (UID: \"0a85b25b-991b-4f0d-9ef4-baab2b59d7b9\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-5552t" Oct 30 16:50:46.954603 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:46.938774 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd769390-6903-4ed3-9ee9-814fb84762e9-istio-podinfo\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-st7mw\" (UID: \"dd769390-6903-4ed3-9ee9-814fb84762e9\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-st7mw" Oct 30 16:50:46.954603 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:46.938809 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a85b25b-991b-4f0d-9ef4-baab2b59d7b9-istiod-ca-cert\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-5552t\" (UID: \"0a85b25b-991b-4f0d-9ef4-baab2b59d7b9\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-5552t" Oct 30 16:50:46.954603 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:46.938910 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd769390-6903-4ed3-9ee9-814fb84762e9-workload-socket\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-st7mw\" (UID: \"dd769390-6903-4ed3-9ee9-814fb84762e9\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-st7mw" Oct 30 16:50:46.954603 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:46.938980 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd769390-6903-4ed3-9ee9-814fb84762e9-istio-data\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-st7mw\" (UID: \"dd769390-6903-4ed3-9ee9-814fb84762e9\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-st7mw" Oct 30 16:50:46.955660 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:46.955628 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tb5md\" (UniqueName: \"kubernetes.io/projected/dd769390-6903-4ed3-9ee9-814fb84762e9-kube-api-access-tb5md\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-st7mw\" (UID: \"dd769390-6903-4ed3-9ee9-814fb84762e9\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-st7mw" Oct 30 16:50:46.955866 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:46.955849 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a85b25b-991b-4f0d-9ef4-baab2b59d7b9-workload-certs\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-5552t\" (UID: \"0a85b25b-991b-4f0d-9ef4-baab2b59d7b9\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-5552t" Oct 30 16:50:46.955975 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:46.955960 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a85b25b-991b-4f0d-9ef4-baab2b59d7b9-istio-data\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-5552t\" (UID: \"0a85b25b-991b-4f0d-9ef4-baab2b59d7b9\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-5552t" Oct 30 16:50:46.956117 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:46.956103 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a85b25b-991b-4f0d-9ef4-baab2b59d7b9-credential-socket\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-5552t\" (UID: \"0a85b25b-991b-4f0d-9ef4-baab2b59d7b9\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-5552t" Oct 30 16:50:46.956230 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:46.956216 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd769390-6903-4ed3-9ee9-814fb84762e9-credential-socket\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-st7mw\" (UID: \"dd769390-6903-4ed3-9ee9-814fb84762e9\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-st7mw" Oct 30 16:50:46.956380 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:46.956364 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a85b25b-991b-4f0d-9ef4-baab2b59d7b9-istio-envoy\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-5552t\" (UID: \"0a85b25b-991b-4f0d-9ef4-baab2b59d7b9\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-5552t" Oct 30 16:50:47.120652 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:47.103925 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a85b25b-991b-4f0d-9ef4-baab2b59d7b9-workload-socket\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-5552t\" (UID: \"0a85b25b-991b-4f0d-9ef4-baab2b59d7b9\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-5552t" Oct 30 16:50:47.120652 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:47.104003 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd769390-6903-4ed3-9ee9-814fb84762e9-workload-certs\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-st7mw\" (UID: \"dd769390-6903-4ed3-9ee9-814fb84762e9\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-st7mw" Oct 30 16:50:47.120652 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:47.104074 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a85b25b-991b-4f0d-9ef4-baab2b59d7b9-istio-token\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-5552t\" (UID: \"0a85b25b-991b-4f0d-9ef4-baab2b59d7b9\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-5552t" Oct 30 16:50:47.120652 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:47.104106 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcthm\" (UniqueName: \"kubernetes.io/projected/0a85b25b-991b-4f0d-9ef4-baab2b59d7b9-kube-api-access-lcthm\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-5552t\" (UID: \"0a85b25b-991b-4f0d-9ef4-baab2b59d7b9\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-5552t" Oct 30 16:50:47.120652 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:47.104149 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd769390-6903-4ed3-9ee9-814fb84762e9-istiod-ca-cert\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-st7mw\" (UID: \"dd769390-6903-4ed3-9ee9-814fb84762e9\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-st7mw" Oct 30 16:50:47.120652 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:47.104384 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a85b25b-991b-4f0d-9ef4-baab2b59d7b9-istio-podinfo\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-5552t\" (UID: \"0a85b25b-991b-4f0d-9ef4-baab2b59d7b9\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-5552t" Oct 30 16:50:47.120652 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:47.104427 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd769390-6903-4ed3-9ee9-814fb84762e9-istio-podinfo\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-st7mw\" (UID: \"dd769390-6903-4ed3-9ee9-814fb84762e9\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-st7mw" Oct 30 16:50:47.120652 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:47.104462 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a85b25b-991b-4f0d-9ef4-baab2b59d7b9-istiod-ca-cert\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-5552t\" (UID: \"0a85b25b-991b-4f0d-9ef4-baab2b59d7b9\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-5552t" Oct 30 16:50:47.120652 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:47.104553 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd769390-6903-4ed3-9ee9-814fb84762e9-workload-socket\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-st7mw\" (UID: \"dd769390-6903-4ed3-9ee9-814fb84762e9\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-st7mw" Oct 30 16:50:47.120652 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:47.104643 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd769390-6903-4ed3-9ee9-814fb84762e9-istio-data\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-st7mw\" (UID: \"dd769390-6903-4ed3-9ee9-814fb84762e9\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-st7mw" Oct 30 16:50:47.120652 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:47.104755 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tb5md\" (UniqueName: \"kubernetes.io/projected/dd769390-6903-4ed3-9ee9-814fb84762e9-kube-api-access-tb5md\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-st7mw\" (UID: \"dd769390-6903-4ed3-9ee9-814fb84762e9\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-st7mw" Oct 30 16:50:47.120652 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:47.104799 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a85b25b-991b-4f0d-9ef4-baab2b59d7b9-workload-certs\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-5552t\" (UID: \"0a85b25b-991b-4f0d-9ef4-baab2b59d7b9\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-5552t" Oct 30 16:50:47.120652 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:47.104823 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a85b25b-991b-4f0d-9ef4-baab2b59d7b9-istio-data\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-5552t\" (UID: \"0a85b25b-991b-4f0d-9ef4-baab2b59d7b9\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-5552t" Oct 30 16:50:47.120652 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:47.104877 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a85b25b-991b-4f0d-9ef4-baab2b59d7b9-credential-socket\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-5552t\" (UID: \"0a85b25b-991b-4f0d-9ef4-baab2b59d7b9\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-5552t" Oct 30 16:50:47.120652 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:47.104910 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd769390-6903-4ed3-9ee9-814fb84762e9-credential-socket\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-st7mw\" (UID: \"dd769390-6903-4ed3-9ee9-814fb84762e9\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-st7mw" Oct 30 16:50:47.120652 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:47.104970 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a85b25b-991b-4f0d-9ef4-baab2b59d7b9-istio-envoy\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-5552t\" (UID: \"0a85b25b-991b-4f0d-9ef4-baab2b59d7b9\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-5552t" Oct 30 16:50:47.120652 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:47.105007 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd769390-6903-4ed3-9ee9-814fb84762e9-istio-envoy\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-st7mw\" (UID: \"dd769390-6903-4ed3-9ee9-814fb84762e9\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-st7mw" Oct 30 16:50:47.120652 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:47.105078 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd769390-6903-4ed3-9ee9-814fb84762e9-istio-token\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-st7mw\" (UID: \"dd769390-6903-4ed3-9ee9-814fb84762e9\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-st7mw" Oct 30 16:50:47.120652 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:47.105869 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a85b25b-991b-4f0d-9ef4-baab2b59d7b9-workload-socket\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-5552t\" (UID: \"0a85b25b-991b-4f0d-9ef4-baab2b59d7b9\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-5552t" Oct 30 16:50:47.120652 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:47.106176 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd769390-6903-4ed3-9ee9-814fb84762e9-istio-data\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-st7mw\" (UID: \"dd769390-6903-4ed3-9ee9-814fb84762e9\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-st7mw" Oct 30 16:50:47.121876 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:47.120738 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a85b25b-991b-4f0d-9ef4-baab2b59d7b9-istio-data\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-5552t\" (UID: \"0a85b25b-991b-4f0d-9ef4-baab2b59d7b9\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-5552t" Oct 30 16:50:47.147620 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:47.130422 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-st7mw"] Oct 30 16:50:47.147620 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:47.138201 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a85b25b-991b-4f0d-9ef4-baab2b59d7b9-credential-socket\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-5552t\" (UID: \"0a85b25b-991b-4f0d-9ef4-baab2b59d7b9\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-5552t" Oct 30 16:50:47.147620 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:47.145291 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a85b25b-991b-4f0d-9ef4-baab2b59d7b9-istiod-ca-cert\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-5552t\" (UID: \"0a85b25b-991b-4f0d-9ef4-baab2b59d7b9\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-5552t" Oct 30 16:50:47.161114 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:47.148115 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd769390-6903-4ed3-9ee9-814fb84762e9-credential-socket\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-st7mw\" (UID: \"dd769390-6903-4ed3-9ee9-814fb84762e9\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-st7mw" Oct 30 16:50:47.161114 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:47.148232 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd769390-6903-4ed3-9ee9-814fb84762e9-workload-certs\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-st7mw\" (UID: \"dd769390-6903-4ed3-9ee9-814fb84762e9\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-st7mw" Oct 30 16:50:47.161114 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:47.148344 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd769390-6903-4ed3-9ee9-814fb84762e9-workload-socket\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-st7mw\" (UID: \"dd769390-6903-4ed3-9ee9-814fb84762e9\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-st7mw" Oct 30 16:50:47.161114 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:47.149202 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd769390-6903-4ed3-9ee9-814fb84762e9-istiod-ca-cert\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-st7mw\" (UID: \"dd769390-6903-4ed3-9ee9-814fb84762e9\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-st7mw" Oct 30 16:50:47.161114 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:47.151407 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a85b25b-991b-4f0d-9ef4-baab2b59d7b9-istio-envoy\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-5552t\" (UID: \"0a85b25b-991b-4f0d-9ef4-baab2b59d7b9\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-5552t" Oct 30 16:50:47.161114 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:47.153497 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a85b25b-991b-4f0d-9ef4-baab2b59d7b9-istio-podinfo\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-5552t\" (UID: \"0a85b25b-991b-4f0d-9ef4-baab2b59d7b9\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-5552t" Oct 30 16:50:47.161114 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:47.155762 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a85b25b-991b-4f0d-9ef4-baab2b59d7b9-workload-certs\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-5552t\" (UID: \"0a85b25b-991b-4f0d-9ef4-baab2b59d7b9\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-5552t" Oct 30 16:50:47.167314 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:47.164430 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcthm\" (UniqueName: \"kubernetes.io/projected/0a85b25b-991b-4f0d-9ef4-baab2b59d7b9-kube-api-access-lcthm\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-5552t\" (UID: \"0a85b25b-991b-4f0d-9ef4-baab2b59d7b9\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-5552t" Oct 30 16:50:47.172954 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:47.172075 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tb5md\" (UniqueName: \"kubernetes.io/projected/dd769390-6903-4ed3-9ee9-814fb84762e9-kube-api-access-tb5md\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-st7mw\" (UID: \"dd769390-6903-4ed3-9ee9-814fb84762e9\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-st7mw" Oct 30 16:50:47.178228 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:47.178203 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd769390-6903-4ed3-9ee9-814fb84762e9-istio-token\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-st7mw\" (UID: \"dd769390-6903-4ed3-9ee9-814fb84762e9\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-st7mw" Oct 30 16:50:47.179803 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:47.179760 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a85b25b-991b-4f0d-9ef4-baab2b59d7b9-istio-token\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-5552t\" (UID: \"0a85b25b-991b-4f0d-9ef4-baab2b59d7b9\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-5552t" Oct 30 16:50:47.180737 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:47.180695 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd769390-6903-4ed3-9ee9-814fb84762e9-istio-podinfo\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-st7mw\" (UID: \"dd769390-6903-4ed3-9ee9-814fb84762e9\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-st7mw" Oct 30 16:50:47.181061 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:47.181025 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd769390-6903-4ed3-9ee9-814fb84762e9-istio-envoy\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-st7mw\" (UID: \"dd769390-6903-4ed3-9ee9-814fb84762e9\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-st7mw" Oct 30 16:50:47.409545 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:47.409498 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-5552t" Oct 30 16:50:47.445530 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:47.445357 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-st7mw" Oct 30 16:50:47.870788 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:50:47.864149 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a85b25b_991b_4f0d_9ef4_baab2b59d7b9.slice/crio-d0b4cebf28a2a0e43200e4d7c48ff12850cb0e3a416aaa5b8ac11a501d504d84 WatchSource:0}: Error finding container d0b4cebf28a2a0e43200e4d7c48ff12850cb0e3a416aaa5b8ac11a501d504d84: Status 404 returned error can't find the container with id d0b4cebf28a2a0e43200e4d7c48ff12850cb0e3a416aaa5b8ac11a501d504d84 Oct 30 16:50:47.911482 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:47.911412 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-5552t"] Oct 30 16:50:47.919960 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:47.917663 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:50:47.919960 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:47.917753 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:50:47.919960 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:47.917793 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:50:47.986377 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:50:47.986333 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd769390_6903_4ed3_9ee9_814fb84762e9.slice/crio-1349c3d88ed412195a24c3882e56cb4f41e23243596692eec9f474518e119b20 WatchSource:0}: Error finding container 1349c3d88ed412195a24c3882e56cb4f41e23243596692eec9f474518e119b20: Status 404 returned error can't find the container with id 1349c3d88ed412195a24c3882e56cb4f41e23243596692eec9f474518e119b20 Oct 30 16:50:47.988376 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:47.988348 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-st7mw"] Oct 30 16:50:47.990839 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:47.990800 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:50:47.990967 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:47.990886 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:50:47.991028 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:47.990968 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:50:48.309105 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:48.290900 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-st7mw" event={"ID":"dd769390-6903-4ed3-9ee9-814fb84762e9","Type":"ContainerStarted","Data":"f7eb5729a6832f55a0cd1d2dd4d9b05cb8a6f5a86b32ae79e079b85a74f79144"} Oct 30 16:50:48.309105 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:48.290971 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-st7mw" event={"ID":"dd769390-6903-4ed3-9ee9-814fb84762e9","Type":"ContainerStarted","Data":"1349c3d88ed412195a24c3882e56cb4f41e23243596692eec9f474518e119b20"} Oct 30 16:50:48.309105 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:48.308646 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-5552t" event={"ID":"0a85b25b-991b-4f0d-9ef4-baab2b59d7b9","Type":"ContainerStarted","Data":"39aa2c7cc642c2b8d067e46e1f02243dab4630349de336f264af6342de2de15a"} Oct 30 16:50:48.309105 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:48.308707 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-5552t" event={"ID":"0a85b25b-991b-4f0d-9ef4-baab2b59d7b9","Type":"ContainerStarted","Data":"d0b4cebf28a2a0e43200e4d7c48ff12850cb0e3a416aaa5b8ac11a501d504d84"} Oct 30 16:50:48.366723 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:48.363850 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-st7mw" podStartSLOduration=2.363829758 podStartE2EDuration="2.363829758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:50:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:50:48.362159605 +0000 UTC m=+1803.889494850" watchObservedRunningTime="2026-10-30 16:50:48.363829758 +0000 UTC m=+1803.891164978" Oct 30 16:50:48.410041 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:48.410005 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-5552t" Oct 30 16:50:48.413709 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:48.413654 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-5552t" podStartSLOduration=2.413635515 podStartE2EDuration="2.413635515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:50:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:50:48.412385743 +0000 UTC m=+1803.939720962" watchObservedRunningTime="2026-10-30 16:50:48.413635515 +0000 UTC m=+1803.940970749" Oct 30 16:50:48.453605 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:48.453174 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-st7mw" Oct 30 16:50:49.026630 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:50:49.026131 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 16:50:49.412037 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:49.411999 5052 patch_prober.go:29] interesting pod/e2e-mt-b8af660d-openshift-default-69fbc6b79c-5552t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Oct 30 16:50:49.412251 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:49.412055 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-5552t" podUID="0a85b25b-991b-4f0d-9ef4-baab2b59d7b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Oct 30 16:50:49.451467 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:49.451432 5052 patch_prober.go:29] interesting pod/e2e-mt-b8af660d-openshift-default-69fbc6b79c-st7mw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Oct 30 16:50:49.451773 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:49.451744 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-st7mw" podUID="dd769390-6903-4ed3-9ee9-814fb84762e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Oct 30 16:50:50.410448 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:50.410397 5052 patch_prober.go:29] interesting pod/e2e-mt-b8af660d-openshift-default-69fbc6b79c-5552t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Oct 30 16:50:50.410930 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:50.410459 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-5552t" podUID="0a85b25b-991b-4f0d-9ef4-baab2b59d7b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Oct 30 16:50:50.455600 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:50.454414 5052 patch_prober.go:29] interesting pod/e2e-mt-b8af660d-openshift-default-69fbc6b79c-st7mw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Oct 30 16:50:50.455600 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:50.454479 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-st7mw" podUID="dd769390-6903-4ed3-9ee9-814fb84762e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Oct 30 16:50:51.156606 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:51.144068 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dthnj"] Oct 30 16:50:51.156606 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:51.148760 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-b8af660d-787d8764b-hq7zs" podUID="b9e65258-2d9f-4341-abae-d786c09a6a56" containerName="payload-processing" containerID="cri-o://4112f8da7f8b4120b5371d0eb443db95af671acdd97f308e429d186d604463c8" gracePeriod=25 Oct 30 16:50:51.186604 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:51.168800 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dthnj" podUID="55ab8e2a-2823-4ff1-a391-572ce6586084" containerName="istio-proxy" containerID="cri-o://da5810e192dfc520f72419d10cb863e67ea65086148476efe3e13a5426cada1d" gracePeriod=30 Oct 30 16:50:51.411335 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:51.409919 5052 patch_prober.go:29] interesting pod/e2e-mt-b8af660d-openshift-default-69fbc6b79c-5552t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Oct 30 16:50:51.411335 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:51.409979 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-5552t" podUID="0a85b25b-991b-4f0d-9ef4-baab2b59d7b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Oct 30 16:50:51.415047 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:51.415020 5052 generic.go:356] "Generic (PLEG): container finished" podID="b9e65258-2d9f-4341-abae-d786c09a6a56" containerID="4112f8da7f8b4120b5371d0eb443db95af671acdd97f308e429d186d604463c8" exitCode=0 Oct 30 16:50:51.415268 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:51.415252 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-b8af660d-787d8764b-hq7zs" event={"ID":"b9e65258-2d9f-4341-abae-d786c09a6a56","Type":"ContainerDied","Data":"4112f8da7f8b4120b5371d0eb443db95af671acdd97f308e429d186d604463c8"} Oct 30 16:50:51.452607 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:51.451646 5052 patch_prober.go:29] interesting pod/e2e-mt-b8af660d-openshift-default-69fbc6b79c-st7mw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Oct 30 16:50:51.452607 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:51.451700 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-st7mw" podUID="dd769390-6903-4ed3-9ee9-814fb84762e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Oct 30 16:50:51.847654 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:51.820592 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-529bc7/test-model-529bc7-kserve-69c945bd47-d6dvw"] Oct 30 16:50:51.847654 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:51.826806 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-529bc7/test-model-529bc7-kserve-69c945bd47-d6dvw" Oct 30 16:50:51.869917 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:51.867867 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-529bc7\"/\"kube-root-ca.crt\"" Oct 30 16:50:51.869917 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:51.867984 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-529bc7\"/\"test-model-529bc7-kserve-self-signed-certs\"" Oct 30 16:50:51.869917 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:51.868093 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-529bc7\"/\"default-dockercfg-8fd5n\"" Oct 30 16:50:51.869917 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:51.868146 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-529bc7\"/\"openshift-service-ca.crt\"" Oct 30 16:50:51.997323 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:51.997284 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc5babf8-ffe2-4a65-bce7-b6b9b9cf174b-tmp-dir\") pod \"test-model-529bc7-kserve-69c945bd47-d6dvw\" (UID: \"cc5babf8-ffe2-4a65-bce7-b6b9b9cf174b\") " pod="ai-tenant-e2e-inf-a-529bc7/test-model-529bc7-kserve-69c945bd47-d6dvw" Oct 30 16:50:51.997571 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:51.997553 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2x7nt\" (UniqueName: \"kubernetes.io/projected/cc5babf8-ffe2-4a65-bce7-b6b9b9cf174b-kube-api-access-2x7nt\") pod \"test-model-529bc7-kserve-69c945bd47-d6dvw\" (UID: \"cc5babf8-ffe2-4a65-bce7-b6b9b9cf174b\") " pod="ai-tenant-e2e-inf-a-529bc7/test-model-529bc7-kserve-69c945bd47-d6dvw" Oct 30 16:50:52.015614 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:51.997958 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc5babf8-ffe2-4a65-bce7-b6b9b9cf174b-model-cache\") pod \"test-model-529bc7-kserve-69c945bd47-d6dvw\" (UID: \"cc5babf8-ffe2-4a65-bce7-b6b9b9cf174b\") " pod="ai-tenant-e2e-inf-a-529bc7/test-model-529bc7-kserve-69c945bd47-d6dvw" Oct 30 16:50:52.015948 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:52.015917 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc5babf8-ffe2-4a65-bce7-b6b9b9cf174b-tls-certs\") pod \"test-model-529bc7-kserve-69c945bd47-d6dvw\" (UID: \"cc5babf8-ffe2-4a65-bce7-b6b9b9cf174b\") " pod="ai-tenant-e2e-inf-a-529bc7/test-model-529bc7-kserve-69c945bd47-d6dvw" Oct 30 16:50:52.016131 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:52.016113 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc5babf8-ffe2-4a65-bce7-b6b9b9cf174b-home\") pod \"test-model-529bc7-kserve-69c945bd47-d6dvw\" (UID: \"cc5babf8-ffe2-4a65-bce7-b6b9b9cf174b\") " pod="ai-tenant-e2e-inf-a-529bc7/test-model-529bc7-kserve-69c945bd47-d6dvw" Oct 30 16:50:52.036967 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:52.036927 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc5babf8-ffe2-4a65-bce7-b6b9b9cf174b-dshm\") pod \"test-model-529bc7-kserve-69c945bd47-d6dvw\" (UID: \"cc5babf8-ffe2-4a65-bce7-b6b9b9cf174b\") " pod="ai-tenant-e2e-inf-a-529bc7/test-model-529bc7-kserve-69c945bd47-d6dvw" Oct 30 16:50:52.139457 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:52.139358 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc5babf8-ffe2-4a65-bce7-b6b9b9cf174b-tmp-dir\") pod \"test-model-529bc7-kserve-69c945bd47-d6dvw\" (UID: \"cc5babf8-ffe2-4a65-bce7-b6b9b9cf174b\") " pod="ai-tenant-e2e-inf-a-529bc7/test-model-529bc7-kserve-69c945bd47-d6dvw" Oct 30 16:50:52.139755 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:52.139729 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2x7nt\" (UniqueName: \"kubernetes.io/projected/cc5babf8-ffe2-4a65-bce7-b6b9b9cf174b-kube-api-access-2x7nt\") pod \"test-model-529bc7-kserve-69c945bd47-d6dvw\" (UID: \"cc5babf8-ffe2-4a65-bce7-b6b9b9cf174b\") " pod="ai-tenant-e2e-inf-a-529bc7/test-model-529bc7-kserve-69c945bd47-d6dvw" Oct 30 16:50:52.140059 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:52.140042 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc5babf8-ffe2-4a65-bce7-b6b9b9cf174b-model-cache\") pod \"test-model-529bc7-kserve-69c945bd47-d6dvw\" (UID: \"cc5babf8-ffe2-4a65-bce7-b6b9b9cf174b\") " pod="ai-tenant-e2e-inf-a-529bc7/test-model-529bc7-kserve-69c945bd47-d6dvw" Oct 30 16:50:52.140199 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:52.140184 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc5babf8-ffe2-4a65-bce7-b6b9b9cf174b-tls-certs\") pod \"test-model-529bc7-kserve-69c945bd47-d6dvw\" (UID: \"cc5babf8-ffe2-4a65-bce7-b6b9b9cf174b\") " pod="ai-tenant-e2e-inf-a-529bc7/test-model-529bc7-kserve-69c945bd47-d6dvw" Oct 30 16:50:52.140313 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:52.140300 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc5babf8-ffe2-4a65-bce7-b6b9b9cf174b-home\") pod \"test-model-529bc7-kserve-69c945bd47-d6dvw\" (UID: \"cc5babf8-ffe2-4a65-bce7-b6b9b9cf174b\") " pod="ai-tenant-e2e-inf-a-529bc7/test-model-529bc7-kserve-69c945bd47-d6dvw" Oct 30 16:50:52.140469 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:52.140457 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc5babf8-ffe2-4a65-bce7-b6b9b9cf174b-dshm\") pod \"test-model-529bc7-kserve-69c945bd47-d6dvw\" (UID: \"cc5babf8-ffe2-4a65-bce7-b6b9b9cf174b\") " pod="ai-tenant-e2e-inf-a-529bc7/test-model-529bc7-kserve-69c945bd47-d6dvw" Oct 30 16:50:52.143712 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:52.141488 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc5babf8-ffe2-4a65-bce7-b6b9b9cf174b-model-cache\") pod \"test-model-529bc7-kserve-69c945bd47-d6dvw\" (UID: \"cc5babf8-ffe2-4a65-bce7-b6b9b9cf174b\") " pod="ai-tenant-e2e-inf-a-529bc7/test-model-529bc7-kserve-69c945bd47-d6dvw" Oct 30 16:50:52.196650 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:52.172729 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc5babf8-ffe2-4a65-bce7-b6b9b9cf174b-tmp-dir\") pod \"test-model-529bc7-kserve-69c945bd47-d6dvw\" (UID: \"cc5babf8-ffe2-4a65-bce7-b6b9b9cf174b\") " pod="ai-tenant-e2e-inf-a-529bc7/test-model-529bc7-kserve-69c945bd47-d6dvw" Oct 30 16:50:52.216679 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:52.216632 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc5babf8-ffe2-4a65-bce7-b6b9b9cf174b-home\") pod \"test-model-529bc7-kserve-69c945bd47-d6dvw\" (UID: \"cc5babf8-ffe2-4a65-bce7-b6b9b9cf174b\") " pod="ai-tenant-e2e-inf-a-529bc7/test-model-529bc7-kserve-69c945bd47-d6dvw" Oct 30 16:50:52.262109 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:52.262066 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-529bc7/test-model-529bc7-kserve-69c945bd47-d6dvw"] Oct 30 16:50:52.310291 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:52.298918 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc5babf8-ffe2-4a65-bce7-b6b9b9cf174b-dshm\") pod \"test-model-529bc7-kserve-69c945bd47-d6dvw\" (UID: \"cc5babf8-ffe2-4a65-bce7-b6b9b9cf174b\") " pod="ai-tenant-e2e-inf-a-529bc7/test-model-529bc7-kserve-69c945bd47-d6dvw" Oct 30 16:50:52.320261 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:52.319183 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc5babf8-ffe2-4a65-bce7-b6b9b9cf174b-tls-certs\") pod \"test-model-529bc7-kserve-69c945bd47-d6dvw\" (UID: \"cc5babf8-ffe2-4a65-bce7-b6b9b9cf174b\") " pod="ai-tenant-e2e-inf-a-529bc7/test-model-529bc7-kserve-69c945bd47-d6dvw" Oct 30 16:50:52.412873 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:52.412841 5052 patch_prober.go:29] interesting pod/e2e-mt-b8af660d-openshift-default-69fbc6b79c-5552t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Oct 30 16:50:52.442296 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:52.442237 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-5552t" podUID="0a85b25b-991b-4f0d-9ef4-baab2b59d7b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Oct 30 16:50:52.472449 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:52.472415 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2x7nt\" (UniqueName: \"kubernetes.io/projected/cc5babf8-ffe2-4a65-bce7-b6b9b9cf174b-kube-api-access-2x7nt\") pod \"test-model-529bc7-kserve-69c945bd47-d6dvw\" (UID: \"cc5babf8-ffe2-4a65-bce7-b6b9b9cf174b\") " pod="ai-tenant-e2e-inf-a-529bc7/test-model-529bc7-kserve-69c945bd47-d6dvw" Oct 30 16:50:52.496980 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:52.495798 5052 patch_prober.go:29] interesting pod/e2e-mt-b8af660d-openshift-default-69fbc6b79c-st7mw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Oct 30 16:50:52.496980 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:52.496926 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-st7mw" podUID="dd769390-6903-4ed3-9ee9-814fb84762e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Oct 30 16:50:52.562319 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:52.562283 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-b8af660d-787d8764b-hq7zs" event={"ID":"b9e65258-2d9f-4341-abae-d786c09a6a56","Type":"ContainerDied","Data":"7a4a3232481c37db391d23da9299abe24826a2ffa21a2255b762c535aaffdff3"} Oct 30 16:50:52.562656 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:52.562636 5052 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a4a3232481c37db391d23da9299abe24826a2ffa21a2255b762c535aaffdff3" Oct 30 16:50:52.563427 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:52.563406 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-529bc7/test-model-529bc7-kserve-69c945bd47-d6dvw" Oct 30 16:50:52.563892 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:52.563876 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-b8af660d-787d8764b-hq7zs" Oct 30 16:50:52.688818 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:52.688718 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-750dcb/distinct-w2-750dcb-kserve-86bbdfcbb-bpn77" podUID="c45190af-d795-4e13-8f2f-670b2c597658" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:50:52.783690 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:52.783004 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d693d0/distinct-2-w6-d693d0-kserve-6498fdc474-dpbjj" podUID="4b98b8ff-72cd-4ddb-a4b8-67078162e611" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:50:52.797664 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:52.792859 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx" podUID="bd662f3a-659f-477c-b8d5-94c64288c0c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:50:52.811641 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:52.802152 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/distinct-w5-967fac-kserve-69cbcccd8b-b85bn" podUID="8f6b9354-378f-4451-9ff7-be7f7fec12f2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:50:52.827765 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:52.826652 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9e65258-2d9f-4341-abae-d786c09a6a56-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9e65258-2d9f-4341-abae-d786c09a6a56-plugins-config-volume\") pod \"b9e65258-2d9f-4341-abae-d786c09a6a56\" (UID: \"b9e65258-2d9f-4341-abae-d786c09a6a56\") " Oct 30 16:50:52.827765 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:52.827070 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9e65258-2d9f-4341-abae-d786c09a6a56-kube-api-access-h29kl\" (UniqueName: \"kubernetes.io/projected/b9e65258-2d9f-4341-abae-d786c09a6a56-kube-api-access-h29kl\") pod \"b9e65258-2d9f-4341-abae-d786c09a6a56\" (UID: \"b9e65258-2d9f-4341-abae-d786c09a6a56\") " Oct 30 16:50:52.911607 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:52.910817 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9e65258-2d9f-4341-abae-d786c09a6a56-plugins-config-volume" pod "b9e65258-2d9f-4341-abae-d786c09a6a56" (UID: "b9e65258-2d9f-4341-abae-d786c09a6a56"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:50:52.992670 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:52.957323 5052 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9e65258-2d9f-4341-abae-d786c09a6a56-plugins-config-volume\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:50:53.007818 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:53.007761 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9e65258-2d9f-4341-abae-d786c09a6a56-kube-api-access-h29kl" pod "b9e65258-2d9f-4341-abae-d786c09a6a56" (UID: "b9e65258-2d9f-4341-abae-d786c09a6a56"). InnerVolumeSpecName "kube-api-access-h29kl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:50:53.110866 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:53.107659 5052 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-750dcb/distinct-w2-750dcb-kserve-86bbdfcbb-bpn77" podUID="c45190af-d795-4e13-8f2f-670b2c597658" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": context deadline exceeded" Oct 30 16:50:53.117488 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:53.117447 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h29kl\" (UniqueName: \"kubernetes.io/projected/b9e65258-2d9f-4341-abae-d786c09a6a56-kube-api-access-h29kl\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:50:53.426103 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:53.426058 5052 patch_prober.go:29] interesting pod/e2e-mt-b8af660d-openshift-default-69fbc6b79c-5552t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Oct 30 16:50:53.426661 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:53.426628 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-5552t" podUID="0a85b25b-991b-4f0d-9ef4-baab2b59d7b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Oct 30 16:50:53.483870 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:53.474672 5052 patch_prober.go:29] interesting pod/e2e-mt-b8af660d-openshift-default-69fbc6b79c-st7mw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Oct 30 16:50:53.483870 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:53.474731 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-st7mw" podUID="dd769390-6903-4ed3-9ee9-814fb84762e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Oct 30 16:50:53.889606 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:53.856239 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-b8af660d-787d8764b-hq7zs" Oct 30 16:50:54.054633 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:54.042618 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-529bc7/test-model-529bc7-kserve-69c945bd47-d6dvw"] Oct 30 16:50:54.160321 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:54.153549 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b8af660d-787d8764b-hq7zs"] Oct 30 16:50:54.207431 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:54.207396 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-x4nqz"] Oct 30 16:50:54.207939 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:54.207907 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-x4nqz" podUID="8dbfcef1-dea9-4360-abd2-d489a86811a3" containerName="istio-proxy" containerID="cri-o://11aa7be13d10e30dc7825024f06221482480bb9e2b6d913788ba250ef8577020" gracePeriod=30 Oct 30 16:50:54.214809 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:54.214779 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b8af660d-787d8764b-hq7zs"] Oct 30 16:50:54.291254 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:54.291218 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b8af660d-6697cd894d-nlcxt"] Oct 30 16:50:54.291720 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:54.291689 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b8af660d-6697cd894d-nlcxt" podUID="2c10ebe7-ea42-47e1-b384-03dedbe51daa" containerName="maas-api" containerID="cri-o://ee24d3dd37eb61d18960faa6453e3a9e8ec90d657e7c0438418d4e1f1c2535ef" gracePeriod=30 Oct 30 16:50:54.410872 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:54.410810 5052 patch_prober.go:29] interesting pod/e2e-mt-b8af660d-openshift-default-69fbc6b79c-5552t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Oct 30 16:50:54.410872 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:54.410857 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-5552t" podUID="0a85b25b-991b-4f0d-9ef4-baab2b59d7b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Oct 30 16:50:54.482681 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:54.448674 5052 patch_prober.go:29] interesting pod/e2e-mt-b8af660d-openshift-default-69fbc6b79c-st7mw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Oct 30 16:50:54.482681 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:54.448729 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-st7mw" podUID="dd769390-6903-4ed3-9ee9-814fb84762e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Oct 30 16:50:54.609655 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:54.609615 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9e65258-2d9f-4341-abae-d786c09a6a56" path="/var/lib/kubelet/pods/b9e65258-2d9f-4341-abae-d786c09a6a56/volumes" Oct 30 16:50:55.181182 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:55.177137 5052 generic.go:356] "Generic (PLEG): container finished" podID="2c10ebe7-ea42-47e1-b384-03dedbe51daa" containerID="ee24d3dd37eb61d18960faa6453e3a9e8ec90d657e7c0438418d4e1f1c2535ef" exitCode=0 Oct 30 16:50:55.181182 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:55.177383 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b8af660d-6697cd894d-nlcxt" event={"ID":"2c10ebe7-ea42-47e1-b384-03dedbe51daa","Type":"ContainerDied","Data":"ee24d3dd37eb61d18960faa6453e3a9e8ec90d657e7c0438418d4e1f1c2535ef"} Oct 30 16:50:55.181182 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:55.180821 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-529bc7/test-model-529bc7-kserve-69c945bd47-d6dvw" event={"ID":"cc5babf8-ffe2-4a65-bce7-b6b9b9cf174b","Type":"ContainerStarted","Data":"3e8351e9c73edc47c1fc4b415cec79f713c1023a081acb7ab3e31dbcf5d29d32"} Oct 30 16:50:55.181182 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:55.180865 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-529bc7/test-model-529bc7-kserve-69c945bd47-d6dvw" event={"ID":"cc5babf8-ffe2-4a65-bce7-b6b9b9cf174b","Type":"ContainerStarted","Data":"6f9ed158d720b562aad8a7796d63c61e9e5ce3bf6d286208f504475605d440bb"} Oct 30 16:50:55.258882 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:55.258819 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-529bc7/test-model-529bc7-kserve-69c945bd47-d6dvw" podStartSLOduration=4.258800552 podStartE2EDuration="4.258800552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:50:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:50:55.255709601 +0000 UTC m=+1810.783044820" watchObservedRunningTime="2026-10-30 16:50:55.258800552 +0000 UTC m=+1810.786135771" Oct 30 16:50:55.306440 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:55.305042 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b8af660d-6697cd894d-nlcxt" Oct 30 16:50:55.375627 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:55.375291 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c10ebe7-ea42-47e1-b384-03dedbe51daa-kube-api-access-8xbnr\" (UniqueName: \"kubernetes.io/projected/2c10ebe7-ea42-47e1-b384-03dedbe51daa-kube-api-access-8xbnr\") pod \"2c10ebe7-ea42-47e1-b384-03dedbe51daa\" (UID: \"2c10ebe7-ea42-47e1-b384-03dedbe51daa\") " Oct 30 16:50:55.375627 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:55.375454 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c10ebe7-ea42-47e1-b384-03dedbe51daa-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c10ebe7-ea42-47e1-b384-03dedbe51daa-metrics-certs\") pod \"2c10ebe7-ea42-47e1-b384-03dedbe51daa\" (UID: \"2c10ebe7-ea42-47e1-b384-03dedbe51daa\") " Oct 30 16:50:55.375627 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:55.375514 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c10ebe7-ea42-47e1-b384-03dedbe51daa-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c10ebe7-ea42-47e1-b384-03dedbe51daa-openshift-service-ca\") pod \"2c10ebe7-ea42-47e1-b384-03dedbe51daa\" (UID: \"2c10ebe7-ea42-47e1-b384-03dedbe51daa\") " Oct 30 16:50:55.375627 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:55.375550 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c10ebe7-ea42-47e1-b384-03dedbe51daa-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c10ebe7-ea42-47e1-b384-03dedbe51daa-maas-api-tls\") pod \"2c10ebe7-ea42-47e1-b384-03dedbe51daa\" (UID: \"2c10ebe7-ea42-47e1-b384-03dedbe51daa\") " Oct 30 16:50:55.378429 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:55.378379 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c10ebe7-ea42-47e1-b384-03dedbe51daa-openshift-service-ca" pod "2c10ebe7-ea42-47e1-b384-03dedbe51daa" (UID: "2c10ebe7-ea42-47e1-b384-03dedbe51daa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:50:55.438758 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:55.412067 5052 patch_prober.go:29] interesting pod/e2e-mt-b8af660d-openshift-default-69fbc6b79c-5552t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Oct 30 16:50:55.438758 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:55.412136 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-5552t" podUID="0a85b25b-991b-4f0d-9ef4-baab2b59d7b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Oct 30 16:50:55.446536 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:55.445801 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c10ebe7-ea42-47e1-b384-03dedbe51daa-maas-api-tls" pod "2c10ebe7-ea42-47e1-b384-03dedbe51daa" (UID: "2c10ebe7-ea42-47e1-b384-03dedbe51daa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:50:55.446536 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:55.445949 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c10ebe7-ea42-47e1-b384-03dedbe51daa-kube-api-access-8xbnr" pod "2c10ebe7-ea42-47e1-b384-03dedbe51daa" (UID: "2c10ebe7-ea42-47e1-b384-03dedbe51daa"). InnerVolumeSpecName "kube-api-access-8xbnr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:50:55.446941 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:55.446831 5052 patch_prober.go:29] interesting pod/e2e-mt-b8af660d-openshift-default-69fbc6b79c-st7mw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Oct 30 16:50:55.447070 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:55.446986 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-st7mw" podUID="dd769390-6903-4ed3-9ee9-814fb84762e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Oct 30 16:50:55.494645 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:55.480624 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c10ebe7-ea42-47e1-b384-03dedbe51daa-metrics-certs" pod "2c10ebe7-ea42-47e1-b384-03dedbe51daa" (UID: "2c10ebe7-ea42-47e1-b384-03dedbe51daa"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:50:55.494645 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:55.490667 5052 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c10ebe7-ea42-47e1-b384-03dedbe51daa-openshift-service-ca\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:50:55.494645 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:55.490733 5052 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c10ebe7-ea42-47e1-b384-03dedbe51daa-maas-api-tls\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:50:55.494645 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:55.490753 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xbnr\" (UniqueName: \"kubernetes.io/projected/2c10ebe7-ea42-47e1-b384-03dedbe51daa-kube-api-access-8xbnr\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:50:55.563973 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:55.563934 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-529bc7/test-model-529bc7-kserve-69c945bd47-d6dvw" Oct 30 16:50:55.592285 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:55.592253 5052 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c10ebe7-ea42-47e1-b384-03dedbe51daa-metrics-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:50:56.193889 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:56.193849 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b8af660d-6697cd894d-nlcxt" Oct 30 16:50:56.193889 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:56.193870 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b8af660d-6697cd894d-nlcxt" event={"ID":"2c10ebe7-ea42-47e1-b384-03dedbe51daa","Type":"ContainerDied","Data":"53d4fcd0df277579806f97c17383b86e6270aec0f07094688062b6422922be4b"} Oct 30 16:50:56.194160 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:56.193941 5052 scope.go:122] "RemoveContainer" containerID="ee24d3dd37eb61d18960faa6453e3a9e8ec90d657e7c0438418d4e1f1c2535ef" Oct 30 16:50:56.255722 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:56.255685 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b8af660d-6697cd894d-nlcxt"] Oct 30 16:50:56.266057 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:56.266026 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b8af660d-6697cd894d-nlcxt"] Oct 30 16:50:56.627914 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:56.627831 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c10ebe7-ea42-47e1-b384-03dedbe51daa" path="/var/lib/kubelet/pods/2c10ebe7-ea42-47e1-b384-03dedbe51daa/volumes" Oct 30 16:50:56.725260 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:56.725220 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dthnj" Oct 30 16:50:56.760180 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:56.759455 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-5552t" Oct 30 16:50:56.760180 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:56.759983 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-5552t" Oct 30 16:50:56.793318 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:56.793278 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55ab8e2a-2823-4ff1-a391-572ce6586084-istio-token\" (UniqueName: \"kubernetes.io/projected/55ab8e2a-2823-4ff1-a391-572ce6586084-istio-token\") pod \"55ab8e2a-2823-4ff1-a391-572ce6586084\" (UID: \"55ab8e2a-2823-4ff1-a391-572ce6586084\") " Oct 30 16:50:56.793489 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:56.793335 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55ab8e2a-2823-4ff1-a391-572ce6586084-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55ab8e2a-2823-4ff1-a391-572ce6586084-workload-socket\") pod \"55ab8e2a-2823-4ff1-a391-572ce6586084\" (UID: \"55ab8e2a-2823-4ff1-a391-572ce6586084\") " Oct 30 16:50:56.793489 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:56.793373 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55ab8e2a-2823-4ff1-a391-572ce6586084-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55ab8e2a-2823-4ff1-a391-572ce6586084-workload-certs\") pod \"55ab8e2a-2823-4ff1-a391-572ce6586084\" (UID: \"55ab8e2a-2823-4ff1-a391-572ce6586084\") " Oct 30 16:50:56.793489 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:56.793401 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55ab8e2a-2823-4ff1-a391-572ce6586084-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55ab8e2a-2823-4ff1-a391-572ce6586084-istio-envoy\") pod \"55ab8e2a-2823-4ff1-a391-572ce6586084\" (UID: \"55ab8e2a-2823-4ff1-a391-572ce6586084\") " Oct 30 16:50:56.793489 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:56.793449 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55ab8e2a-2823-4ff1-a391-572ce6586084-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55ab8e2a-2823-4ff1-a391-572ce6586084-istiod-ca-cert\") pod \"55ab8e2a-2823-4ff1-a391-572ce6586084\" (UID: \"55ab8e2a-2823-4ff1-a391-572ce6586084\") " Oct 30 16:50:56.793860 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:56.793562 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55ab8e2a-2823-4ff1-a391-572ce6586084-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55ab8e2a-2823-4ff1-a391-572ce6586084-istio-data\") pod \"55ab8e2a-2823-4ff1-a391-572ce6586084\" (UID: \"55ab8e2a-2823-4ff1-a391-572ce6586084\") " Oct 30 16:50:56.793860 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:56.793690 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55ab8e2a-2823-4ff1-a391-572ce6586084-kube-api-access-vtmtb\" (UniqueName: \"kubernetes.io/projected/55ab8e2a-2823-4ff1-a391-572ce6586084-kube-api-access-vtmtb\") pod \"55ab8e2a-2823-4ff1-a391-572ce6586084\" (UID: \"55ab8e2a-2823-4ff1-a391-572ce6586084\") " Oct 30 16:50:56.793860 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:56.793742 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55ab8e2a-2823-4ff1-a391-572ce6586084-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55ab8e2a-2823-4ff1-a391-572ce6586084-credential-socket\") pod \"55ab8e2a-2823-4ff1-a391-572ce6586084\" (UID: \"55ab8e2a-2823-4ff1-a391-572ce6586084\") " Oct 30 16:50:56.793860 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:56.793832 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/55ab8e2a-2823-4ff1-a391-572ce6586084-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55ab8e2a-2823-4ff1-a391-572ce6586084-istio-podinfo\") pod \"55ab8e2a-2823-4ff1-a391-572ce6586084\" (UID: \"55ab8e2a-2823-4ff1-a391-572ce6586084\") " Oct 30 16:50:56.795350 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:56.795325 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55ab8e2a-2823-4ff1-a391-572ce6586084-credential-socket" pod "55ab8e2a-2823-4ff1-a391-572ce6586084" (UID: "55ab8e2a-2823-4ff1-a391-572ce6586084"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:50:56.796236 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:56.796178 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55ab8e2a-2823-4ff1-a391-572ce6586084-workload-certs" pod "55ab8e2a-2823-4ff1-a391-572ce6586084" (UID: "55ab8e2a-2823-4ff1-a391-572ce6586084"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:50:56.796341 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:56.796271 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55ab8e2a-2823-4ff1-a391-572ce6586084-istio-data" pod "55ab8e2a-2823-4ff1-a391-572ce6586084" (UID: "55ab8e2a-2823-4ff1-a391-572ce6586084"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:50:56.796642 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:56.796597 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55ab8e2a-2823-4ff1-a391-572ce6586084-istiod-ca-cert" pod "55ab8e2a-2823-4ff1-a391-572ce6586084" (UID: "55ab8e2a-2823-4ff1-a391-572ce6586084"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:50:56.796795 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:56.796766 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55ab8e2a-2823-4ff1-a391-572ce6586084-workload-socket" pod "55ab8e2a-2823-4ff1-a391-572ce6586084" (UID: "55ab8e2a-2823-4ff1-a391-572ce6586084"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:50:56.798469 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:56.798344 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55ab8e2a-2823-4ff1-a391-572ce6586084-kube-api-access-vtmtb" pod "55ab8e2a-2823-4ff1-a391-572ce6586084" (UID: "55ab8e2a-2823-4ff1-a391-572ce6586084"). InnerVolumeSpecName "kube-api-access-vtmtb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:50:56.800565 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:56.800528 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/55ab8e2a-2823-4ff1-a391-572ce6586084-istio-podinfo" pod "55ab8e2a-2823-4ff1-a391-572ce6586084" (UID: "55ab8e2a-2823-4ff1-a391-572ce6586084"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:50:56.804048 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:56.804020 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55ab8e2a-2823-4ff1-a391-572ce6586084-istio-token" pod "55ab8e2a-2823-4ff1-a391-572ce6586084" (UID: "55ab8e2a-2823-4ff1-a391-572ce6586084"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:50:56.805105 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:56.805081 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55ab8e2a-2823-4ff1-a391-572ce6586084-istio-envoy" pod "55ab8e2a-2823-4ff1-a391-572ce6586084" (UID: "55ab8e2a-2823-4ff1-a391-572ce6586084"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:50:56.806300 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:56.806279 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-5552t" Oct 30 16:50:56.844953 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:56.843626 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-st7mw" Oct 30 16:50:56.845322 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:56.845147 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-st7mw" Oct 30 16:50:56.847573 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:56.847521 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-st7mw" Oct 30 16:50:56.901240 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:56.901212 5052 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55ab8e2a-2823-4ff1-a391-572ce6586084-istiod-ca-cert\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:50:56.901240 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:56.901240 5052 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55ab8e2a-2823-4ff1-a391-572ce6586084-istio-data\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:50:56.901447 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:56.901254 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtmtb\" (UniqueName: \"kubernetes.io/projected/55ab8e2a-2823-4ff1-a391-572ce6586084-kube-api-access-vtmtb\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:50:56.901447 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:56.901267 5052 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55ab8e2a-2823-4ff1-a391-572ce6586084-credential-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:50:56.901447 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:56.901281 5052 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55ab8e2a-2823-4ff1-a391-572ce6586084-istio-podinfo\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:50:56.901447 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:56.901292 5052 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55ab8e2a-2823-4ff1-a391-572ce6586084-istio-token\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:50:56.901447 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:56.901305 5052 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55ab8e2a-2823-4ff1-a391-572ce6586084-workload-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:50:56.901447 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:56.901317 5052 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55ab8e2a-2823-4ff1-a391-572ce6586084-workload-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:50:56.901447 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:56.901328 5052 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55ab8e2a-2823-4ff1-a391-572ce6586084-istio-envoy\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:50:57.205440 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:57.205342 5052 generic.go:356] "Generic (PLEG): container finished" podID="55ab8e2a-2823-4ff1-a391-572ce6586084" containerID="da5810e192dfc520f72419d10cb863e67ea65086148476efe3e13a5426cada1d" exitCode=0 Oct 30 16:50:57.205440 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:57.205396 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dthnj" Oct 30 16:50:57.205751 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:57.205425 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dthnj" event={"ID":"55ab8e2a-2823-4ff1-a391-572ce6586084","Type":"ContainerDied","Data":"da5810e192dfc520f72419d10cb863e67ea65086148476efe3e13a5426cada1d"} Oct 30 16:50:57.205751 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:57.205500 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dthnj" event={"ID":"55ab8e2a-2823-4ff1-a391-572ce6586084","Type":"ContainerDied","Data":"cd1956d9fb1bb537f0e839ef55b2f30083d8414219ac3e7613cad66424a4d69d"} Oct 30 16:50:57.205751 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:57.205536 5052 scope.go:122] "RemoveContainer" containerID="da5810e192dfc520f72419d10cb863e67ea65086148476efe3e13a5426cada1d" Oct 30 16:50:57.229455 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:57.228536 5052 scope.go:122] "RemoveContainer" containerID="da5810e192dfc520f72419d10cb863e67ea65086148476efe3e13a5426cada1d" Oct 30 16:50:57.230268 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:50:57.230190 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da5810e192dfc520f72419d10cb863e67ea65086148476efe3e13a5426cada1d\": container with ID starting with da5810e192dfc520f72419d10cb863e67ea65086148476efe3e13a5426cada1d not found: ID does not exist" containerID="da5810e192dfc520f72419d10cb863e67ea65086148476efe3e13a5426cada1d" Oct 30 16:50:57.230268 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:57.230230 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da5810e192dfc520f72419d10cb863e67ea65086148476efe3e13a5426cada1d"} err="failed to get container status \"da5810e192dfc520f72419d10cb863e67ea65086148476efe3e13a5426cada1d\": rpc error: code = NotFound desc = could not find container \"da5810e192dfc520f72419d10cb863e67ea65086148476efe3e13a5426cada1d\": container with ID starting with da5810e192dfc520f72419d10cb863e67ea65086148476efe3e13a5426cada1d not found: ID does not exist" Oct 30 16:50:57.265052 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:57.265021 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dthnj"] Oct 30 16:50:57.274040 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:57.274011 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dthnj"] Oct 30 16:50:58.599823 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:58.599780 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55ab8e2a-2823-4ff1-a391-572ce6586084" path="/var/lib/kubelet/pods/55ab8e2a-2823-4ff1-a391-572ce6586084/volumes" Oct 30 16:50:59.516300 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:59.516238 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-x4nqz" Oct 30 16:50:59.573033 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:59.572992 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8dbfcef1-dea9-4360-abd2-d489a86811a3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8dbfcef1-dea9-4360-abd2-d489a86811a3-istio-envoy\") pod \"8dbfcef1-dea9-4360-abd2-d489a86811a3\" (UID: \"8dbfcef1-dea9-4360-abd2-d489a86811a3\") " Oct 30 16:50:59.573194 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:59.573094 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8dbfcef1-dea9-4360-abd2-d489a86811a3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8dbfcef1-dea9-4360-abd2-d489a86811a3-istiod-ca-cert\") pod \"8dbfcef1-dea9-4360-abd2-d489a86811a3\" (UID: \"8dbfcef1-dea9-4360-abd2-d489a86811a3\") " Oct 30 16:50:59.573194 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:59.573116 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8dbfcef1-dea9-4360-abd2-d489a86811a3-istio-token\" (UniqueName: \"kubernetes.io/projected/8dbfcef1-dea9-4360-abd2-d489a86811a3-istio-token\") pod \"8dbfcef1-dea9-4360-abd2-d489a86811a3\" (UID: \"8dbfcef1-dea9-4360-abd2-d489a86811a3\") " Oct 30 16:50:59.573194 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:59.573173 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8dbfcef1-dea9-4360-abd2-d489a86811a3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8dbfcef1-dea9-4360-abd2-d489a86811a3-workload-certs\") pod \"8dbfcef1-dea9-4360-abd2-d489a86811a3\" (UID: \"8dbfcef1-dea9-4360-abd2-d489a86811a3\") " Oct 30 16:50:59.573461 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:59.573435 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8dbfcef1-dea9-4360-abd2-d489a86811a3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dbfcef1-dea9-4360-abd2-d489a86811a3-workload-socket\") pod \"8dbfcef1-dea9-4360-abd2-d489a86811a3\" (UID: \"8dbfcef1-dea9-4360-abd2-d489a86811a3\") " Oct 30 16:50:59.573620 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:59.573504 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8dbfcef1-dea9-4360-abd2-d489a86811a3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dbfcef1-dea9-4360-abd2-d489a86811a3-credential-socket\") pod \"8dbfcef1-dea9-4360-abd2-d489a86811a3\" (UID: \"8dbfcef1-dea9-4360-abd2-d489a86811a3\") " Oct 30 16:50:59.573620 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:59.573568 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8dbfcef1-dea9-4360-abd2-d489a86811a3-kube-api-access-fnj7j\" (UniqueName: \"kubernetes.io/projected/8dbfcef1-dea9-4360-abd2-d489a86811a3-kube-api-access-fnj7j\") pod \"8dbfcef1-dea9-4360-abd2-d489a86811a3\" (UID: \"8dbfcef1-dea9-4360-abd2-d489a86811a3\") " Oct 30 16:50:59.573755 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:59.573650 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8dbfcef1-dea9-4360-abd2-d489a86811a3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8dbfcef1-dea9-4360-abd2-d489a86811a3-istio-data\") pod \"8dbfcef1-dea9-4360-abd2-d489a86811a3\" (UID: \"8dbfcef1-dea9-4360-abd2-d489a86811a3\") " Oct 30 16:50:59.573755 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:59.573723 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8dbfcef1-dea9-4360-abd2-d489a86811a3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8dbfcef1-dea9-4360-abd2-d489a86811a3-istio-podinfo\") pod \"8dbfcef1-dea9-4360-abd2-d489a86811a3\" (UID: \"8dbfcef1-dea9-4360-abd2-d489a86811a3\") " Oct 30 16:50:59.573859 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:59.573501 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dbfcef1-dea9-4360-abd2-d489a86811a3-workload-certs" pod "8dbfcef1-dea9-4360-abd2-d489a86811a3" (UID: "8dbfcef1-dea9-4360-abd2-d489a86811a3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:50:59.573859 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:59.573507 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8dbfcef1-dea9-4360-abd2-d489a86811a3-istiod-ca-cert" pod "8dbfcef1-dea9-4360-abd2-d489a86811a3" (UID: "8dbfcef1-dea9-4360-abd2-d489a86811a3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:50:59.573859 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:59.573747 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dbfcef1-dea9-4360-abd2-d489a86811a3-workload-socket" pod "8dbfcef1-dea9-4360-abd2-d489a86811a3" (UID: "8dbfcef1-dea9-4360-abd2-d489a86811a3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:50:59.573859 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:59.573762 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dbfcef1-dea9-4360-abd2-d489a86811a3-credential-socket" pod "8dbfcef1-dea9-4360-abd2-d489a86811a3" (UID: "8dbfcef1-dea9-4360-abd2-d489a86811a3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:50:59.574094 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:59.573986 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dbfcef1-dea9-4360-abd2-d489a86811a3-istio-data" pod "8dbfcef1-dea9-4360-abd2-d489a86811a3" (UID: "8dbfcef1-dea9-4360-abd2-d489a86811a3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:50:59.574570 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:59.574546 5052 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8dbfcef1-dea9-4360-abd2-d489a86811a3-istiod-ca-cert\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:50:59.574715 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:59.574574 5052 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8dbfcef1-dea9-4360-abd2-d489a86811a3-workload-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:50:59.574715 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:59.574603 5052 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dbfcef1-dea9-4360-abd2-d489a86811a3-workload-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:50:59.574715 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:59.574618 5052 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dbfcef1-dea9-4360-abd2-d489a86811a3-credential-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:50:59.574715 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:59.574632 5052 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8dbfcef1-dea9-4360-abd2-d489a86811a3-istio-data\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:50:59.575540 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:59.575514 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dbfcef1-dea9-4360-abd2-d489a86811a3-istio-envoy" pod "8dbfcef1-dea9-4360-abd2-d489a86811a3" (UID: "8dbfcef1-dea9-4360-abd2-d489a86811a3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:50:59.575704 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:59.575685 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dbfcef1-dea9-4360-abd2-d489a86811a3-istio-token" pod "8dbfcef1-dea9-4360-abd2-d489a86811a3" (UID: "8dbfcef1-dea9-4360-abd2-d489a86811a3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:50:59.575851 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:59.575825 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dbfcef1-dea9-4360-abd2-d489a86811a3-kube-api-access-fnj7j" pod "8dbfcef1-dea9-4360-abd2-d489a86811a3" (UID: "8dbfcef1-dea9-4360-abd2-d489a86811a3"). InnerVolumeSpecName "kube-api-access-fnj7j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:50:59.576382 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:59.576357 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8dbfcef1-dea9-4360-abd2-d489a86811a3-istio-podinfo" pod "8dbfcef1-dea9-4360-abd2-d489a86811a3" (UID: "8dbfcef1-dea9-4360-abd2-d489a86811a3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:50:59.676215 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:59.676177 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnj7j\" (UniqueName: \"kubernetes.io/projected/8dbfcef1-dea9-4360-abd2-d489a86811a3-kube-api-access-fnj7j\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:50:59.676215 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:59.676210 5052 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8dbfcef1-dea9-4360-abd2-d489a86811a3-istio-podinfo\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:50:59.676215 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:59.676220 5052 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8dbfcef1-dea9-4360-abd2-d489a86811a3-istio-envoy\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:50:59.676755 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:50:59.676228 5052 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8dbfcef1-dea9-4360-abd2-d489a86811a3-istio-token\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:51:00.273555 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:00.273469 5052 generic.go:356] "Generic (PLEG): container finished" podID="8dbfcef1-dea9-4360-abd2-d489a86811a3" containerID="11aa7be13d10e30dc7825024f06221482480bb9e2b6d913788ba250ef8577020" exitCode=0 Oct 30 16:51:00.273555 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:00.273522 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-x4nqz" Oct 30 16:51:00.273857 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:00.273550 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-x4nqz" event={"ID":"8dbfcef1-dea9-4360-abd2-d489a86811a3","Type":"ContainerDied","Data":"11aa7be13d10e30dc7825024f06221482480bb9e2b6d913788ba250ef8577020"} Oct 30 16:51:00.273857 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:00.273633 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-x4nqz" event={"ID":"8dbfcef1-dea9-4360-abd2-d489a86811a3","Type":"ContainerDied","Data":"4101738a563d18fd1e226d6a9ed4ef0b5f5bced184232780f03a640d20408bc3"} Oct 30 16:51:00.273857 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:00.273657 5052 scope.go:122] "RemoveContainer" containerID="11aa7be13d10e30dc7825024f06221482480bb9e2b6d913788ba250ef8577020" Oct 30 16:51:00.275512 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:00.275485 5052 generic.go:356] "Generic (PLEG): container finished" podID="fe851cb7-f6e1-4baf-b75e-69018da63e35" containerID="2e130b1db7c8d1085dd61c563edc847dfa62b1e77499d4cdd0050cc8604b1682" exitCode=0 Oct 30 16:51:00.275689 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:00.275551 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3ee682-openshift-default-5f7887cb79-6tsdx" event={"ID":"fe851cb7-f6e1-4baf-b75e-69018da63e35","Type":"ContainerDied","Data":"2e130b1db7c8d1085dd61c563edc847dfa62b1e77499d4cdd0050cc8604b1682"} Oct 30 16:51:00.276062 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:00.276046 5052 scope.go:122] "RemoveContainer" containerID="2e130b1db7c8d1085dd61c563edc847dfa62b1e77499d4cdd0050cc8604b1682" Oct 30 16:51:00.277791 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:00.277760 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:51:00.277893 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:00.277829 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:51:00.277893 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:00.277867 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:51:00.286028 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:00.286012 5052 scope.go:122] "RemoveContainer" containerID="11aa7be13d10e30dc7825024f06221482480bb9e2b6d913788ba250ef8577020" Oct 30 16:51:00.286311 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:51:00.286289 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11aa7be13d10e30dc7825024f06221482480bb9e2b6d913788ba250ef8577020\": container with ID starting with 11aa7be13d10e30dc7825024f06221482480bb9e2b6d913788ba250ef8577020 not found: ID does not exist" containerID="11aa7be13d10e30dc7825024f06221482480bb9e2b6d913788ba250ef8577020" Oct 30 16:51:00.286369 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:00.286323 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11aa7be13d10e30dc7825024f06221482480bb9e2b6d913788ba250ef8577020"} err="failed to get container status \"11aa7be13d10e30dc7825024f06221482480bb9e2b6d913788ba250ef8577020\": rpc error: code = NotFound desc = could not find container \"11aa7be13d10e30dc7825024f06221482480bb9e2b6d913788ba250ef8577020\": container with ID starting with 11aa7be13d10e30dc7825024f06221482480bb9e2b6d913788ba250ef8577020 not found: ID does not exist" Oct 30 16:51:00.400502 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:00.397027 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-x4nqz"] Oct 30 16:51:00.404043 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:00.404012 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-x4nqz"] Oct 30 16:51:00.603504 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:00.603409 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8dbfcef1-dea9-4360-abd2-d489a86811a3" path="/var/lib/kubelet/pods/8dbfcef1-dea9-4360-abd2-d489a86811a3/volumes" Oct 30 16:51:01.286752 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:01.286717 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3ee682-openshift-default-5f7887cb79-6tsdx" event={"ID":"fe851cb7-f6e1-4baf-b75e-69018da63e35","Type":"ContainerStarted","Data":"e7214d0e42e8d6965f3d213fc9d90a7c8d038188814f10218c5a77a5a9a8b6d2"} Oct 30 16:51:01.367903 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:01.367875 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-3ee682-openshift-default-5f7887cb79-6tsdx" Oct 30 16:51:01.368865 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:01.368843 5052 patch_prober.go:29] interesting pod/e2e-inf-b-3ee682-openshift-default-5f7887cb79-6tsdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 30 16:51:01.368973 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:01.368887 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3ee682-openshift-default-5f7887cb79-6tsdx" podUID="fe851cb7-f6e1-4baf-b75e-69018da63e35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 30 16:51:02.367923 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:02.367889 5052 patch_prober.go:29] interesting pod/e2e-inf-b-3ee682-openshift-default-5f7887cb79-6tsdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 30 16:51:02.368328 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:02.367954 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3ee682-openshift-default-5f7887cb79-6tsdx" podUID="fe851cb7-f6e1-4baf-b75e-69018da63e35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 30 16:51:02.563818 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:02.563780 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-529bc7/test-model-529bc7-kserve-69c945bd47-d6dvw" Oct 30 16:51:02.577305 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:02.577279 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-529bc7/test-model-529bc7-kserve-69c945bd47-d6dvw" Oct 30 16:51:03.313132 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:03.313103 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-529bc7/test-model-529bc7-kserve-69c945bd47-d6dvw" Oct 30 16:51:03.368672 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:03.368630 5052 patch_prober.go:29] interesting pod/e2e-inf-b-3ee682-openshift-default-5f7887cb79-6tsdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 30 16:51:03.369091 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:03.368699 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3ee682-openshift-default-5f7887cb79-6tsdx" podUID="fe851cb7-f6e1-4baf-b75e-69018da63e35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 30 16:51:04.372607 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:04.372340 5052 patch_prober.go:29] interesting pod/e2e-inf-b-3ee682-openshift-default-5f7887cb79-6tsdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 30 16:51:04.372607 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:04.372415 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3ee682-openshift-default-5f7887cb79-6tsdx" podUID="fe851cb7-f6e1-4baf-b75e-69018da63e35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 30 16:51:05.368329 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:05.368290 5052 patch_prober.go:29] interesting pod/e2e-inf-b-3ee682-openshift-default-5f7887cb79-6tsdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 30 16:51:05.368503 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:05.368352 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3ee682-openshift-default-5f7887cb79-6tsdx" podUID="fe851cb7-f6e1-4baf-b75e-69018da63e35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 30 16:51:05.908374 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:05.908343 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-3ee682-openshift-default-5f7887cb79-6tsdx" Oct 30 16:51:06.368320 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:06.368228 5052 patch_prober.go:29] interesting pod/e2e-inf-b-3ee682-openshift-default-5f7887cb79-6tsdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 30 16:51:06.368320 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:06.368286 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3ee682-openshift-default-5f7887cb79-6tsdx" podUID="fe851cb7-f6e1-4baf-b75e-69018da63e35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 30 16:51:07.368813 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:07.368778 5052 patch_prober.go:29] interesting pod/e2e-inf-b-3ee682-openshift-default-5f7887cb79-6tsdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 30 16:51:07.369237 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:07.368835 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3ee682-openshift-default-5f7887cb79-6tsdx" podUID="fe851cb7-f6e1-4baf-b75e-69018da63e35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 30 16:51:08.373945 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:08.373917 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-3ee682-openshift-default-5f7887cb79-6tsdx" Oct 30 16:51:08.375185 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:08.375161 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-3ee682-openshift-default-5f7887cb79-6tsdx" Oct 30 16:51:10.324729 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:10.324691 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cdwcj"] Oct 30 16:51:18.268102 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:18.268059 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-3ee682/test-model-3ee682-kserve-7c6cd66757-dm6lb"] Oct 30 16:51:18.269790 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:18.269767 5052 cpu_manager.go:414] "containerMap: removing container" podUID="8dbfcef1-dea9-4360-abd2-d489a86811a3" containerName="istio-proxy" Oct 30 16:51:18.269940 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:18.269927 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8dbfcef1-dea9-4360-abd2-d489a86811a3" containerName="istio-proxy" Oct 30 16:51:18.270066 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:18.270055 5052 cpu_manager.go:414] "containerMap: removing container" podUID="2c10ebe7-ea42-47e1-b384-03dedbe51daa" containerName="maas-api" Oct 30 16:51:18.270162 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:18.270151 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c10ebe7-ea42-47e1-b384-03dedbe51daa" containerName="maas-api" Oct 30 16:51:18.270298 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:18.270288 5052 cpu_manager.go:414] "containerMap: removing container" podUID="55ab8e2a-2823-4ff1-a391-572ce6586084" containerName="istio-proxy" Oct 30 16:51:18.270374 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:18.270364 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55ab8e2a-2823-4ff1-a391-572ce6586084" containerName="istio-proxy" Oct 30 16:51:18.270466 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:18.270456 5052 cpu_manager.go:414] "containerMap: removing container" podUID="b9e65258-2d9f-4341-abae-d786c09a6a56" containerName="payload-processing" Oct 30 16:51:18.270569 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:18.270529 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9e65258-2d9f-4341-abae-d786c09a6a56" containerName="payload-processing" Oct 30 16:51:18.270925 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:18.270910 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="8dbfcef1-dea9-4360-abd2-d489a86811a3" containerName="istio-proxy" Oct 30 16:51:18.271025 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:18.271016 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c10ebe7-ea42-47e1-b384-03dedbe51daa" containerName="maas-api" Oct 30 16:51:18.271114 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:18.271105 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="55ab8e2a-2823-4ff1-a391-572ce6586084" containerName="istio-proxy" Oct 30 16:51:18.271192 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:18.271182 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9e65258-2d9f-4341-abae-d786c09a6a56" containerName="payload-processing" Oct 30 16:51:18.278972 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:18.278947 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-3ee682/test-model-3ee682-kserve-7c6cd66757-dm6lb" Oct 30 16:51:18.290511 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:18.290487 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-3ee682/test-model-3ee682-kserve-7c6cd66757-dm6lb"] Oct 30 16:51:18.313416 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:18.313392 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-3ee682\"/\"openshift-service-ca.crt\"" Oct 30 16:51:18.313714 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:18.313692 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-3ee682\"/\"kube-root-ca.crt\"" Oct 30 16:51:18.375963 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:18.375938 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-3ee682\"/\"default-dockercfg-kjppk\"" Oct 30 16:51:18.376204 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:18.376183 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-3ee682\"/\"test-model-3ee682-kserve-self-signed-certs\"" Oct 30 16:51:18.378244 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:18.378214 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a6bd706-03cb-4f01-a5ac-ffdd2757f279-home\") pod \"test-model-3ee682-kserve-7c6cd66757-dm6lb\" (UID: \"4a6bd706-03cb-4f01-a5ac-ffdd2757f279\") " pod="ai-tenant-e2e-inf-b-3ee682/test-model-3ee682-kserve-7c6cd66757-dm6lb" Oct 30 16:51:18.378530 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:18.378510 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a6bd706-03cb-4f01-a5ac-ffdd2757f279-dshm\") pod \"test-model-3ee682-kserve-7c6cd66757-dm6lb\" (UID: \"4a6bd706-03cb-4f01-a5ac-ffdd2757f279\") " pod="ai-tenant-e2e-inf-b-3ee682/test-model-3ee682-kserve-7c6cd66757-dm6lb" Oct 30 16:51:18.378732 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:18.378712 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a6bd706-03cb-4f01-a5ac-ffdd2757f279-tls-certs\") pod \"test-model-3ee682-kserve-7c6cd66757-dm6lb\" (UID: \"4a6bd706-03cb-4f01-a5ac-ffdd2757f279\") " pod="ai-tenant-e2e-inf-b-3ee682/test-model-3ee682-kserve-7c6cd66757-dm6lb" Oct 30 16:51:18.378893 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:18.378877 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a6bd706-03cb-4f01-a5ac-ffdd2757f279-model-cache\") pod \"test-model-3ee682-kserve-7c6cd66757-dm6lb\" (UID: \"4a6bd706-03cb-4f01-a5ac-ffdd2757f279\") " pod="ai-tenant-e2e-inf-b-3ee682/test-model-3ee682-kserve-7c6cd66757-dm6lb" Oct 30 16:51:18.379015 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:18.378999 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpggj\" (UniqueName: \"kubernetes.io/projected/4a6bd706-03cb-4f01-a5ac-ffdd2757f279-kube-api-access-mpggj\") pod \"test-model-3ee682-kserve-7c6cd66757-dm6lb\" (UID: \"4a6bd706-03cb-4f01-a5ac-ffdd2757f279\") " pod="ai-tenant-e2e-inf-b-3ee682/test-model-3ee682-kserve-7c6cd66757-dm6lb" Oct 30 16:51:18.379212 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:18.379196 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a6bd706-03cb-4f01-a5ac-ffdd2757f279-tmp-dir\") pod \"test-model-3ee682-kserve-7c6cd66757-dm6lb\" (UID: \"4a6bd706-03cb-4f01-a5ac-ffdd2757f279\") " pod="ai-tenant-e2e-inf-b-3ee682/test-model-3ee682-kserve-7c6cd66757-dm6lb" Oct 30 16:51:18.480835 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:18.480800 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a6bd706-03cb-4f01-a5ac-ffdd2757f279-home\") pod \"test-model-3ee682-kserve-7c6cd66757-dm6lb\" (UID: \"4a6bd706-03cb-4f01-a5ac-ffdd2757f279\") " pod="ai-tenant-e2e-inf-b-3ee682/test-model-3ee682-kserve-7c6cd66757-dm6lb" Oct 30 16:51:18.481002 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:18.480934 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a6bd706-03cb-4f01-a5ac-ffdd2757f279-dshm\") pod \"test-model-3ee682-kserve-7c6cd66757-dm6lb\" (UID: \"4a6bd706-03cb-4f01-a5ac-ffdd2757f279\") " pod="ai-tenant-e2e-inf-b-3ee682/test-model-3ee682-kserve-7c6cd66757-dm6lb" Oct 30 16:51:18.481002 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:18.480994 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a6bd706-03cb-4f01-a5ac-ffdd2757f279-tls-certs\") pod \"test-model-3ee682-kserve-7c6cd66757-dm6lb\" (UID: \"4a6bd706-03cb-4f01-a5ac-ffdd2757f279\") " pod="ai-tenant-e2e-inf-b-3ee682/test-model-3ee682-kserve-7c6cd66757-dm6lb" Oct 30 16:51:18.481078 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:18.481043 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a6bd706-03cb-4f01-a5ac-ffdd2757f279-model-cache\") pod \"test-model-3ee682-kserve-7c6cd66757-dm6lb\" (UID: \"4a6bd706-03cb-4f01-a5ac-ffdd2757f279\") " pod="ai-tenant-e2e-inf-b-3ee682/test-model-3ee682-kserve-7c6cd66757-dm6lb" Oct 30 16:51:18.481078 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:18.481070 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpggj\" (UniqueName: \"kubernetes.io/projected/4a6bd706-03cb-4f01-a5ac-ffdd2757f279-kube-api-access-mpggj\") pod \"test-model-3ee682-kserve-7c6cd66757-dm6lb\" (UID: \"4a6bd706-03cb-4f01-a5ac-ffdd2757f279\") " pod="ai-tenant-e2e-inf-b-3ee682/test-model-3ee682-kserve-7c6cd66757-dm6lb" Oct 30 16:51:18.481192 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:18.481176 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a6bd706-03cb-4f01-a5ac-ffdd2757f279-tmp-dir\") pod \"test-model-3ee682-kserve-7c6cd66757-dm6lb\" (UID: \"4a6bd706-03cb-4f01-a5ac-ffdd2757f279\") " pod="ai-tenant-e2e-inf-b-3ee682/test-model-3ee682-kserve-7c6cd66757-dm6lb" Oct 30 16:51:18.481243 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:18.481226 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a6bd706-03cb-4f01-a5ac-ffdd2757f279-home\") pod \"test-model-3ee682-kserve-7c6cd66757-dm6lb\" (UID: \"4a6bd706-03cb-4f01-a5ac-ffdd2757f279\") " pod="ai-tenant-e2e-inf-b-3ee682/test-model-3ee682-kserve-7c6cd66757-dm6lb" Oct 30 16:51:18.481553 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:18.481527 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a6bd706-03cb-4f01-a5ac-ffdd2757f279-tmp-dir\") pod \"test-model-3ee682-kserve-7c6cd66757-dm6lb\" (UID: \"4a6bd706-03cb-4f01-a5ac-ffdd2757f279\") " pod="ai-tenant-e2e-inf-b-3ee682/test-model-3ee682-kserve-7c6cd66757-dm6lb" Oct 30 16:51:18.481783 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:18.481762 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a6bd706-03cb-4f01-a5ac-ffdd2757f279-model-cache\") pod \"test-model-3ee682-kserve-7c6cd66757-dm6lb\" (UID: \"4a6bd706-03cb-4f01-a5ac-ffdd2757f279\") " pod="ai-tenant-e2e-inf-b-3ee682/test-model-3ee682-kserve-7c6cd66757-dm6lb" Oct 30 16:51:18.483161 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:18.483142 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a6bd706-03cb-4f01-a5ac-ffdd2757f279-dshm\") pod \"test-model-3ee682-kserve-7c6cd66757-dm6lb\" (UID: \"4a6bd706-03cb-4f01-a5ac-ffdd2757f279\") " pod="ai-tenant-e2e-inf-b-3ee682/test-model-3ee682-kserve-7c6cd66757-dm6lb" Oct 30 16:51:18.483912 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:18.483889 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a6bd706-03cb-4f01-a5ac-ffdd2757f279-tls-certs\") pod \"test-model-3ee682-kserve-7c6cd66757-dm6lb\" (UID: \"4a6bd706-03cb-4f01-a5ac-ffdd2757f279\") " pod="ai-tenant-e2e-inf-b-3ee682/test-model-3ee682-kserve-7c6cd66757-dm6lb" Oct 30 16:51:18.506708 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:18.506685 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpggj\" (UniqueName: \"kubernetes.io/projected/4a6bd706-03cb-4f01-a5ac-ffdd2757f279-kube-api-access-mpggj\") pod \"test-model-3ee682-kserve-7c6cd66757-dm6lb\" (UID: \"4a6bd706-03cb-4f01-a5ac-ffdd2757f279\") " pod="ai-tenant-e2e-inf-b-3ee682/test-model-3ee682-kserve-7c6cd66757-dm6lb" Oct 30 16:51:18.599856 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:18.599785 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-3ee682/test-model-3ee682-kserve-7c6cd66757-dm6lb" Oct 30 16:51:19.075538 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:51:19.075496 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 16:51:19.337224 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:51:19.337184 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4a6bd706_03cb_4f01_a5ac_ffdd2757f279.slice/crio-5d3d1e51969e41b34e2c2ff5dee68536325469f6bf2cfa24a367ba0b91f2105a WatchSource:0}: Error finding container 5d3d1e51969e41b34e2c2ff5dee68536325469f6bf2cfa24a367ba0b91f2105a: Status 404 returned error can't find the container with id 5d3d1e51969e41b34e2c2ff5dee68536325469f6bf2cfa24a367ba0b91f2105a Oct 30 16:51:19.337695 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:19.337601 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-3ee682/test-model-3ee682-kserve-7c6cd66757-dm6lb"] Oct 30 16:51:19.877842 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:19.877794 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-3ee682/test-model-3ee682-kserve-7c6cd66757-dm6lb" event={"ID":"4a6bd706-03cb-4f01-a5ac-ffdd2757f279","Type":"ContainerStarted","Data":"98faf4b68d5dbf67c77767a9d36ce4e6ecbd182b39788ca216a0aa59d8e362d4"} Oct 30 16:51:19.878067 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:19.877879 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-3ee682/test-model-3ee682-kserve-7c6cd66757-dm6lb" event={"ID":"4a6bd706-03cb-4f01-a5ac-ffdd2757f279","Type":"ContainerStarted","Data":"5d3d1e51969e41b34e2c2ff5dee68536325469f6bf2cfa24a367ba0b91f2105a"} Oct 30 16:51:19.923450 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:19.923381 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-3ee682/test-model-3ee682-kserve-7c6cd66757-dm6lb" podStartSLOduration=1.923362207 podStartE2EDuration="1.923362207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:51:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:51:19.921554377 +0000 UTC m=+1835.448889597" watchObservedRunningTime="2026-10-30 16:51:19.923362207 +0000 UTC m=+1835.450697429" Oct 30 16:51:20.601337 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:20.601308 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-3ee682/test-model-3ee682-kserve-7c6cd66757-dm6lb" Oct 30 16:51:24.964086 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:24.964053 5052 generic.go:356] "Generic (PLEG): container finished" podID="ba6262c9-b3ea-4c61-bb84-4f60954d02a0" containerID="32ccfb376ada51e1c34cb823a8a1cc25e29e84c2523661656fa4b3b14e66ec53" exitCode=0 Oct 30 16:51:24.964518 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:24.964125 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ff1189-openshift-default-7d9f9f5c65-jjdbw" event={"ID":"ba6262c9-b3ea-4c61-bb84-4f60954d02a0","Type":"ContainerDied","Data":"32ccfb376ada51e1c34cb823a8a1cc25e29e84c2523661656fa4b3b14e66ec53"} Oct 30 16:51:24.964574 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:24.964560 5052 scope.go:122] "RemoveContainer" containerID="32ccfb376ada51e1c34cb823a8a1cc25e29e84c2523661656fa4b3b14e66ec53" Oct 30 16:51:24.966435 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:24.966407 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:51:24.966532 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:24.966490 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:51:24.966532 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:24.966523 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:51:25.106947 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:25.106915 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-480a60-7c447cb756-6ct9j"] Oct 30 16:51:25.112131 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:25.112108 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-480a60-7c447cb756-6ct9j" Oct 30 16:51:25.130765 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:25.130736 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-480a60-7c447cb756-6ct9j"] Oct 30 16:51:25.161620 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:25.152975 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-480a60-dockercfg-xcxng\"" Oct 30 16:51:25.161620 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:25.153092 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-480a60\"" Oct 30 16:51:25.379654 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:25.277058 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-445bh\" (UniqueName: \"kubernetes.io/projected/3b3da2b1-c132-453a-b8e9-512a43e613ff-kube-api-access-445bh\") pod \"payload-processing-e2e-inf-b-480a60-7c447cb756-6ct9j\" (UID: \"3b3da2b1-c132-453a-b8e9-512a43e613ff\") " pod="openshift-ingress/payload-processing-e2e-inf-b-480a60-7c447cb756-6ct9j" Oct 30 16:51:25.379654 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:25.277110 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b3da2b1-c132-453a-b8e9-512a43e613ff-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-480a60-7c447cb756-6ct9j\" (UID: \"3b3da2b1-c132-453a-b8e9-512a43e613ff\") " pod="openshift-ingress/payload-processing-e2e-inf-b-480a60-7c447cb756-6ct9j" Oct 30 16:51:25.379654 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:25.379017 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b3da2b1-c132-453a-b8e9-512a43e613ff-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-480a60-7c447cb756-6ct9j\" (UID: \"3b3da2b1-c132-453a-b8e9-512a43e613ff\") " pod="openshift-ingress/payload-processing-e2e-inf-b-480a60-7c447cb756-6ct9j" Oct 30 16:51:25.379654 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:25.379485 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-445bh\" (UniqueName: \"kubernetes.io/projected/3b3da2b1-c132-453a-b8e9-512a43e613ff-kube-api-access-445bh\") pod \"payload-processing-e2e-inf-b-480a60-7c447cb756-6ct9j\" (UID: \"3b3da2b1-c132-453a-b8e9-512a43e613ff\") " pod="openshift-ingress/payload-processing-e2e-inf-b-480a60-7c447cb756-6ct9j" Oct 30 16:51:25.417606 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:25.380716 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b3da2b1-c132-453a-b8e9-512a43e613ff-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-480a60-7c447cb756-6ct9j\" (UID: \"3b3da2b1-c132-453a-b8e9-512a43e613ff\") " pod="openshift-ingress/payload-processing-e2e-inf-b-480a60-7c447cb756-6ct9j" Oct 30 16:51:25.523475 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:25.523441 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-445bh\" (UniqueName: \"kubernetes.io/projected/3b3da2b1-c132-453a-b8e9-512a43e613ff-kube-api-access-445bh\") pod \"payload-processing-e2e-inf-b-480a60-7c447cb756-6ct9j\" (UID: \"3b3da2b1-c132-453a-b8e9-512a43e613ff\") " pod="openshift-ingress/payload-processing-e2e-inf-b-480a60-7c447cb756-6ct9j" Oct 30 16:51:25.730041 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:25.729997 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-480a60-7c447cb756-6ct9j" Oct 30 16:51:26.003653 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:26.003553 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ff1189-openshift-default-7d9f9f5c65-jjdbw" event={"ID":"ba6262c9-b3ea-4c61-bb84-4f60954d02a0","Type":"ContainerStarted","Data":"4e8706eab0d61e2df4b675f8a4d18d470b42ce6fa1fcb61ad1ba4ba12d340e54"} Oct 30 16:51:26.062416 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:51:26.062385 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b3da2b1_c132_453a_b8e9_512a43e613ff.slice/crio-d8aaf5c925e4a9ba6e7f4d1baa25dda32fcf84acb9b967e01bc64a4c50a99117 WatchSource:0}: Error finding container d8aaf5c925e4a9ba6e7f4d1baa25dda32fcf84acb9b967e01bc64a4c50a99117: Status 404 returned error can't find the container with id d8aaf5c925e4a9ba6e7f4d1baa25dda32fcf84acb9b967e01bc64a4c50a99117 Oct 30 16:51:26.063529 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:26.063503 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-480a60-7c447cb756-6ct9j"] Oct 30 16:51:26.420149 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:26.420118 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-ff1189-openshift-default-7d9f9f5c65-jjdbw" Oct 30 16:51:26.420925 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:26.420898 5052 patch_prober.go:29] interesting pod/e2e-inf-a-ff1189-openshift-default-7d9f9f5c65-jjdbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Oct 30 16:51:26.421048 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:26.420955 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ff1189-openshift-default-7d9f9f5c65-jjdbw" podUID="ba6262c9-b3ea-4c61-bb84-4f60954d02a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Oct 30 16:51:27.012307 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:27.012264 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-480a60-7c447cb756-6ct9j" event={"ID":"3b3da2b1-c132-453a-b8e9-512a43e613ff","Type":"ContainerStarted","Data":"945f122630787d3d009e107bdd51c4b40193e6395f1aacc12a034321f9458c32"} Oct 30 16:51:27.012772 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:27.012332 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-480a60-7c447cb756-6ct9j" event={"ID":"3b3da2b1-c132-453a-b8e9-512a43e613ff","Type":"ContainerStarted","Data":"d8aaf5c925e4a9ba6e7f4d1baa25dda32fcf84acb9b967e01bc64a4c50a99117"} Oct 30 16:51:27.062763 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:27.062697 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-480a60-7c447cb756-6ct9j" podStartSLOduration=2.062683297 podStartE2EDuration="2.062683297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:51:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:51:27.059404377 +0000 UTC m=+1842.586739597" watchObservedRunningTime="2026-10-30 16:51:27.062683297 +0000 UTC m=+1842.590018515" Oct 30 16:51:27.419741 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:27.419711 5052 patch_prober.go:29] interesting pod/e2e-inf-a-ff1189-openshift-default-7d9f9f5c65-jjdbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Oct 30 16:51:27.419918 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:27.419762 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ff1189-openshift-default-7d9f9f5c65-jjdbw" podUID="ba6262c9-b3ea-4c61-bb84-4f60954d02a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Oct 30 16:51:28.020493 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:28.020463 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-480a60-7c447cb756-6ct9j" Oct 30 16:51:28.420267 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:28.420231 5052 patch_prober.go:29] interesting pod/e2e-inf-a-ff1189-openshift-default-7d9f9f5c65-jjdbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Oct 30 16:51:28.420475 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:28.420287 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ff1189-openshift-default-7d9f9f5c65-jjdbw" podUID="ba6262c9-b3ea-4c61-bb84-4f60954d02a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Oct 30 16:51:28.600386 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:28.600355 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-3ee682/test-model-3ee682-kserve-7c6cd66757-dm6lb" Oct 30 16:51:28.613039 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:28.613013 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-3ee682/test-model-3ee682-kserve-7c6cd66757-dm6lb" Oct 30 16:51:29.071283 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:29.071250 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-3ee682/test-model-3ee682-kserve-7c6cd66757-dm6lb" Oct 30 16:51:29.420118 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:29.420084 5052 patch_prober.go:29] interesting pod/e2e-inf-a-ff1189-openshift-default-7d9f9f5c65-jjdbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Oct 30 16:51:29.420307 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:29.420139 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ff1189-openshift-default-7d9f9f5c65-jjdbw" podUID="ba6262c9-b3ea-4c61-bb84-4f60954d02a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Oct 30 16:51:30.420153 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:30.420119 5052 patch_prober.go:29] interesting pod/e2e-inf-a-ff1189-openshift-default-7d9f9f5c65-jjdbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Oct 30 16:51:30.420622 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:30.420174 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ff1189-openshift-default-7d9f9f5c65-jjdbw" podUID="ba6262c9-b3ea-4c61-bb84-4f60954d02a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Oct 30 16:51:31.431154 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:31.421948 5052 patch_prober.go:29] interesting pod/e2e-inf-a-ff1189-openshift-default-7d9f9f5c65-jjdbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Oct 30 16:51:31.431154 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:31.422013 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ff1189-openshift-default-7d9f9f5c65-jjdbw" podUID="ba6262c9-b3ea-4c61-bb84-4f60954d02a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Oct 30 16:51:32.424880 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:32.424852 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-ff1189-openshift-default-7d9f9f5c65-jjdbw" Oct 30 16:51:32.425357 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:32.425332 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-ff1189-openshift-default-7d9f9f5c65-jjdbw" Oct 30 16:51:32.426079 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:32.426051 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-ff1189-openshift-default-7d9f9f5c65-jjdbw" Oct 30 16:51:32.670972 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:32.670936 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e5674c-5969bf95bc-jrsfj"] Oct 30 16:51:32.671441 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:32.671151 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e5674c-5969bf95bc-jrsfj" podUID="8dbf7c5d-6618-43f3-8a40-94e3a7d14662" containerName="maas-api" containerID="cri-o://75373b1db1fd33a998a1ffb62471e48229104d78c1378ea07fc83a518f4b9447" gracePeriod=30 Oct 30 16:51:33.271536 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:33.270956 5052 generic.go:356] "Generic (PLEG): container finished" podID="8dbf7c5d-6618-43f3-8a40-94e3a7d14662" containerID="75373b1db1fd33a998a1ffb62471e48229104d78c1378ea07fc83a518f4b9447" exitCode=0 Oct 30 16:51:33.271712 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:33.271650 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e5674c-5969bf95bc-jrsfj" event={"ID":"8dbf7c5d-6618-43f3-8a40-94e3a7d14662","Type":"ContainerDied","Data":"75373b1db1fd33a998a1ffb62471e48229104d78c1378ea07fc83a518f4b9447"} Oct 30 16:51:33.271800 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:33.271721 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e5674c-5969bf95bc-jrsfj" event={"ID":"8dbf7c5d-6618-43f3-8a40-94e3a7d14662","Type":"ContainerDied","Data":"cc194b36e87717c632b78a60195cc42833dd9863f7f7d92466f715561061f4d2"} Oct 30 16:51:33.271800 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:33.271745 5052 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc194b36e87717c632b78a60195cc42833dd9863f7f7d92466f715561061f4d2" Oct 30 16:51:33.371047 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:33.370953 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e5674c-5969bf95bc-jrsfj" Oct 30 16:51:33.477466 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:33.477426 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8dbf7c5d-6618-43f3-8a40-94e3a7d14662-kube-api-access-c4wld\" (UniqueName: \"kubernetes.io/projected/8dbf7c5d-6618-43f3-8a40-94e3a7d14662-kube-api-access-c4wld\") pod \"8dbf7c5d-6618-43f3-8a40-94e3a7d14662\" (UID: \"8dbf7c5d-6618-43f3-8a40-94e3a7d14662\") " Oct 30 16:51:33.477664 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:33.477543 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8dbf7c5d-6618-43f3-8a40-94e3a7d14662-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8dbf7c5d-6618-43f3-8a40-94e3a7d14662-openshift-service-ca\") pod \"8dbf7c5d-6618-43f3-8a40-94e3a7d14662\" (UID: \"8dbf7c5d-6618-43f3-8a40-94e3a7d14662\") " Oct 30 16:51:33.477749 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:33.477689 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8dbf7c5d-6618-43f3-8a40-94e3a7d14662-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8dbf7c5d-6618-43f3-8a40-94e3a7d14662-maas-api-tls\") pod \"8dbf7c5d-6618-43f3-8a40-94e3a7d14662\" (UID: \"8dbf7c5d-6618-43f3-8a40-94e3a7d14662\") " Oct 30 16:51:33.477945 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:33.477921 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8dbf7c5d-6618-43f3-8a40-94e3a7d14662-openshift-service-ca" pod "8dbf7c5d-6618-43f3-8a40-94e3a7d14662" (UID: "8dbf7c5d-6618-43f3-8a40-94e3a7d14662"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:51:33.478254 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:33.478225 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8dbf7c5d-6618-43f3-8a40-94e3a7d14662-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8dbf7c5d-6618-43f3-8a40-94e3a7d14662-metrics-certs\") pod \"8dbf7c5d-6618-43f3-8a40-94e3a7d14662\" (UID: \"8dbf7c5d-6618-43f3-8a40-94e3a7d14662\") " Oct 30 16:51:33.479022 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:33.478998 5052 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8dbf7c5d-6618-43f3-8a40-94e3a7d14662-openshift-service-ca\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:51:33.480407 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:33.480381 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8dbf7c5d-6618-43f3-8a40-94e3a7d14662-maas-api-tls" pod "8dbf7c5d-6618-43f3-8a40-94e3a7d14662" (UID: "8dbf7c5d-6618-43f3-8a40-94e3a7d14662"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:51:33.480508 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:33.480469 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8dbf7c5d-6618-43f3-8a40-94e3a7d14662-metrics-certs" pod "8dbf7c5d-6618-43f3-8a40-94e3a7d14662" (UID: "8dbf7c5d-6618-43f3-8a40-94e3a7d14662"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:51:33.486222 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:33.486202 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dbf7c5d-6618-43f3-8a40-94e3a7d14662-kube-api-access-c4wld" pod "8dbf7c5d-6618-43f3-8a40-94e3a7d14662" (UID: "8dbf7c5d-6618-43f3-8a40-94e3a7d14662"). InnerVolumeSpecName "kube-api-access-c4wld". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:51:33.581059 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:33.581024 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4wld\" (UniqueName: \"kubernetes.io/projected/8dbf7c5d-6618-43f3-8a40-94e3a7d14662-kube-api-access-c4wld\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:51:33.581059 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:33.581053 5052 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8dbf7c5d-6618-43f3-8a40-94e3a7d14662-maas-api-tls\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:51:33.581059 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:33.581063 5052 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8dbf7c5d-6618-43f3-8a40-94e3a7d14662-metrics-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:51:34.278375 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:34.278343 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e5674c-5969bf95bc-jrsfj" Oct 30 16:51:34.339630 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:34.339599 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e5674c-5969bf95bc-jrsfj"] Oct 30 16:51:34.353475 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:34.353444 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e5674c-5969bf95bc-jrsfj"] Oct 30 16:51:34.602707 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:34.602627 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8dbf7c5d-6618-43f3-8a40-94e3a7d14662" path="/var/lib/kubelet/pods/8dbf7c5d-6618-43f3-8a40-94e3a7d14662/volumes" Oct 30 16:51:39.054845 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:39.054809 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-480a60-7c447cb756-6ct9j" Oct 30 16:51:43.446897 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:43.446861 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-a59502-openshift-default-74889b7c5f-9m8g5"] Oct 30 16:51:43.447936 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:43.447908 5052 cpu_manager.go:414] "containerMap: removing container" podUID="8dbf7c5d-6618-43f3-8a40-94e3a7d14662" containerName="maas-api" Oct 30 16:51:43.447936 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:43.447934 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8dbf7c5d-6618-43f3-8a40-94e3a7d14662" containerName="maas-api" Oct 30 16:51:43.448175 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:43.448158 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="8dbf7c5d-6618-43f3-8a40-94e3a7d14662" containerName="maas-api" Oct 30 16:51:43.454207 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:43.454186 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a59502-openshift-default-74889b7c5f-9m8g5" Oct 30 16:51:43.479315 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:43.479286 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a59502-openshift-default-74889b7c5f-9m8g5"] Oct 30 16:51:43.485881 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:43.485860 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-a59502-openshift-default-dockercfg-st59z\"" Oct 30 16:51:43.522323 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:43.522296 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7f77c25-893f-40a2-a406-1891790e4cb5-istio-token\") pod \"e2e-inf-a-a59502-openshift-default-74889b7c5f-9m8g5\" (UID: \"d7f77c25-893f-40a2-a406-1891790e4cb5\") " pod="openshift-ingress/e2e-inf-a-a59502-openshift-default-74889b7c5f-9m8g5" Oct 30 16:51:43.522468 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:43.522335 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7f77c25-893f-40a2-a406-1891790e4cb5-workload-socket\") pod \"e2e-inf-a-a59502-openshift-default-74889b7c5f-9m8g5\" (UID: \"d7f77c25-893f-40a2-a406-1891790e4cb5\") " pod="openshift-ingress/e2e-inf-a-a59502-openshift-default-74889b7c5f-9m8g5" Oct 30 16:51:43.522468 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:43.522356 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7f77c25-893f-40a2-a406-1891790e4cb5-istio-podinfo\") pod \"e2e-inf-a-a59502-openshift-default-74889b7c5f-9m8g5\" (UID: \"d7f77c25-893f-40a2-a406-1891790e4cb5\") " pod="openshift-ingress/e2e-inf-a-a59502-openshift-default-74889b7c5f-9m8g5" Oct 30 16:51:43.522468 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:43.522437 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7f77c25-893f-40a2-a406-1891790e4cb5-credential-socket\") pod \"e2e-inf-a-a59502-openshift-default-74889b7c5f-9m8g5\" (UID: \"d7f77c25-893f-40a2-a406-1891790e4cb5\") " pod="openshift-ingress/e2e-inf-a-a59502-openshift-default-74889b7c5f-9m8g5" Oct 30 16:51:43.522660 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:43.522476 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7f77c25-893f-40a2-a406-1891790e4cb5-istiod-ca-cert\") pod \"e2e-inf-a-a59502-openshift-default-74889b7c5f-9m8g5\" (UID: \"d7f77c25-893f-40a2-a406-1891790e4cb5\") " pod="openshift-ingress/e2e-inf-a-a59502-openshift-default-74889b7c5f-9m8g5" Oct 30 16:51:43.522783 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:43.522758 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7f77c25-893f-40a2-a406-1891790e4cb5-workload-certs\") pod \"e2e-inf-a-a59502-openshift-default-74889b7c5f-9m8g5\" (UID: \"d7f77c25-893f-40a2-a406-1891790e4cb5\") " pod="openshift-ingress/e2e-inf-a-a59502-openshift-default-74889b7c5f-9m8g5" Oct 30 16:51:43.522835 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:43.522820 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7f77c25-893f-40a2-a406-1891790e4cb5-istio-envoy\") pod \"e2e-inf-a-a59502-openshift-default-74889b7c5f-9m8g5\" (UID: \"d7f77c25-893f-40a2-a406-1891790e4cb5\") " pod="openshift-ingress/e2e-inf-a-a59502-openshift-default-74889b7c5f-9m8g5" Oct 30 16:51:43.522934 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:43.522917 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5cn6\" (UniqueName: \"kubernetes.io/projected/d7f77c25-893f-40a2-a406-1891790e4cb5-kube-api-access-d5cn6\") pod \"e2e-inf-a-a59502-openshift-default-74889b7c5f-9m8g5\" (UID: \"d7f77c25-893f-40a2-a406-1891790e4cb5\") " pod="openshift-ingress/e2e-inf-a-a59502-openshift-default-74889b7c5f-9m8g5" Oct 30 16:51:43.522997 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:43.522952 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7f77c25-893f-40a2-a406-1891790e4cb5-istio-data\") pod \"e2e-inf-a-a59502-openshift-default-74889b7c5f-9m8g5\" (UID: \"d7f77c25-893f-40a2-a406-1891790e4cb5\") " pod="openshift-ingress/e2e-inf-a-a59502-openshift-default-74889b7c5f-9m8g5" Oct 30 16:51:43.638698 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:43.632181 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7f77c25-893f-40a2-a406-1891790e4cb5-istio-envoy\") pod \"e2e-inf-a-a59502-openshift-default-74889b7c5f-9m8g5\" (UID: \"d7f77c25-893f-40a2-a406-1891790e4cb5\") " pod="openshift-ingress/e2e-inf-a-a59502-openshift-default-74889b7c5f-9m8g5" Oct 30 16:51:43.638698 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:43.632285 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5cn6\" (UniqueName: \"kubernetes.io/projected/d7f77c25-893f-40a2-a406-1891790e4cb5-kube-api-access-d5cn6\") pod \"e2e-inf-a-a59502-openshift-default-74889b7c5f-9m8g5\" (UID: \"d7f77c25-893f-40a2-a406-1891790e4cb5\") " pod="openshift-ingress/e2e-inf-a-a59502-openshift-default-74889b7c5f-9m8g5" Oct 30 16:51:43.638698 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:43.632319 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7f77c25-893f-40a2-a406-1891790e4cb5-istio-data\") pod \"e2e-inf-a-a59502-openshift-default-74889b7c5f-9m8g5\" (UID: \"d7f77c25-893f-40a2-a406-1891790e4cb5\") " pod="openshift-ingress/e2e-inf-a-a59502-openshift-default-74889b7c5f-9m8g5" Oct 30 16:51:43.638698 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:43.632455 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7f77c25-893f-40a2-a406-1891790e4cb5-istio-token\") pod \"e2e-inf-a-a59502-openshift-default-74889b7c5f-9m8g5\" (UID: \"d7f77c25-893f-40a2-a406-1891790e4cb5\") " pod="openshift-ingress/e2e-inf-a-a59502-openshift-default-74889b7c5f-9m8g5" Oct 30 16:51:43.638698 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:43.632505 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7f77c25-893f-40a2-a406-1891790e4cb5-workload-socket\") pod \"e2e-inf-a-a59502-openshift-default-74889b7c5f-9m8g5\" (UID: \"d7f77c25-893f-40a2-a406-1891790e4cb5\") " pod="openshift-ingress/e2e-inf-a-a59502-openshift-default-74889b7c5f-9m8g5" Oct 30 16:51:43.638698 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:43.632535 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7f77c25-893f-40a2-a406-1891790e4cb5-istio-podinfo\") pod \"e2e-inf-a-a59502-openshift-default-74889b7c5f-9m8g5\" (UID: \"d7f77c25-893f-40a2-a406-1891790e4cb5\") " pod="openshift-ingress/e2e-inf-a-a59502-openshift-default-74889b7c5f-9m8g5" Oct 30 16:51:43.638698 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:43.632610 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7f77c25-893f-40a2-a406-1891790e4cb5-credential-socket\") pod \"e2e-inf-a-a59502-openshift-default-74889b7c5f-9m8g5\" (UID: \"d7f77c25-893f-40a2-a406-1891790e4cb5\") " pod="openshift-ingress/e2e-inf-a-a59502-openshift-default-74889b7c5f-9m8g5" Oct 30 16:51:43.638698 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:43.632639 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7f77c25-893f-40a2-a406-1891790e4cb5-istiod-ca-cert\") pod \"e2e-inf-a-a59502-openshift-default-74889b7c5f-9m8g5\" (UID: \"d7f77c25-893f-40a2-a406-1891790e4cb5\") " pod="openshift-ingress/e2e-inf-a-a59502-openshift-default-74889b7c5f-9m8g5" Oct 30 16:51:43.638698 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:43.632881 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7f77c25-893f-40a2-a406-1891790e4cb5-workload-certs\") pod \"e2e-inf-a-a59502-openshift-default-74889b7c5f-9m8g5\" (UID: \"d7f77c25-893f-40a2-a406-1891790e4cb5\") " pod="openshift-ingress/e2e-inf-a-a59502-openshift-default-74889b7c5f-9m8g5" Oct 30 16:51:43.638698 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:43.633374 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7f77c25-893f-40a2-a406-1891790e4cb5-workload-certs\") pod \"e2e-inf-a-a59502-openshift-default-74889b7c5f-9m8g5\" (UID: \"d7f77c25-893f-40a2-a406-1891790e4cb5\") " pod="openshift-ingress/e2e-inf-a-a59502-openshift-default-74889b7c5f-9m8g5" Oct 30 16:51:43.708727 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:43.706128 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7f77c25-893f-40a2-a406-1891790e4cb5-istio-podinfo\") pod \"e2e-inf-a-a59502-openshift-default-74889b7c5f-9m8g5\" (UID: \"d7f77c25-893f-40a2-a406-1891790e4cb5\") " pod="openshift-ingress/e2e-inf-a-a59502-openshift-default-74889b7c5f-9m8g5" Oct 30 16:51:43.708727 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:43.708233 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7f77c25-893f-40a2-a406-1891790e4cb5-istio-data\") pod \"e2e-inf-a-a59502-openshift-default-74889b7c5f-9m8g5\" (UID: \"d7f77c25-893f-40a2-a406-1891790e4cb5\") " pod="openshift-ingress/e2e-inf-a-a59502-openshift-default-74889b7c5f-9m8g5" Oct 30 16:51:43.710673 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:43.709755 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7f77c25-893f-40a2-a406-1891790e4cb5-workload-socket\") pod \"e2e-inf-a-a59502-openshift-default-74889b7c5f-9m8g5\" (UID: \"d7f77c25-893f-40a2-a406-1891790e4cb5\") " pod="openshift-ingress/e2e-inf-a-a59502-openshift-default-74889b7c5f-9m8g5" Oct 30 16:51:43.710673 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:43.710160 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7f77c25-893f-40a2-a406-1891790e4cb5-credential-socket\") pod \"e2e-inf-a-a59502-openshift-default-74889b7c5f-9m8g5\" (UID: \"d7f77c25-893f-40a2-a406-1891790e4cb5\") " pod="openshift-ingress/e2e-inf-a-a59502-openshift-default-74889b7c5f-9m8g5" Oct 30 16:51:43.711706 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:43.711683 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7f77c25-893f-40a2-a406-1891790e4cb5-istiod-ca-cert\") pod \"e2e-inf-a-a59502-openshift-default-74889b7c5f-9m8g5\" (UID: \"d7f77c25-893f-40a2-a406-1891790e4cb5\") " pod="openshift-ingress/e2e-inf-a-a59502-openshift-default-74889b7c5f-9m8g5" Oct 30 16:51:43.754742 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:43.754562 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5cn6\" (UniqueName: \"kubernetes.io/projected/d7f77c25-893f-40a2-a406-1891790e4cb5-kube-api-access-d5cn6\") pod \"e2e-inf-a-a59502-openshift-default-74889b7c5f-9m8g5\" (UID: \"d7f77c25-893f-40a2-a406-1891790e4cb5\") " pod="openshift-ingress/e2e-inf-a-a59502-openshift-default-74889b7c5f-9m8g5" Oct 30 16:51:43.881771 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:43.881653 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7f77c25-893f-40a2-a406-1891790e4cb5-istio-token\") pod \"e2e-inf-a-a59502-openshift-default-74889b7c5f-9m8g5\" (UID: \"d7f77c25-893f-40a2-a406-1891790e4cb5\") " pod="openshift-ingress/e2e-inf-a-a59502-openshift-default-74889b7c5f-9m8g5" Oct 30 16:51:43.898645 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:43.898611 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7f77c25-893f-40a2-a406-1891790e4cb5-istio-envoy\") pod \"e2e-inf-a-a59502-openshift-default-74889b7c5f-9m8g5\" (UID: \"d7f77c25-893f-40a2-a406-1891790e4cb5\") " pod="openshift-ingress/e2e-inf-a-a59502-openshift-default-74889b7c5f-9m8g5" Oct 30 16:51:44.085843 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:44.085767 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a59502-openshift-default-74889b7c5f-9m8g5" Oct 30 16:51:44.486159 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:44.486123 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a59502-openshift-default-74889b7c5f-9m8g5"] Oct 30 16:51:44.486511 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:51:44.486169 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7f77c25_893f_40a2_a406_1891790e4cb5.slice/crio-1de9440a9b246f56fd568ad961d5ca4ea5093d14d22e94fa5cdc24352a499c08 WatchSource:0}: Error finding container 1de9440a9b246f56fd568ad961d5ca4ea5093d14d22e94fa5cdc24352a499c08: Status 404 returned error can't find the container with id 1de9440a9b246f56fd568ad961d5ca4ea5093d14d22e94fa5cdc24352a499c08 Oct 30 16:51:44.489025 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:44.488995 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:51:44.489113 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:44.489061 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:51:44.489113 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:44.489090 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:51:44.633605 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:44.632902 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ca5971-7d5f4777f7-cttvv"] Oct 30 16:51:44.633605 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:44.633151 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ca5971-7d5f4777f7-cttvv" podUID="607c64a6-dc52-4862-b415-e25194a55649" containerName="maas-api" containerID="cri-o://265af70373f1c98c6c9a5de1a78df5071f059d30703664774f987636224fa085" gracePeriod=30 Oct 30 16:51:45.391652 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:45.382041 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ca5971-7d5f4777f7-cttvv" Oct 30 16:51:45.499730 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:45.499690 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a59502-openshift-default-74889b7c5f-9m8g5" event={"ID":"d7f77c25-893f-40a2-a406-1891790e4cb5","Type":"ContainerStarted","Data":"bede44d865571edbe12a4667fdc6a7856a8ffd6bebc876fb78a0e14ca5361bbe"} Oct 30 16:51:45.500318 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:45.500291 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a59502-openshift-default-74889b7c5f-9m8g5" event={"ID":"d7f77c25-893f-40a2-a406-1891790e4cb5","Type":"ContainerStarted","Data":"1de9440a9b246f56fd568ad961d5ca4ea5093d14d22e94fa5cdc24352a499c08"} Oct 30 16:51:45.502950 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:45.502922 5052 generic.go:356] "Generic (PLEG): container finished" podID="607c64a6-dc52-4862-b415-e25194a55649" containerID="265af70373f1c98c6c9a5de1a78df5071f059d30703664774f987636224fa085" exitCode=0 Oct 30 16:51:45.503120 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:45.502974 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ca5971-7d5f4777f7-cttvv" Oct 30 16:51:45.503268 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:45.502965 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ca5971-7d5f4777f7-cttvv" event={"ID":"607c64a6-dc52-4862-b415-e25194a55649","Type":"ContainerDied","Data":"265af70373f1c98c6c9a5de1a78df5071f059d30703664774f987636224fa085"} Oct 30 16:51:45.503336 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:45.503304 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ca5971-7d5f4777f7-cttvv" event={"ID":"607c64a6-dc52-4862-b415-e25194a55649","Type":"ContainerDied","Data":"01c75b4dccacc4be8f8c99da8890784e2a40c782202747950abe583e3a9a3571"} Oct 30 16:51:45.503383 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:45.503364 5052 scope.go:122] "RemoveContainer" containerID="265af70373f1c98c6c9a5de1a78df5071f059d30703664774f987636224fa085" Oct 30 16:51:45.514625 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:45.514608 5052 scope.go:122] "RemoveContainer" containerID="265af70373f1c98c6c9a5de1a78df5071f059d30703664774f987636224fa085" Oct 30 16:51:45.514909 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:51:45.514886 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"265af70373f1c98c6c9a5de1a78df5071f059d30703664774f987636224fa085\": container with ID starting with 265af70373f1c98c6c9a5de1a78df5071f059d30703664774f987636224fa085 not found: ID does not exist" containerID="265af70373f1c98c6c9a5de1a78df5071f059d30703664774f987636224fa085" Oct 30 16:51:45.514970 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:45.514917 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"265af70373f1c98c6c9a5de1a78df5071f059d30703664774f987636224fa085"} err="failed to get container status \"265af70373f1c98c6c9a5de1a78df5071f059d30703664774f987636224fa085\": rpc error: code = NotFound desc = could not find container \"265af70373f1c98c6c9a5de1a78df5071f059d30703664774f987636224fa085\": container with ID starting with 265af70373f1c98c6c9a5de1a78df5071f059d30703664774f987636224fa085 not found: ID does not exist" Oct 30 16:51:45.567966 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:45.567913 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/607c64a6-dc52-4862-b415-e25194a55649-metrics-certs\" (UniqueName: \"kubernetes.io/secret/607c64a6-dc52-4862-b415-e25194a55649-metrics-certs\") pod \"607c64a6-dc52-4862-b415-e25194a55649\" (UID: \"607c64a6-dc52-4862-b415-e25194a55649\") " Oct 30 16:51:45.568813 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:45.568784 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/607c64a6-dc52-4862-b415-e25194a55649-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/607c64a6-dc52-4862-b415-e25194a55649-maas-api-tls\") pod \"607c64a6-dc52-4862-b415-e25194a55649\" (UID: \"607c64a6-dc52-4862-b415-e25194a55649\") " Oct 30 16:51:45.568972 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:45.568872 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/607c64a6-dc52-4862-b415-e25194a55649-kube-api-access-4sz5f\" (UniqueName: \"kubernetes.io/projected/607c64a6-dc52-4862-b415-e25194a55649-kube-api-access-4sz5f\") pod \"607c64a6-dc52-4862-b415-e25194a55649\" (UID: \"607c64a6-dc52-4862-b415-e25194a55649\") " Oct 30 16:51:45.569182 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:45.569159 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/607c64a6-dc52-4862-b415-e25194a55649-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/607c64a6-dc52-4862-b415-e25194a55649-openshift-service-ca\") pod \"607c64a6-dc52-4862-b415-e25194a55649\" (UID: \"607c64a6-dc52-4862-b415-e25194a55649\") " Oct 30 16:51:45.569660 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:45.569631 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/607c64a6-dc52-4862-b415-e25194a55649-openshift-service-ca" pod "607c64a6-dc52-4862-b415-e25194a55649" (UID: "607c64a6-dc52-4862-b415-e25194a55649"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:51:45.570079 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:45.570044 5052 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/607c64a6-dc52-4862-b415-e25194a55649-openshift-service-ca\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:51:45.571277 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:45.571220 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-a59502-openshift-default-74889b7c5f-9m8g5" podStartSLOduration=2.5712029899999997 podStartE2EDuration="2.57120299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:51:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:51:45.565692686 +0000 UTC m=+1861.093027927" watchObservedRunningTime="2026-10-30 16:51:45.57120299 +0000 UTC m=+1861.098538210" Oct 30 16:51:45.571411 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:45.571360 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/607c64a6-dc52-4862-b415-e25194a55649-metrics-certs" pod "607c64a6-dc52-4862-b415-e25194a55649" (UID: "607c64a6-dc52-4862-b415-e25194a55649"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:51:45.573003 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:45.572979 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/607c64a6-dc52-4862-b415-e25194a55649-kube-api-access-4sz5f" pod "607c64a6-dc52-4862-b415-e25194a55649" (UID: "607c64a6-dc52-4862-b415-e25194a55649"). InnerVolumeSpecName "kube-api-access-4sz5f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:51:45.573375 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:45.573358 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/607c64a6-dc52-4862-b415-e25194a55649-maas-api-tls" pod "607c64a6-dc52-4862-b415-e25194a55649" (UID: "607c64a6-dc52-4862-b415-e25194a55649"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:51:45.693611 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:45.692631 5052 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/607c64a6-dc52-4862-b415-e25194a55649-metrics-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:51:45.693611 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:45.692669 5052 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/607c64a6-dc52-4862-b415-e25194a55649-maas-api-tls\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:51:45.693611 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:45.692684 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4sz5f\" (UniqueName: \"kubernetes.io/projected/607c64a6-dc52-4862-b415-e25194a55649-kube-api-access-4sz5f\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:51:45.947287 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:45.947198 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ca5971-7d5f4777f7-cttvv"] Oct 30 16:51:45.960225 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:45.960183 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ca5971-7d5f4777f7-cttvv"] Oct 30 16:51:46.086739 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:46.086705 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-a59502-openshift-default-74889b7c5f-9m8g5" Oct 30 16:51:46.092771 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:46.092746 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-a59502-openshift-default-74889b7c5f-9m8g5" Oct 30 16:51:46.434919 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:46.434885 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-5552t"] Oct 30 16:51:46.435140 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:46.435115 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-5552t" podUID="0a85b25b-991b-4f0d-9ef4-baab2b59d7b9" containerName="istio-proxy" containerID="cri-o://39aa2c7cc642c2b8d067e46e1f02243dab4630349de336f264af6342de2de15a" gracePeriod=30 Oct 30 16:51:46.443106 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:46.443079 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-st7mw"] Oct 30 16:51:46.443340 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:46.443300 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-st7mw" podUID="dd769390-6903-4ed3-9ee9-814fb84762e9" containerName="istio-proxy" containerID="cri-o://f7eb5729a6832f55a0cd1d2dd4d9b05cb8a6f5a86b32ae79e079b85a74f79144" gracePeriod=30 Oct 30 16:51:46.447880 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:46.447855 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-zhqln"] Oct 30 16:51:46.448037 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:46.448018 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-zhqln" podUID="9d9e35b9-801d-4b2d-aba2-c34409d041ad" containerName="istio-proxy" containerID="cri-o://546d7652e7eba58d0c7db12381af0885040996537309f07c9e71cd75e0b85a5e" gracePeriod=30 Oct 30 16:51:46.512102 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:46.512075 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a59502-openshift-default-74889b7c5f-9m8g5" Oct 30 16:51:46.513114 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:46.513094 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-a59502-openshift-default-74889b7c5f-9m8g5" Oct 30 16:51:46.882836 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:46.882737 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="607c64a6-dc52-4862-b415-e25194a55649" path="/var/lib/kubelet/pods/607c64a6-dc52-4862-b415-e25194a55649/volumes" Oct 30 16:51:47.854111 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:47.854055 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ca5971-7d5f4777f7-cttvv" podUID="607c64a6-dc52-4862-b415-e25194a55649" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.133:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:51:48.851813 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:51:48.851776 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 16:51:51.632256 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:51.632224 5052 generic.go:356] "Generic (PLEG): container finished" podID="dd769390-6903-4ed3-9ee9-814fb84762e9" containerID="f7eb5729a6832f55a0cd1d2dd4d9b05cb8a6f5a86b32ae79e079b85a74f79144" exitCode=0 Oct 30 16:51:51.632670 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:51.632368 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-st7mw" event={"ID":"dd769390-6903-4ed3-9ee9-814fb84762e9","Type":"ContainerDied","Data":"f7eb5729a6832f55a0cd1d2dd4d9b05cb8a6f5a86b32ae79e079b85a74f79144"} Oct 30 16:51:51.870309 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:51.868525 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-st7mw" Oct 30 16:51:51.900604 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:51.899828 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd769390-6903-4ed3-9ee9-814fb84762e9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd769390-6903-4ed3-9ee9-814fb84762e9-istiod-ca-cert\") pod \"dd769390-6903-4ed3-9ee9-814fb84762e9\" (UID: \"dd769390-6903-4ed3-9ee9-814fb84762e9\") " Oct 30 16:51:51.900604 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:51.899918 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd769390-6903-4ed3-9ee9-814fb84762e9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd769390-6903-4ed3-9ee9-814fb84762e9-workload-certs\") pod \"dd769390-6903-4ed3-9ee9-814fb84762e9\" (UID: \"dd769390-6903-4ed3-9ee9-814fb84762e9\") " Oct 30 16:51:51.900604 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:51.900088 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd769390-6903-4ed3-9ee9-814fb84762e9-istio-token\" (UniqueName: \"kubernetes.io/projected/dd769390-6903-4ed3-9ee9-814fb84762e9-istio-token\") pod \"dd769390-6903-4ed3-9ee9-814fb84762e9\" (UID: \"dd769390-6903-4ed3-9ee9-814fb84762e9\") " Oct 30 16:51:51.900604 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:51.900459 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd769390-6903-4ed3-9ee9-814fb84762e9-istiod-ca-cert" pod "dd769390-6903-4ed3-9ee9-814fb84762e9" (UID: "dd769390-6903-4ed3-9ee9-814fb84762e9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:51:51.900604 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:51.900511 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd769390-6903-4ed3-9ee9-814fb84762e9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd769390-6903-4ed3-9ee9-814fb84762e9-credential-socket\") pod \"dd769390-6903-4ed3-9ee9-814fb84762e9\" (UID: \"dd769390-6903-4ed3-9ee9-814fb84762e9\") " Oct 30 16:51:51.900604 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:51.900576 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dd769390-6903-4ed3-9ee9-814fb84762e9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd769390-6903-4ed3-9ee9-814fb84762e9-istio-podinfo\") pod \"dd769390-6903-4ed3-9ee9-814fb84762e9\" (UID: \"dd769390-6903-4ed3-9ee9-814fb84762e9\") " Oct 30 16:51:51.901069 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:51.900666 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd769390-6903-4ed3-9ee9-814fb84762e9-kube-api-access-tb5md\" (UniqueName: \"kubernetes.io/projected/dd769390-6903-4ed3-9ee9-814fb84762e9-kube-api-access-tb5md\") pod \"dd769390-6903-4ed3-9ee9-814fb84762e9\" (UID: \"dd769390-6903-4ed3-9ee9-814fb84762e9\") " Oct 30 16:51:51.901069 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:51.900751 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd769390-6903-4ed3-9ee9-814fb84762e9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd769390-6903-4ed3-9ee9-814fb84762e9-workload-socket\") pod \"dd769390-6903-4ed3-9ee9-814fb84762e9\" (UID: \"dd769390-6903-4ed3-9ee9-814fb84762e9\") " Oct 30 16:51:51.901069 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:51.900791 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd769390-6903-4ed3-9ee9-814fb84762e9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd769390-6903-4ed3-9ee9-814fb84762e9-istio-envoy\") pod \"dd769390-6903-4ed3-9ee9-814fb84762e9\" (UID: \"dd769390-6903-4ed3-9ee9-814fb84762e9\") " Oct 30 16:51:51.901069 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:51.900937 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd769390-6903-4ed3-9ee9-814fb84762e9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd769390-6903-4ed3-9ee9-814fb84762e9-istio-data\") pod \"dd769390-6903-4ed3-9ee9-814fb84762e9\" (UID: \"dd769390-6903-4ed3-9ee9-814fb84762e9\") " Oct 30 16:51:51.903645 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:51.901556 5052 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd769390-6903-4ed3-9ee9-814fb84762e9-istiod-ca-cert\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:51:51.903645 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:51.901833 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd769390-6903-4ed3-9ee9-814fb84762e9-credential-socket" pod "dd769390-6903-4ed3-9ee9-814fb84762e9" (UID: "dd769390-6903-4ed3-9ee9-814fb84762e9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:51:51.903645 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:51.901878 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd769390-6903-4ed3-9ee9-814fb84762e9-istio-data" pod "dd769390-6903-4ed3-9ee9-814fb84762e9" (UID: "dd769390-6903-4ed3-9ee9-814fb84762e9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:51:51.903645 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:51.902076 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd769390-6903-4ed3-9ee9-814fb84762e9-workload-socket" pod "dd769390-6903-4ed3-9ee9-814fb84762e9" (UID: "dd769390-6903-4ed3-9ee9-814fb84762e9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:51:51.903645 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:51.902690 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd769390-6903-4ed3-9ee9-814fb84762e9-workload-certs" pod "dd769390-6903-4ed3-9ee9-814fb84762e9" (UID: "dd769390-6903-4ed3-9ee9-814fb84762e9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:51:51.903645 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:51.902835 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd769390-6903-4ed3-9ee9-814fb84762e9-istio-token" pod "dd769390-6903-4ed3-9ee9-814fb84762e9" (UID: "dd769390-6903-4ed3-9ee9-814fb84762e9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:51:51.907002 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:51.906973 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd769390-6903-4ed3-9ee9-814fb84762e9-istio-envoy" pod "dd769390-6903-4ed3-9ee9-814fb84762e9" (UID: "dd769390-6903-4ed3-9ee9-814fb84762e9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:51:51.907156 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:51.907130 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd769390-6903-4ed3-9ee9-814fb84762e9-kube-api-access-tb5md" pod "dd769390-6903-4ed3-9ee9-814fb84762e9" (UID: "dd769390-6903-4ed3-9ee9-814fb84762e9"). InnerVolumeSpecName "kube-api-access-tb5md". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:51:51.907375 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:51.907351 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dd769390-6903-4ed3-9ee9-814fb84762e9-istio-podinfo" pod "dd769390-6903-4ed3-9ee9-814fb84762e9" (UID: "dd769390-6903-4ed3-9ee9-814fb84762e9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:51:51.974594 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:51.974004 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-zhqln" Oct 30 16:51:51.978369 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:51.978342 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-5552t" Oct 30 16:51:52.002323 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:52.002286 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9d9e35b9-801d-4b2d-aba2-c34409d041ad-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d9e35b9-801d-4b2d-aba2-c34409d041ad-istio-podinfo\") pod \"9d9e35b9-801d-4b2d-aba2-c34409d041ad\" (UID: \"9d9e35b9-801d-4b2d-aba2-c34409d041ad\") " Oct 30 16:51:52.002487 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:52.002362 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a85b25b-991b-4f0d-9ef4-baab2b59d7b9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a85b25b-991b-4f0d-9ef4-baab2b59d7b9-istio-data\") pod \"0a85b25b-991b-4f0d-9ef4-baab2b59d7b9\" (UID: \"0a85b25b-991b-4f0d-9ef4-baab2b59d7b9\") " Oct 30 16:51:52.002487 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:52.002394 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a85b25b-991b-4f0d-9ef4-baab2b59d7b9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a85b25b-991b-4f0d-9ef4-baab2b59d7b9-istio-envoy\") pod \"0a85b25b-991b-4f0d-9ef4-baab2b59d7b9\" (UID: \"0a85b25b-991b-4f0d-9ef4-baab2b59d7b9\") " Oct 30 16:51:52.002487 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:52.002457 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a85b25b-991b-4f0d-9ef4-baab2b59d7b9-kube-api-access-lcthm\" (UniqueName: \"kubernetes.io/projected/0a85b25b-991b-4f0d-9ef4-baab2b59d7b9-kube-api-access-lcthm\") pod \"0a85b25b-991b-4f0d-9ef4-baab2b59d7b9\" (UID: \"0a85b25b-991b-4f0d-9ef4-baab2b59d7b9\") " Oct 30 16:51:52.002673 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:52.002495 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0a85b25b-991b-4f0d-9ef4-baab2b59d7b9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a85b25b-991b-4f0d-9ef4-baab2b59d7b9-istio-podinfo\") pod \"0a85b25b-991b-4f0d-9ef4-baab2b59d7b9\" (UID: \"0a85b25b-991b-4f0d-9ef4-baab2b59d7b9\") " Oct 30 16:51:52.002673 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:52.002536 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a85b25b-991b-4f0d-9ef4-baab2b59d7b9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a85b25b-991b-4f0d-9ef4-baab2b59d7b9-workload-certs\") pod \"0a85b25b-991b-4f0d-9ef4-baab2b59d7b9\" (UID: \"0a85b25b-991b-4f0d-9ef4-baab2b59d7b9\") " Oct 30 16:51:52.002673 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:52.002628 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d9e35b9-801d-4b2d-aba2-c34409d041ad-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d9e35b9-801d-4b2d-aba2-c34409d041ad-istio-envoy\") pod \"9d9e35b9-801d-4b2d-aba2-c34409d041ad\" (UID: \"9d9e35b9-801d-4b2d-aba2-c34409d041ad\") " Oct 30 16:51:52.002844 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:52.002687 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a85b25b-991b-4f0d-9ef4-baab2b59d7b9-istio-data" pod "0a85b25b-991b-4f0d-9ef4-baab2b59d7b9" (UID: "0a85b25b-991b-4f0d-9ef4-baab2b59d7b9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:51:52.002844 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:52.002776 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d9e35b9-801d-4b2d-aba2-c34409d041ad-kube-api-access-t8n4m\" (UniqueName: \"kubernetes.io/projected/9d9e35b9-801d-4b2d-aba2-c34409d041ad-kube-api-access-t8n4m\") pod \"9d9e35b9-801d-4b2d-aba2-c34409d041ad\" (UID: \"9d9e35b9-801d-4b2d-aba2-c34409d041ad\") " Oct 30 16:51:52.002962 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:52.002877 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a85b25b-991b-4f0d-9ef4-baab2b59d7b9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a85b25b-991b-4f0d-9ef4-baab2b59d7b9-credential-socket\") pod \"0a85b25b-991b-4f0d-9ef4-baab2b59d7b9\" (UID: \"0a85b25b-991b-4f0d-9ef4-baab2b59d7b9\") " Oct 30 16:51:52.002962 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:52.002915 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d9e35b9-801d-4b2d-aba2-c34409d041ad-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d9e35b9-801d-4b2d-aba2-c34409d041ad-workload-socket\") pod \"9d9e35b9-801d-4b2d-aba2-c34409d041ad\" (UID: \"9d9e35b9-801d-4b2d-aba2-c34409d041ad\") " Oct 30 16:51:52.003074 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:52.002962 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d9e35b9-801d-4b2d-aba2-c34409d041ad-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d9e35b9-801d-4b2d-aba2-c34409d041ad-workload-certs\") pod \"9d9e35b9-801d-4b2d-aba2-c34409d041ad\" (UID: \"9d9e35b9-801d-4b2d-aba2-c34409d041ad\") " Oct 30 16:51:52.003074 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:52.003006 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a85b25b-991b-4f0d-9ef4-baab2b59d7b9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a85b25b-991b-4f0d-9ef4-baab2b59d7b9-istiod-ca-cert\") pod \"0a85b25b-991b-4f0d-9ef4-baab2b59d7b9\" (UID: \"0a85b25b-991b-4f0d-9ef4-baab2b59d7b9\") " Oct 30 16:51:52.003178 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:52.003089 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d9e35b9-801d-4b2d-aba2-c34409d041ad-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d9e35b9-801d-4b2d-aba2-c34409d041ad-istio-data\") pod \"9d9e35b9-801d-4b2d-aba2-c34409d041ad\" (UID: \"9d9e35b9-801d-4b2d-aba2-c34409d041ad\") " Oct 30 16:51:52.003178 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:52.003134 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d9e35b9-801d-4b2d-aba2-c34409d041ad-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d9e35b9-801d-4b2d-aba2-c34409d041ad-istiod-ca-cert\") pod \"9d9e35b9-801d-4b2d-aba2-c34409d041ad\" (UID: \"9d9e35b9-801d-4b2d-aba2-c34409d041ad\") " Oct 30 16:51:52.003292 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:52.003175 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a85b25b-991b-4f0d-9ef4-baab2b59d7b9-istio-token\" (UniqueName: \"kubernetes.io/projected/0a85b25b-991b-4f0d-9ef4-baab2b59d7b9-istio-token\") pod \"0a85b25b-991b-4f0d-9ef4-baab2b59d7b9\" (UID: \"0a85b25b-991b-4f0d-9ef4-baab2b59d7b9\") " Oct 30 16:51:52.003292 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:52.003277 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d9e35b9-801d-4b2d-aba2-c34409d041ad-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d9e35b9-801d-4b2d-aba2-c34409d041ad-credential-socket\") pod \"9d9e35b9-801d-4b2d-aba2-c34409d041ad\" (UID: \"9d9e35b9-801d-4b2d-aba2-c34409d041ad\") " Oct 30 16:51:52.003402 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:52.003309 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a85b25b-991b-4f0d-9ef4-baab2b59d7b9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a85b25b-991b-4f0d-9ef4-baab2b59d7b9-workload-socket\") pod \"0a85b25b-991b-4f0d-9ef4-baab2b59d7b9\" (UID: \"0a85b25b-991b-4f0d-9ef4-baab2b59d7b9\") " Oct 30 16:51:52.003402 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:52.003350 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a85b25b-991b-4f0d-9ef4-baab2b59d7b9-workload-certs" pod "0a85b25b-991b-4f0d-9ef4-baab2b59d7b9" (UID: "0a85b25b-991b-4f0d-9ef4-baab2b59d7b9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:51:52.003402 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:52.003394 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d9e35b9-801d-4b2d-aba2-c34409d041ad-istio-token\" (UniqueName: \"kubernetes.io/projected/9d9e35b9-801d-4b2d-aba2-c34409d041ad-istio-token\") pod \"9d9e35b9-801d-4b2d-aba2-c34409d041ad\" (UID: \"9d9e35b9-801d-4b2d-aba2-c34409d041ad\") " Oct 30 16:51:52.004439 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:52.003731 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a85b25b-991b-4f0d-9ef4-baab2b59d7b9-istiod-ca-cert" pod "0a85b25b-991b-4f0d-9ef4-baab2b59d7b9" (UID: "0a85b25b-991b-4f0d-9ef4-baab2b59d7b9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:51:52.004439 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:52.004162 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tb5md\" (UniqueName: \"kubernetes.io/projected/dd769390-6903-4ed3-9ee9-814fb84762e9-kube-api-access-tb5md\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:51:52.004439 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:52.004180 5052 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd769390-6903-4ed3-9ee9-814fb84762e9-workload-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:51:52.004439 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:52.004191 5052 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd769390-6903-4ed3-9ee9-814fb84762e9-istio-envoy\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:51:52.004439 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:52.004200 5052 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a85b25b-991b-4f0d-9ef4-baab2b59d7b9-istiod-ca-cert\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:51:52.004439 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:52.004208 5052 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd769390-6903-4ed3-9ee9-814fb84762e9-istio-data\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:51:52.004439 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:52.004215 5052 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd769390-6903-4ed3-9ee9-814fb84762e9-workload-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:51:52.004439 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:52.004222 5052 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a85b25b-991b-4f0d-9ef4-baab2b59d7b9-istio-data\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:51:52.004439 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:52.004230 5052 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a85b25b-991b-4f0d-9ef4-baab2b59d7b9-workload-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:51:52.004439 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:52.004238 5052 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd769390-6903-4ed3-9ee9-814fb84762e9-istio-token\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:51:52.004439 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:52.004245 5052 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd769390-6903-4ed3-9ee9-814fb84762e9-credential-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:51:52.004439 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:52.004253 5052 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd769390-6903-4ed3-9ee9-814fb84762e9-istio-podinfo\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:51:52.005206 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:52.004913 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a85b25b-991b-4f0d-9ef4-baab2b59d7b9-credential-socket" pod "0a85b25b-991b-4f0d-9ef4-baab2b59d7b9" (UID: "0a85b25b-991b-4f0d-9ef4-baab2b59d7b9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:51:52.005206 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:52.005118 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d9e35b9-801d-4b2d-aba2-c34409d041ad-workload-socket" pod "9d9e35b9-801d-4b2d-aba2-c34409d041ad" (UID: "9d9e35b9-801d-4b2d-aba2-c34409d041ad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:51:52.005370 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:52.005341 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d9e35b9-801d-4b2d-aba2-c34409d041ad-workload-certs" pod "9d9e35b9-801d-4b2d-aba2-c34409d041ad" (UID: "9d9e35b9-801d-4b2d-aba2-c34409d041ad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:51:52.005546 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:52.005526 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d9e35b9-801d-4b2d-aba2-c34409d041ad-istio-data" pod "9d9e35b9-801d-4b2d-aba2-c34409d041ad" (UID: "9d9e35b9-801d-4b2d-aba2-c34409d041ad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:51:52.005679 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:52.005556 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d9e35b9-801d-4b2d-aba2-c34409d041ad-credential-socket" pod "9d9e35b9-801d-4b2d-aba2-c34409d041ad" (UID: "9d9e35b9-801d-4b2d-aba2-c34409d041ad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:51:52.005679 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:52.005621 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9d9e35b9-801d-4b2d-aba2-c34409d041ad-istio-podinfo" pod "9d9e35b9-801d-4b2d-aba2-c34409d041ad" (UID: "9d9e35b9-801d-4b2d-aba2-c34409d041ad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:51:52.005679 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:52.005652 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a85b25b-991b-4f0d-9ef4-baab2b59d7b9-istio-envoy" pod "0a85b25b-991b-4f0d-9ef4-baab2b59d7b9" (UID: "0a85b25b-991b-4f0d-9ef4-baab2b59d7b9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:51:52.005867 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:52.005835 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a85b25b-991b-4f0d-9ef4-baab2b59d7b9-workload-socket" pod "0a85b25b-991b-4f0d-9ef4-baab2b59d7b9" (UID: "0a85b25b-991b-4f0d-9ef4-baab2b59d7b9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:51:52.005988 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:52.005969 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d9e35b9-801d-4b2d-aba2-c34409d041ad-istiod-ca-cert" pod "9d9e35b9-801d-4b2d-aba2-c34409d041ad" (UID: "9d9e35b9-801d-4b2d-aba2-c34409d041ad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:51:52.006322 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:52.006295 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d9e35b9-801d-4b2d-aba2-c34409d041ad-istio-envoy" pod "9d9e35b9-801d-4b2d-aba2-c34409d041ad" (UID: "9d9e35b9-801d-4b2d-aba2-c34409d041ad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:51:52.006417 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:52.006372 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0a85b25b-991b-4f0d-9ef4-baab2b59d7b9-istio-podinfo" pod "0a85b25b-991b-4f0d-9ef4-baab2b59d7b9" (UID: "0a85b25b-991b-4f0d-9ef4-baab2b59d7b9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:51:52.006678 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:52.006659 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a85b25b-991b-4f0d-9ef4-baab2b59d7b9-kube-api-access-lcthm" pod "0a85b25b-991b-4f0d-9ef4-baab2b59d7b9" (UID: "0a85b25b-991b-4f0d-9ef4-baab2b59d7b9"). InnerVolumeSpecName "kube-api-access-lcthm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:51:52.006938 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:52.006917 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d9e35b9-801d-4b2d-aba2-c34409d041ad-istio-token" pod "9d9e35b9-801d-4b2d-aba2-c34409d041ad" (UID: "9d9e35b9-801d-4b2d-aba2-c34409d041ad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:51:52.007344 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:52.007317 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d9e35b9-801d-4b2d-aba2-c34409d041ad-kube-api-access-t8n4m" pod "9d9e35b9-801d-4b2d-aba2-c34409d041ad" (UID: "9d9e35b9-801d-4b2d-aba2-c34409d041ad"). InnerVolumeSpecName "kube-api-access-t8n4m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:51:52.007415 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:52.007357 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a85b25b-991b-4f0d-9ef4-baab2b59d7b9-istio-token" pod "0a85b25b-991b-4f0d-9ef4-baab2b59d7b9" (UID: "0a85b25b-991b-4f0d-9ef4-baab2b59d7b9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:51:52.134675 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:52.118252 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8n4m\" (UniqueName: \"kubernetes.io/projected/9d9e35b9-801d-4b2d-aba2-c34409d041ad-kube-api-access-t8n4m\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:51:52.134675 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:52.118302 5052 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a85b25b-991b-4f0d-9ef4-baab2b59d7b9-credential-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:51:52.134675 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:52.118317 5052 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d9e35b9-801d-4b2d-aba2-c34409d041ad-workload-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:51:52.134675 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:52.118331 5052 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d9e35b9-801d-4b2d-aba2-c34409d041ad-workload-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:51:52.134675 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:52.118345 5052 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d9e35b9-801d-4b2d-aba2-c34409d041ad-istio-data\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:51:52.134675 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:52.118357 5052 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d9e35b9-801d-4b2d-aba2-c34409d041ad-istiod-ca-cert\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:51:52.134675 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:52.118371 5052 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a85b25b-991b-4f0d-9ef4-baab2b59d7b9-istio-token\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:51:52.134675 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:52.118384 5052 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d9e35b9-801d-4b2d-aba2-c34409d041ad-credential-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:51:52.134675 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:52.118397 5052 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a85b25b-991b-4f0d-9ef4-baab2b59d7b9-workload-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:51:52.134675 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:52.118410 5052 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d9e35b9-801d-4b2d-aba2-c34409d041ad-istio-token\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:51:52.134675 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:52.118423 5052 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d9e35b9-801d-4b2d-aba2-c34409d041ad-istio-podinfo\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:51:52.134675 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:52.118435 5052 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a85b25b-991b-4f0d-9ef4-baab2b59d7b9-istio-envoy\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:51:52.134675 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:52.118449 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcthm\" (UniqueName: \"kubernetes.io/projected/0a85b25b-991b-4f0d-9ef4-baab2b59d7b9-kube-api-access-lcthm\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:51:52.134675 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:52.118461 5052 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a85b25b-991b-4f0d-9ef4-baab2b59d7b9-istio-podinfo\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:51:52.134675 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:52.118484 5052 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d9e35b9-801d-4b2d-aba2-c34409d041ad-istio-envoy\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:51:52.642274 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:52.642246 5052 generic.go:356] "Generic (PLEG): container finished" podID="9d9e35b9-801d-4b2d-aba2-c34409d041ad" containerID="546d7652e7eba58d0c7db12381af0885040996537309f07c9e71cd75e0b85a5e" exitCode=0 Oct 30 16:51:52.642702 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:52.642338 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-zhqln" Oct 30 16:51:52.642702 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:52.642338 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-zhqln" event={"ID":"9d9e35b9-801d-4b2d-aba2-c34409d041ad","Type":"ContainerDied","Data":"546d7652e7eba58d0c7db12381af0885040996537309f07c9e71cd75e0b85a5e"} Oct 30 16:51:52.642702 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:52.642480 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-zhqln" event={"ID":"9d9e35b9-801d-4b2d-aba2-c34409d041ad","Type":"ContainerDied","Data":"d116f570c3f5cdb4c272c568220eb1b154deb483dd1574597626a583361f0981"} Oct 30 16:51:52.642702 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:52.642512 5052 scope.go:122] "RemoveContainer" containerID="546d7652e7eba58d0c7db12381af0885040996537309f07c9e71cd75e0b85a5e" Oct 30 16:51:52.643969 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:52.643943 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-st7mw" Oct 30 16:51:52.643969 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:52.643958 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-st7mw" event={"ID":"dd769390-6903-4ed3-9ee9-814fb84762e9","Type":"ContainerDied","Data":"1349c3d88ed412195a24c3882e56cb4f41e23243596692eec9f474518e119b20"} Oct 30 16:51:52.645727 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:52.645701 5052 generic.go:356] "Generic (PLEG): container finished" podID="0a85b25b-991b-4f0d-9ef4-baab2b59d7b9" containerID="39aa2c7cc642c2b8d067e46e1f02243dab4630349de336f264af6342de2de15a" exitCode=0 Oct 30 16:51:52.645888 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:52.645745 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-5552t" Oct 30 16:51:52.645888 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:52.645781 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-5552t" event={"ID":"0a85b25b-991b-4f0d-9ef4-baab2b59d7b9","Type":"ContainerDied","Data":"39aa2c7cc642c2b8d067e46e1f02243dab4630349de336f264af6342de2de15a"} Oct 30 16:51:52.645888 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:52.645825 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-5552t" event={"ID":"0a85b25b-991b-4f0d-9ef4-baab2b59d7b9","Type":"ContainerDied","Data":"d0b4cebf28a2a0e43200e4d7c48ff12850cb0e3a416aaa5b8ac11a501d504d84"} Oct 30 16:51:52.657923 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:52.657907 5052 scope.go:122] "RemoveContainer" containerID="546d7652e7eba58d0c7db12381af0885040996537309f07c9e71cd75e0b85a5e" Oct 30 16:51:52.658224 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:51:52.658199 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"546d7652e7eba58d0c7db12381af0885040996537309f07c9e71cd75e0b85a5e\": container with ID starting with 546d7652e7eba58d0c7db12381af0885040996537309f07c9e71cd75e0b85a5e not found: ID does not exist" containerID="546d7652e7eba58d0c7db12381af0885040996537309f07c9e71cd75e0b85a5e" Oct 30 16:51:52.658267 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:52.658236 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"546d7652e7eba58d0c7db12381af0885040996537309f07c9e71cd75e0b85a5e"} err="failed to get container status \"546d7652e7eba58d0c7db12381af0885040996537309f07c9e71cd75e0b85a5e\": rpc error: code = NotFound desc = could not find container \"546d7652e7eba58d0c7db12381af0885040996537309f07c9e71cd75e0b85a5e\": container with ID starting with 546d7652e7eba58d0c7db12381af0885040996537309f07c9e71cd75e0b85a5e not found: ID does not exist" Oct 30 16:51:52.658267 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:52.658258 5052 scope.go:122] "RemoveContainer" containerID="f7eb5729a6832f55a0cd1d2dd4d9b05cb8a6f5a86b32ae79e079b85a74f79144" Oct 30 16:51:52.668347 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:52.668326 5052 scope.go:122] "RemoveContainer" containerID="39aa2c7cc642c2b8d067e46e1f02243dab4630349de336f264af6342de2de15a" Oct 30 16:51:52.678471 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:52.678455 5052 scope.go:122] "RemoveContainer" containerID="39aa2c7cc642c2b8d067e46e1f02243dab4630349de336f264af6342de2de15a" Oct 30 16:51:52.678709 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:51:52.678680 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39aa2c7cc642c2b8d067e46e1f02243dab4630349de336f264af6342de2de15a\": container with ID starting with 39aa2c7cc642c2b8d067e46e1f02243dab4630349de336f264af6342de2de15a not found: ID does not exist" containerID="39aa2c7cc642c2b8d067e46e1f02243dab4630349de336f264af6342de2de15a" Oct 30 16:51:52.678770 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:52.678703 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39aa2c7cc642c2b8d067e46e1f02243dab4630349de336f264af6342de2de15a"} err="failed to get container status \"39aa2c7cc642c2b8d067e46e1f02243dab4630349de336f264af6342de2de15a\": rpc error: code = NotFound desc = could not find container \"39aa2c7cc642c2b8d067e46e1f02243dab4630349de336f264af6342de2de15a\": container with ID starting with 39aa2c7cc642c2b8d067e46e1f02243dab4630349de336f264af6342de2de15a not found: ID does not exist" Oct 30 16:51:52.701361 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:52.701343 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-5552t"] Oct 30 16:51:52.715167 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:52.715146 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-5552t"] Oct 30 16:51:52.752738 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:52.752654 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-zhqln"] Oct 30 16:51:52.756480 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:52.756451 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-zhqln"] Oct 30 16:51:52.794548 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:52.794514 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-st7mw"] Oct 30 16:51:52.798745 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:52.798712 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-st7mw"] Oct 30 16:51:54.510850 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:54.510817 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-ca5971-openshift-default-f54d56b47-qrn9w"] Oct 30 16:51:54.511252 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:54.511074 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-ca5971-openshift-default-f54d56b47-qrn9w" podUID="cbb6daa4-cde3-4e3f-8343-1b8ca62f8390" containerName="istio-proxy" containerID="cri-o://5b98b79917c766e2f2ed7c3ee8d22e323fe8cc20d7b38e8161f6d39037082135" gracePeriod=30 Oct 30 16:51:54.599326 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:54.599297 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a85b25b-991b-4f0d-9ef4-baab2b59d7b9" path="/var/lib/kubelet/pods/0a85b25b-991b-4f0d-9ef4-baab2b59d7b9/volumes" Oct 30 16:51:54.599928 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:54.599912 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d9e35b9-801d-4b2d-aba2-c34409d041ad" path="/var/lib/kubelet/pods/9d9e35b9-801d-4b2d-aba2-c34409d041ad/volumes" Oct 30 16:51:54.600387 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:54.600374 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd769390-6903-4ed3-9ee9-814fb84762e9" path="/var/lib/kubelet/pods/dd769390-6903-4ed3-9ee9-814fb84762e9/volumes" Oct 30 16:51:56.060899 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:56.060865 5052 patch_prober.go:29] interesting pod/e2e-api-b-ca5971-openshift-default-f54d56b47-qrn9w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 16:51:56.061343 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:56.060925 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-ca5971-openshift-default-f54d56b47-qrn9w" podUID="cbb6daa4-cde3-4e3f-8343-1b8ca62f8390" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 16:51:56.253602 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:56.246747 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cdwcj"] Oct 30 16:51:59.734802 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:59.734763 5052 generic.go:356] "Generic (PLEG): container finished" podID="cbb6daa4-cde3-4e3f-8343-1b8ca62f8390" containerID="5b98b79917c766e2f2ed7c3ee8d22e323fe8cc20d7b38e8161f6d39037082135" exitCode=0 Oct 30 16:51:59.736431 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:59.735330 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ca5971-openshift-default-f54d56b47-qrn9w" event={"ID":"cbb6daa4-cde3-4e3f-8343-1b8ca62f8390","Type":"ContainerDied","Data":"5b98b79917c766e2f2ed7c3ee8d22e323fe8cc20d7b38e8161f6d39037082135"} Oct 30 16:51:59.736772 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:59.736718 5052 scope.go:122] "RemoveContainer" containerID="aed4f211ad9931452ee7d8b380b6e6031a24664f56d8843560dad937ceeb9356" Oct 30 16:51:59.901993 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:59.901967 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ca5971-openshift-default-f54d56b47-qrn9w" Oct 30 16:51:59.990168 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:59.990137 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbb6daa4-cde3-4e3f-8343-1b8ca62f8390-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbb6daa4-cde3-4e3f-8343-1b8ca62f8390-istiod-ca-cert\") pod \"cbb6daa4-cde3-4e3f-8343-1b8ca62f8390\" (UID: \"cbb6daa4-cde3-4e3f-8343-1b8ca62f8390\") " Oct 30 16:51:59.990338 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:59.990211 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbb6daa4-cde3-4e3f-8343-1b8ca62f8390-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbb6daa4-cde3-4e3f-8343-1b8ca62f8390-workload-socket\") pod \"cbb6daa4-cde3-4e3f-8343-1b8ca62f8390\" (UID: \"cbb6daa4-cde3-4e3f-8343-1b8ca62f8390\") " Oct 30 16:51:59.990425 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:59.990399 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbb6daa4-cde3-4e3f-8343-1b8ca62f8390-istio-token\" (UniqueName: \"kubernetes.io/projected/cbb6daa4-cde3-4e3f-8343-1b8ca62f8390-istio-token\") pod \"cbb6daa4-cde3-4e3f-8343-1b8ca62f8390\" (UID: \"cbb6daa4-cde3-4e3f-8343-1b8ca62f8390\") " Oct 30 16:51:59.990555 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:59.990522 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbb6daa4-cde3-4e3f-8343-1b8ca62f8390-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbb6daa4-cde3-4e3f-8343-1b8ca62f8390-workload-certs\") pod \"cbb6daa4-cde3-4e3f-8343-1b8ca62f8390\" (UID: \"cbb6daa4-cde3-4e3f-8343-1b8ca62f8390\") " Oct 30 16:51:59.990667 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:59.990556 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbb6daa4-cde3-4e3f-8343-1b8ca62f8390-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbb6daa4-cde3-4e3f-8343-1b8ca62f8390-istio-data\") pod \"cbb6daa4-cde3-4e3f-8343-1b8ca62f8390\" (UID: \"cbb6daa4-cde3-4e3f-8343-1b8ca62f8390\") " Oct 30 16:51:59.990667 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:59.990603 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbb6daa4-cde3-4e3f-8343-1b8ca62f8390-kube-api-access-dh6pq\" (UniqueName: \"kubernetes.io/projected/cbb6daa4-cde3-4e3f-8343-1b8ca62f8390-kube-api-access-dh6pq\") pod \"cbb6daa4-cde3-4e3f-8343-1b8ca62f8390\" (UID: \"cbb6daa4-cde3-4e3f-8343-1b8ca62f8390\") " Oct 30 16:51:59.990667 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:59.990571 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbb6daa4-cde3-4e3f-8343-1b8ca62f8390-istiod-ca-cert" pod "cbb6daa4-cde3-4e3f-8343-1b8ca62f8390" (UID: "cbb6daa4-cde3-4e3f-8343-1b8ca62f8390"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:51:59.990826 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:59.990708 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbb6daa4-cde3-4e3f-8343-1b8ca62f8390-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbb6daa4-cde3-4e3f-8343-1b8ca62f8390-istio-envoy\") pod \"cbb6daa4-cde3-4e3f-8343-1b8ca62f8390\" (UID: \"cbb6daa4-cde3-4e3f-8343-1b8ca62f8390\") " Oct 30 16:51:59.990826 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:59.990764 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbb6daa4-cde3-4e3f-8343-1b8ca62f8390-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbb6daa4-cde3-4e3f-8343-1b8ca62f8390-credential-socket\") pod \"cbb6daa4-cde3-4e3f-8343-1b8ca62f8390\" (UID: \"cbb6daa4-cde3-4e3f-8343-1b8ca62f8390\") " Oct 30 16:51:59.990826 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:59.990804 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cbb6daa4-cde3-4e3f-8343-1b8ca62f8390-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbb6daa4-cde3-4e3f-8343-1b8ca62f8390-istio-podinfo\") pod \"cbb6daa4-cde3-4e3f-8343-1b8ca62f8390\" (UID: \"cbb6daa4-cde3-4e3f-8343-1b8ca62f8390\") " Oct 30 16:51:59.990981 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:59.990913 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbb6daa4-cde3-4e3f-8343-1b8ca62f8390-workload-certs" pod "cbb6daa4-cde3-4e3f-8343-1b8ca62f8390" (UID: "cbb6daa4-cde3-4e3f-8343-1b8ca62f8390"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:51:59.991226 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:59.991179 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbb6daa4-cde3-4e3f-8343-1b8ca62f8390-istio-data" pod "cbb6daa4-cde3-4e3f-8343-1b8ca62f8390" (UID: "cbb6daa4-cde3-4e3f-8343-1b8ca62f8390"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:51:59.991365 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:59.991247 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbb6daa4-cde3-4e3f-8343-1b8ca62f8390-credential-socket" pod "cbb6daa4-cde3-4e3f-8343-1b8ca62f8390" (UID: "cbb6daa4-cde3-4e3f-8343-1b8ca62f8390"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:51:59.991471 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:59.991456 5052 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbb6daa4-cde3-4e3f-8343-1b8ca62f8390-workload-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:51:59.991540 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:59.991473 5052 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbb6daa4-cde3-4e3f-8343-1b8ca62f8390-istio-data\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:51:59.991540 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:59.991482 5052 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbb6daa4-cde3-4e3f-8343-1b8ca62f8390-credential-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:51:59.991540 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:59.991491 5052 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbb6daa4-cde3-4e3f-8343-1b8ca62f8390-istiod-ca-cert\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:51:59.991738 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:59.991725 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbb6daa4-cde3-4e3f-8343-1b8ca62f8390-workload-socket" pod "cbb6daa4-cde3-4e3f-8343-1b8ca62f8390" (UID: "cbb6daa4-cde3-4e3f-8343-1b8ca62f8390"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:51:59.993433 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:59.993406 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cbb6daa4-cde3-4e3f-8343-1b8ca62f8390-istio-podinfo" pod "cbb6daa4-cde3-4e3f-8343-1b8ca62f8390" (UID: "cbb6daa4-cde3-4e3f-8343-1b8ca62f8390"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:51:59.993678 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:59.993446 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbb6daa4-cde3-4e3f-8343-1b8ca62f8390-istio-envoy" pod "cbb6daa4-cde3-4e3f-8343-1b8ca62f8390" (UID: "cbb6daa4-cde3-4e3f-8343-1b8ca62f8390"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:51:59.993835 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:59.993813 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbb6daa4-cde3-4e3f-8343-1b8ca62f8390-kube-api-access-dh6pq" pod "cbb6daa4-cde3-4e3f-8343-1b8ca62f8390" (UID: "cbb6daa4-cde3-4e3f-8343-1b8ca62f8390"). InnerVolumeSpecName "kube-api-access-dh6pq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:51:59.993989 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:51:59.993964 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbb6daa4-cde3-4e3f-8343-1b8ca62f8390-istio-token" pod "cbb6daa4-cde3-4e3f-8343-1b8ca62f8390" (UID: "cbb6daa4-cde3-4e3f-8343-1b8ca62f8390"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:52:00.093108 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:00.093076 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dh6pq\" (UniqueName: \"kubernetes.io/projected/cbb6daa4-cde3-4e3f-8343-1b8ca62f8390-kube-api-access-dh6pq\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:52:00.093108 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:00.093104 5052 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbb6daa4-cde3-4e3f-8343-1b8ca62f8390-istio-envoy\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:52:00.093108 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:00.093115 5052 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbb6daa4-cde3-4e3f-8343-1b8ca62f8390-istio-podinfo\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:52:00.093342 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:00.093125 5052 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbb6daa4-cde3-4e3f-8343-1b8ca62f8390-workload-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:52:00.093342 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:00.093134 5052 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbb6daa4-cde3-4e3f-8343-1b8ca62f8390-istio-token\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:52:00.872888 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:00.865921 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ca5971-openshift-default-f54d56b47-qrn9w" event={"ID":"cbb6daa4-cde3-4e3f-8343-1b8ca62f8390","Type":"ContainerDied","Data":"ce4f146644a650cf6d64d0d656f90824e6011f7fabd8c666d6c7a4349fe21630"} Oct 30 16:52:00.872888 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:00.865987 5052 scope.go:122] "RemoveContainer" containerID="5b98b79917c766e2f2ed7c3ee8d22e323fe8cc20d7b38e8161f6d39037082135" Oct 30 16:52:00.872888 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:00.866125 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ca5971-openshift-default-f54d56b47-qrn9w" Oct 30 16:52:00.929313 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:00.929280 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-ca5971-openshift-default-f54d56b47-qrn9w"] Oct 30 16:52:00.941687 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:00.941656 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-ca5971-openshift-default-f54d56b47-qrn9w"] Oct 30 16:52:02.662691 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:02.662008 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbb6daa4-cde3-4e3f-8343-1b8ca62f8390" path="/var/lib/kubelet/pods/cbb6daa4-cde3-4e3f-8343-1b8ca62f8390/volumes" Oct 30 16:52:05.845625 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:05.845591 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-c4z7h"] Oct 30 16:52:05.846426 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:05.846408 5052 cpu_manager.go:414] "containerMap: removing container" podUID="cbb6daa4-cde3-4e3f-8343-1b8ca62f8390" containerName="istio-proxy" Oct 30 16:52:05.846426 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:05.846427 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbb6daa4-cde3-4e3f-8343-1b8ca62f8390" containerName="istio-proxy" Oct 30 16:52:05.846608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:05.846447 5052 cpu_manager.go:414] "containerMap: removing container" podUID="607c64a6-dc52-4862-b415-e25194a55649" containerName="maas-api" Oct 30 16:52:05.846608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:05.846452 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="607c64a6-dc52-4862-b415-e25194a55649" containerName="maas-api" Oct 30 16:52:05.846608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:05.846459 5052 cpu_manager.go:414] "containerMap: removing container" podUID="dd769390-6903-4ed3-9ee9-814fb84762e9" containerName="istio-proxy" Oct 30 16:52:05.846608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:05.846464 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd769390-6903-4ed3-9ee9-814fb84762e9" containerName="istio-proxy" Oct 30 16:52:05.846608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:05.846473 5052 cpu_manager.go:414] "containerMap: removing container" podUID="9d9e35b9-801d-4b2d-aba2-c34409d041ad" containerName="istio-proxy" Oct 30 16:52:05.846608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:05.846483 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d9e35b9-801d-4b2d-aba2-c34409d041ad" containerName="istio-proxy" Oct 30 16:52:05.846608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:05.846494 5052 cpu_manager.go:414] "containerMap: removing container" podUID="cbb6daa4-cde3-4e3f-8343-1b8ca62f8390" containerName="istio-proxy" Oct 30 16:52:05.846608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:05.846499 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbb6daa4-cde3-4e3f-8343-1b8ca62f8390" containerName="istio-proxy" Oct 30 16:52:05.846608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:05.846507 5052 cpu_manager.go:414] "containerMap: removing container" podUID="0a85b25b-991b-4f0d-9ef4-baab2b59d7b9" containerName="istio-proxy" Oct 30 16:52:05.846608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:05.846511 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a85b25b-991b-4f0d-9ef4-baab2b59d7b9" containerName="istio-proxy" Oct 30 16:52:05.846924 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:05.846685 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbb6daa4-cde3-4e3f-8343-1b8ca62f8390" containerName="istio-proxy" Oct 30 16:52:05.846924 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:05.846698 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="607c64a6-dc52-4862-b415-e25194a55649" containerName="maas-api" Oct 30 16:52:05.846924 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:05.846703 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd769390-6903-4ed3-9ee9-814fb84762e9" containerName="istio-proxy" Oct 30 16:52:05.846924 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:05.846710 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d9e35b9-801d-4b2d-aba2-c34409d041ad" containerName="istio-proxy" Oct 30 16:52:05.846924 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:05.846718 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbb6daa4-cde3-4e3f-8343-1b8ca62f8390" containerName="istio-proxy" Oct 30 16:52:05.846924 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:05.846722 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a85b25b-991b-4f0d-9ef4-baab2b59d7b9" containerName="istio-proxy" Oct 30 16:52:05.852388 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:05.852365 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-c4z7h" Oct 30 16:52:05.883999 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:05.883966 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-17f60f-openshift-default-dockercfg-65cvq\"" Oct 30 16:52:05.887913 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:05.887884 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-c4z7h"] Oct 30 16:52:06.014605 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:06.003926 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd7dff09-720a-411c-9c69-97bc38ac2178-workload-certs\") pod \"e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-c4z7h\" (UID: \"dd7dff09-720a-411c-9c69-97bc38ac2178\") " pod="openshift-ingress/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-c4z7h" Oct 30 16:52:06.014605 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:06.004067 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd7dff09-720a-411c-9c69-97bc38ac2178-istio-data\") pod \"e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-c4z7h\" (UID: \"dd7dff09-720a-411c-9c69-97bc38ac2178\") " pod="openshift-ingress/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-c4z7h" Oct 30 16:52:06.014605 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:06.004196 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd7dff09-720a-411c-9c69-97bc38ac2178-istio-podinfo\") pod \"e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-c4z7h\" (UID: \"dd7dff09-720a-411c-9c69-97bc38ac2178\") " pod="openshift-ingress/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-c4z7h" Oct 30 16:52:06.014605 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:06.004221 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd7dff09-720a-411c-9c69-97bc38ac2178-istio-token\") pod \"e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-c4z7h\" (UID: \"dd7dff09-720a-411c-9c69-97bc38ac2178\") " pod="openshift-ingress/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-c4z7h" Oct 30 16:52:06.014605 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:06.004273 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8z7cn\" (UniqueName: \"kubernetes.io/projected/dd7dff09-720a-411c-9c69-97bc38ac2178-kube-api-access-8z7cn\") pod \"e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-c4z7h\" (UID: \"dd7dff09-720a-411c-9c69-97bc38ac2178\") " pod="openshift-ingress/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-c4z7h" Oct 30 16:52:06.014605 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:06.004443 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd7dff09-720a-411c-9c69-97bc38ac2178-credential-socket\") pod \"e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-c4z7h\" (UID: \"dd7dff09-720a-411c-9c69-97bc38ac2178\") " pod="openshift-ingress/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-c4z7h" Oct 30 16:52:06.014605 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:06.004518 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd7dff09-720a-411c-9c69-97bc38ac2178-istiod-ca-cert\") pod \"e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-c4z7h\" (UID: \"dd7dff09-720a-411c-9c69-97bc38ac2178\") " pod="openshift-ingress/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-c4z7h" Oct 30 16:52:06.014605 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:06.004560 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd7dff09-720a-411c-9c69-97bc38ac2178-workload-socket\") pod \"e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-c4z7h\" (UID: \"dd7dff09-720a-411c-9c69-97bc38ac2178\") " pod="openshift-ingress/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-c4z7h" Oct 30 16:52:06.014605 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:06.004610 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd7dff09-720a-411c-9c69-97bc38ac2178-istio-envoy\") pod \"e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-c4z7h\" (UID: \"dd7dff09-720a-411c-9c69-97bc38ac2178\") " pod="openshift-ingress/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-c4z7h" Oct 30 16:52:06.180877 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:06.180835 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd7dff09-720a-411c-9c69-97bc38ac2178-istio-envoy\") pod \"e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-c4z7h\" (UID: \"dd7dff09-720a-411c-9c69-97bc38ac2178\") " pod="openshift-ingress/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-c4z7h" Oct 30 16:52:06.198719 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:06.198683 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd7dff09-720a-411c-9c69-97bc38ac2178-workload-certs\") pod \"e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-c4z7h\" (UID: \"dd7dff09-720a-411c-9c69-97bc38ac2178\") " pod="openshift-ingress/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-c4z7h" Oct 30 16:52:06.199032 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:06.199008 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd7dff09-720a-411c-9c69-97bc38ac2178-istio-data\") pod \"e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-c4z7h\" (UID: \"dd7dff09-720a-411c-9c69-97bc38ac2178\") " pod="openshift-ingress/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-c4z7h" Oct 30 16:52:06.199301 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:06.199281 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd7dff09-720a-411c-9c69-97bc38ac2178-istio-podinfo\") pod \"e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-c4z7h\" (UID: \"dd7dff09-720a-411c-9c69-97bc38ac2178\") " pod="openshift-ingress/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-c4z7h" Oct 30 16:52:06.199427 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:06.199412 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd7dff09-720a-411c-9c69-97bc38ac2178-istio-token\") pod \"e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-c4z7h\" (UID: \"dd7dff09-720a-411c-9c69-97bc38ac2178\") " pod="openshift-ingress/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-c4z7h" Oct 30 16:52:06.199695 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:06.199674 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8z7cn\" (UniqueName: \"kubernetes.io/projected/dd7dff09-720a-411c-9c69-97bc38ac2178-kube-api-access-8z7cn\") pod \"e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-c4z7h\" (UID: \"dd7dff09-720a-411c-9c69-97bc38ac2178\") " pod="openshift-ingress/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-c4z7h" Oct 30 16:52:06.199987 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:06.199965 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd7dff09-720a-411c-9c69-97bc38ac2178-credential-socket\") pod \"e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-c4z7h\" (UID: \"dd7dff09-720a-411c-9c69-97bc38ac2178\") " pod="openshift-ingress/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-c4z7h" Oct 30 16:52:06.200186 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:06.200166 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd7dff09-720a-411c-9c69-97bc38ac2178-istiod-ca-cert\") pod \"e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-c4z7h\" (UID: \"dd7dff09-720a-411c-9c69-97bc38ac2178\") " pod="openshift-ingress/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-c4z7h" Oct 30 16:52:06.200327 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:06.200311 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd7dff09-720a-411c-9c69-97bc38ac2178-workload-socket\") pod \"e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-c4z7h\" (UID: \"dd7dff09-720a-411c-9c69-97bc38ac2178\") " pod="openshift-ingress/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-c4z7h" Oct 30 16:52:06.200921 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:06.200898 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd7dff09-720a-411c-9c69-97bc38ac2178-workload-socket\") pod \"e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-c4z7h\" (UID: \"dd7dff09-720a-411c-9c69-97bc38ac2178\") " pod="openshift-ingress/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-c4z7h" Oct 30 16:52:06.201325 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:06.201303 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd7dff09-720a-411c-9c69-97bc38ac2178-workload-certs\") pod \"e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-c4z7h\" (UID: \"dd7dff09-720a-411c-9c69-97bc38ac2178\") " pod="openshift-ingress/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-c4z7h" Oct 30 16:52:06.219915 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:06.219878 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd7dff09-720a-411c-9c69-97bc38ac2178-istio-data\") pod \"e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-c4z7h\" (UID: \"dd7dff09-720a-411c-9c69-97bc38ac2178\") " pod="openshift-ingress/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-c4z7h" Oct 30 16:52:06.222967 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:06.222941 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd7dff09-720a-411c-9c69-97bc38ac2178-credential-socket\") pod \"e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-c4z7h\" (UID: \"dd7dff09-720a-411c-9c69-97bc38ac2178\") " pod="openshift-ingress/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-c4z7h" Oct 30 16:52:06.390654 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:06.223849 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd7dff09-720a-411c-9c69-97bc38ac2178-istiod-ca-cert\") pod \"e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-c4z7h\" (UID: \"dd7dff09-720a-411c-9c69-97bc38ac2178\") " pod="openshift-ingress/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-c4z7h" Oct 30 16:52:06.390654 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:06.346914 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd7dff09-720a-411c-9c69-97bc38ac2178-istio-token\") pod \"e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-c4z7h\" (UID: \"dd7dff09-720a-411c-9c69-97bc38ac2178\") " pod="openshift-ingress/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-c4z7h" Oct 30 16:52:06.392808 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:06.391442 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-7lh24"] Oct 30 16:52:06.392808 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:06.391749 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-7lh24" podUID="0e461432-f429-4153-9e32-954f2ca3fb61" containerName="istio-proxy" containerID="cri-o://1676a9ec909fdd11894cdf18ab1500259cbbb7eb1394f4717fe5cd66cb86f5d6" gracePeriod=30 Oct 30 16:52:06.409231 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:06.407550 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8z7cn\" (UniqueName: \"kubernetes.io/projected/dd7dff09-720a-411c-9c69-97bc38ac2178-kube-api-access-8z7cn\") pod \"e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-c4z7h\" (UID: \"dd7dff09-720a-411c-9c69-97bc38ac2178\") " pod="openshift-ingress/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-c4z7h" Oct 30 16:52:06.410023 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:06.409995 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd7dff09-720a-411c-9c69-97bc38ac2178-istio-podinfo\") pod \"e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-c4z7h\" (UID: \"dd7dff09-720a-411c-9c69-97bc38ac2178\") " pod="openshift-ingress/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-c4z7h" Oct 30 16:52:06.427104 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:06.423643 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd7dff09-720a-411c-9c69-97bc38ac2178-istio-envoy\") pod \"e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-c4z7h\" (UID: \"dd7dff09-720a-411c-9c69-97bc38ac2178\") " pod="openshift-ingress/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-c4z7h" Oct 30 16:52:06.503065 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:06.502990 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-c4z7h" Oct 30 16:52:06.848494 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:06.848463 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-c4z7h"] Oct 30 16:52:06.849201 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:52:06.849172 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd7dff09_720a_411c_9c69_97bc38ac2178.slice/crio-0169dd9ab829153b6613bad338316816820b7cb6c99b3612b15d72892eebf4b4 WatchSource:0}: Error finding container 0169dd9ab829153b6613bad338316816820b7cb6c99b3612b15d72892eebf4b4: Status 404 returned error can't find the container with id 0169dd9ab829153b6613bad338316816820b7cb6c99b3612b15d72892eebf4b4 Oct 30 16:52:06.851941 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:06.851909 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:52:06.852008 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:06.851978 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:52:06.852043 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:06.852007 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:52:07.320188 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:07.320149 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-c4z7h" event={"ID":"dd7dff09-720a-411c-9c69-97bc38ac2178","Type":"ContainerStarted","Data":"d320e735460998219605a75ccf5c0b2679ce5f7943727b561b6bacccaf698dab"} Oct 30 16:52:07.320373 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:07.320215 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-c4z7h" event={"ID":"dd7dff09-720a-411c-9c69-97bc38ac2178","Type":"ContainerStarted","Data":"0169dd9ab829153b6613bad338316816820b7cb6c99b3612b15d72892eebf4b4"} Oct 30 16:52:07.370481 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:07.370422 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-c4z7h" podStartSLOduration=2.370404354 podStartE2EDuration="2.370404354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:52:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:52:07.368757545 +0000 UTC m=+1882.896092768" watchObservedRunningTime="2026-10-30 16:52:07.370404354 +0000 UTC m=+1882.897739574" Oct 30 16:52:07.513033 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:07.510921 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-c4z7h" Oct 30 16:52:07.517877 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:07.516707 5052 patch_prober.go:29] interesting pod/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-c4z7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Oct 30 16:52:07.517877 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:07.516771 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-c4z7h" podUID="dd7dff09-720a-411c-9c69-97bc38ac2178" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Oct 30 16:52:08.508801 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:08.508775 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-c4z7h" Oct 30 16:52:08.629104 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:08.629062 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9bfcac-86969d48d7-j9dgf"] Oct 30 16:52:08.633732 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:08.633710 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9bfcac-86969d48d7-j9dgf" Oct 30 16:52:08.656812 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:08.656784 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9bfcac-86969d48d7-j9dgf"] Oct 30 16:52:08.716604 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:08.709784 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-9bfcac\"" Oct 30 16:52:08.726607 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:08.725142 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-9bfcac-dockercfg-n6pwx\"" Oct 30 16:52:08.806653 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:08.804200 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c979b538-a34c-49dc-8269-98ef5063e17c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-9bfcac-86969d48d7-j9dgf\" (UID: \"c979b538-a34c-49dc-8269-98ef5063e17c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9bfcac-86969d48d7-j9dgf" Oct 30 16:52:08.806653 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:08.804293 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vpjq\" (UniqueName: \"kubernetes.io/projected/c979b538-a34c-49dc-8269-98ef5063e17c-kube-api-access-8vpjq\") pod \"payload-pre-processing-e2e-inf-b-9bfcac-86969d48d7-j9dgf\" (UID: \"c979b538-a34c-49dc-8269-98ef5063e17c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9bfcac-86969d48d7-j9dgf" Oct 30 16:52:08.943448 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:08.937046 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c979b538-a34c-49dc-8269-98ef5063e17c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-9bfcac-86969d48d7-j9dgf\" (UID: \"c979b538-a34c-49dc-8269-98ef5063e17c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9bfcac-86969d48d7-j9dgf" Oct 30 16:52:08.943448 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:08.937157 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vpjq\" (UniqueName: \"kubernetes.io/projected/c979b538-a34c-49dc-8269-98ef5063e17c-kube-api-access-8vpjq\") pod \"payload-pre-processing-e2e-inf-b-9bfcac-86969d48d7-j9dgf\" (UID: \"c979b538-a34c-49dc-8269-98ef5063e17c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9bfcac-86969d48d7-j9dgf" Oct 30 16:52:08.943448 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:08.938851 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c979b538-a34c-49dc-8269-98ef5063e17c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-9bfcac-86969d48d7-j9dgf\" (UID: \"c979b538-a34c-49dc-8269-98ef5063e17c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9bfcac-86969d48d7-j9dgf" Oct 30 16:52:08.974645 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:08.972431 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vpjq\" (UniqueName: \"kubernetes.io/projected/c979b538-a34c-49dc-8269-98ef5063e17c-kube-api-access-8vpjq\") pod \"payload-pre-processing-e2e-inf-b-9bfcac-86969d48d7-j9dgf\" (UID: \"c979b538-a34c-49dc-8269-98ef5063e17c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9bfcac-86969d48d7-j9dgf" Oct 30 16:52:09.252005 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:09.251974 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9bfcac-86969d48d7-j9dgf" Oct 30 16:52:09.337758 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:09.337730 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-c4z7h" Oct 30 16:52:09.338835 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:09.338807 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-c4z7h" Oct 30 16:52:09.565574 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:52:09.565540 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc979b538_a34c_49dc_8269_98ef5063e17c.slice/crio-9efaee2db2339f8ed315537a2b1144a0f398eb7ce8be4a16aab93ded06e5688f WatchSource:0}: Error finding container 9efaee2db2339f8ed315537a2b1144a0f398eb7ce8be4a16aab93ded06e5688f: Status 404 returned error can't find the container with id 9efaee2db2339f8ed315537a2b1144a0f398eb7ce8be4a16aab93ded06e5688f Oct 30 16:52:09.566029 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:09.565706 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9bfcac-86969d48d7-j9dgf"] Oct 30 16:52:10.345952 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:10.345911 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9bfcac-86969d48d7-j9dgf" event={"ID":"c979b538-a34c-49dc-8269-98ef5063e17c","Type":"ContainerStarted","Data":"e3c481be9b5f6ee677c001e9f8fb8243806843fd8b61536d9b70617f8175315d"} Oct 30 16:52:10.346164 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:10.345974 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9bfcac-86969d48d7-j9dgf" event={"ID":"c979b538-a34c-49dc-8269-98ef5063e17c","Type":"ContainerStarted","Data":"9efaee2db2339f8ed315537a2b1144a0f398eb7ce8be4a16aab93ded06e5688f"} Oct 30 16:52:10.394347 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:10.394293 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9bfcac-86969d48d7-j9dgf" podStartSLOduration=2.394281312 podStartE2EDuration="2.394281312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:52:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:52:10.391801216 +0000 UTC m=+1885.919136438" watchObservedRunningTime="2026-10-30 16:52:10.394281312 +0000 UTC m=+1885.921616530" Oct 30 16:52:11.359624 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:11.354980 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9bfcac-86969d48d7-j9dgf" Oct 30 16:52:11.720679 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:11.720649 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-17f60f-5fbf6b59c9-vvk5s"] Oct 30 16:52:11.726920 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:11.726896 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-17f60f-5fbf6b59c9-vvk5s" Oct 30 16:52:11.729994 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:11.729971 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-7lh24" Oct 30 16:52:11.749385 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:11.749356 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-17f60f-5fbf6b59c9-vvk5s"] Oct 30 16:52:11.757189 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:11.757163 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-17f60f-dockercfg-dk94v\"" Oct 30 16:52:11.814223 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:11.814196 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-17f60f\"" Oct 30 16:52:11.824826 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:11.824794 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e461432-f429-4153-9e32-954f2ca3fb61-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e461432-f429-4153-9e32-954f2ca3fb61-istiod-ca-cert\") pod \"0e461432-f429-4153-9e32-954f2ca3fb61\" (UID: \"0e461432-f429-4153-9e32-954f2ca3fb61\") " Oct 30 16:52:11.824977 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:11.824961 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e461432-f429-4153-9e32-954f2ca3fb61-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e461432-f429-4153-9e32-954f2ca3fb61-istio-data\") pod \"0e461432-f429-4153-9e32-954f2ca3fb61\" (UID: \"0e461432-f429-4153-9e32-954f2ca3fb61\") " Oct 30 16:52:11.825070 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:11.825053 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e461432-f429-4153-9e32-954f2ca3fb61-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e461432-f429-4153-9e32-954f2ca3fb61-workload-socket\") pod \"0e461432-f429-4153-9e32-954f2ca3fb61\" (UID: \"0e461432-f429-4153-9e32-954f2ca3fb61\") " Oct 30 16:52:11.825141 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:11.825088 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e461432-f429-4153-9e32-954f2ca3fb61-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e461432-f429-4153-9e32-954f2ca3fb61-credential-socket\") pod \"0e461432-f429-4153-9e32-954f2ca3fb61\" (UID: \"0e461432-f429-4153-9e32-954f2ca3fb61\") " Oct 30 16:52:11.825141 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:11.825133 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e461432-f429-4153-9e32-954f2ca3fb61-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e461432-f429-4153-9e32-954f2ca3fb61-workload-certs\") pod \"0e461432-f429-4153-9e32-954f2ca3fb61\" (UID: \"0e461432-f429-4153-9e32-954f2ca3fb61\") " Oct 30 16:52:11.825257 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:11.825198 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e461432-f429-4153-9e32-954f2ca3fb61-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e461432-f429-4153-9e32-954f2ca3fb61-istio-envoy\") pod \"0e461432-f429-4153-9e32-954f2ca3fb61\" (UID: \"0e461432-f429-4153-9e32-954f2ca3fb61\") " Oct 30 16:52:11.825257 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:11.825217 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e461432-f429-4153-9e32-954f2ca3fb61-istiod-ca-cert" pod "0e461432-f429-4153-9e32-954f2ca3fb61" (UID: "0e461432-f429-4153-9e32-954f2ca3fb61"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:52:11.825371 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:11.825267 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0e461432-f429-4153-9e32-954f2ca3fb61-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e461432-f429-4153-9e32-954f2ca3fb61-istio-podinfo\") pod \"0e461432-f429-4153-9e32-954f2ca3fb61\" (UID: \"0e461432-f429-4153-9e32-954f2ca3fb61\") " Oct 30 16:52:11.825371 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:11.825319 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e461432-f429-4153-9e32-954f2ca3fb61-istio-data" pod "0e461432-f429-4153-9e32-954f2ca3fb61" (UID: "0e461432-f429-4153-9e32-954f2ca3fb61"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:52:11.825371 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:11.825350 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e461432-f429-4153-9e32-954f2ca3fb61-istio-token\" (UniqueName: \"kubernetes.io/projected/0e461432-f429-4153-9e32-954f2ca3fb61-istio-token\") pod \"0e461432-f429-4153-9e32-954f2ca3fb61\" (UID: \"0e461432-f429-4153-9e32-954f2ca3fb61\") " Oct 30 16:52:11.825525 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:11.825431 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e461432-f429-4153-9e32-954f2ca3fb61-workload-socket" pod "0e461432-f429-4153-9e32-954f2ca3fb61" (UID: "0e461432-f429-4153-9e32-954f2ca3fb61"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:52:11.825525 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:11.825471 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e461432-f429-4153-9e32-954f2ca3fb61-kube-api-access-kd7ww\" (UniqueName: \"kubernetes.io/projected/0e461432-f429-4153-9e32-954f2ca3fb61-kube-api-access-kd7ww\") pod \"0e461432-f429-4153-9e32-954f2ca3fb61\" (UID: \"0e461432-f429-4153-9e32-954f2ca3fb61\") " Oct 30 16:52:11.825943 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:11.825869 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4snk8\" (UniqueName: \"kubernetes.io/projected/141c2e28-1616-44fb-8e0b-275678b894b0-kube-api-access-4snk8\") pod \"payload-pre-processing-e2e-inf-a-17f60f-5fbf6b59c9-vvk5s\" (UID: \"141c2e28-1616-44fb-8e0b-275678b894b0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-17f60f-5fbf6b59c9-vvk5s" Oct 30 16:52:11.826096 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:11.826042 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e461432-f429-4153-9e32-954f2ca3fb61-credential-socket" pod "0e461432-f429-4153-9e32-954f2ca3fb61" (UID: "0e461432-f429-4153-9e32-954f2ca3fb61"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:52:11.826173 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:11.826145 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/141c2e28-1616-44fb-8e0b-275678b894b0-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-17f60f-5fbf6b59c9-vvk5s\" (UID: \"141c2e28-1616-44fb-8e0b-275678b894b0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-17f60f-5fbf6b59c9-vvk5s" Oct 30 16:52:11.826671 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:11.826339 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e461432-f429-4153-9e32-954f2ca3fb61-workload-certs" pod "0e461432-f429-4153-9e32-954f2ca3fb61" (UID: "0e461432-f429-4153-9e32-954f2ca3fb61"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:52:11.826671 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:11.826489 5052 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e461432-f429-4153-9e32-954f2ca3fb61-istiod-ca-cert\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:52:11.826671 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:11.826503 5052 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e461432-f429-4153-9e32-954f2ca3fb61-istio-data\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:52:11.826671 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:11.826513 5052 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e461432-f429-4153-9e32-954f2ca3fb61-workload-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:52:11.826671 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:11.826522 5052 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e461432-f429-4153-9e32-954f2ca3fb61-credential-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:52:11.826671 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:11.826534 5052 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e461432-f429-4153-9e32-954f2ca3fb61-workload-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:52:11.828239 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:11.828217 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e461432-f429-4153-9e32-954f2ca3fb61-istio-token" pod "0e461432-f429-4153-9e32-954f2ca3fb61" (UID: "0e461432-f429-4153-9e32-954f2ca3fb61"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:52:11.828374 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:11.828351 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0e461432-f429-4153-9e32-954f2ca3fb61-istio-podinfo" pod "0e461432-f429-4153-9e32-954f2ca3fb61" (UID: "0e461432-f429-4153-9e32-954f2ca3fb61"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:52:11.828685 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:11.828664 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e461432-f429-4153-9e32-954f2ca3fb61-istio-envoy" pod "0e461432-f429-4153-9e32-954f2ca3fb61" (UID: "0e461432-f429-4153-9e32-954f2ca3fb61"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:52:11.830089 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:11.830060 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e461432-f429-4153-9e32-954f2ca3fb61-kube-api-access-kd7ww" pod "0e461432-f429-4153-9e32-954f2ca3fb61" (UID: "0e461432-f429-4153-9e32-954f2ca3fb61"). InnerVolumeSpecName "kube-api-access-kd7ww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:52:11.928108 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:11.928075 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/141c2e28-1616-44fb-8e0b-275678b894b0-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-17f60f-5fbf6b59c9-vvk5s\" (UID: \"141c2e28-1616-44fb-8e0b-275678b894b0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-17f60f-5fbf6b59c9-vvk5s" Oct 30 16:52:11.928264 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:11.928248 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4snk8\" (UniqueName: \"kubernetes.io/projected/141c2e28-1616-44fb-8e0b-275678b894b0-kube-api-access-4snk8\") pod \"payload-pre-processing-e2e-inf-a-17f60f-5fbf6b59c9-vvk5s\" (UID: \"141c2e28-1616-44fb-8e0b-275678b894b0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-17f60f-5fbf6b59c9-vvk5s" Oct 30 16:52:11.928353 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:11.928342 5052 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e461432-f429-4153-9e32-954f2ca3fb61-istio-envoy\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:52:11.928396 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:11.928354 5052 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e461432-f429-4153-9e32-954f2ca3fb61-istio-podinfo\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:52:11.928396 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:11.928363 5052 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e461432-f429-4153-9e32-954f2ca3fb61-istio-token\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:52:11.928396 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:11.928372 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kd7ww\" (UniqueName: \"kubernetes.io/projected/0e461432-f429-4153-9e32-954f2ca3fb61-kube-api-access-kd7ww\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:52:11.929008 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:11.928984 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/141c2e28-1616-44fb-8e0b-275678b894b0-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-17f60f-5fbf6b59c9-vvk5s\" (UID: \"141c2e28-1616-44fb-8e0b-275678b894b0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-17f60f-5fbf6b59c9-vvk5s" Oct 30 16:52:12.041201 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:12.040751 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4snk8\" (UniqueName: \"kubernetes.io/projected/141c2e28-1616-44fb-8e0b-275678b894b0-kube-api-access-4snk8\") pod \"payload-pre-processing-e2e-inf-a-17f60f-5fbf6b59c9-vvk5s\" (UID: \"141c2e28-1616-44fb-8e0b-275678b894b0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-17f60f-5fbf6b59c9-vvk5s" Oct 30 16:52:12.041664 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:12.041087 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-17f60f-5fbf6b59c9-vvk5s" Oct 30 16:52:12.365698 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:12.365621 5052 generic.go:356] "Generic (PLEG): container finished" podID="0e461432-f429-4153-9e32-954f2ca3fb61" containerID="1676a9ec909fdd11894cdf18ab1500259cbbb7eb1394f4717fe5cd66cb86f5d6" exitCode=0 Oct 30 16:52:12.366176 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:12.365722 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-7lh24" Oct 30 16:52:12.366176 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:12.365754 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-7lh24" event={"ID":"0e461432-f429-4153-9e32-954f2ca3fb61","Type":"ContainerDied","Data":"1676a9ec909fdd11894cdf18ab1500259cbbb7eb1394f4717fe5cd66cb86f5d6"} Oct 30 16:52:12.366176 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:12.365823 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-7lh24" event={"ID":"0e461432-f429-4153-9e32-954f2ca3fb61","Type":"ContainerDied","Data":"6180db34cc71ab459ecae8298f7fb469f7ac64f43a603e8ea9fb81f19075e6a9"} Oct 30 16:52:12.366176 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:12.365859 5052 scope.go:122] "RemoveContainer" containerID="1676a9ec909fdd11894cdf18ab1500259cbbb7eb1394f4717fe5cd66cb86f5d6" Oct 30 16:52:12.382004 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:12.381921 5052 scope.go:122] "RemoveContainer" containerID="1676a9ec909fdd11894cdf18ab1500259cbbb7eb1394f4717fe5cd66cb86f5d6" Oct 30 16:52:12.393956 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:52:12.393911 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1676a9ec909fdd11894cdf18ab1500259cbbb7eb1394f4717fe5cd66cb86f5d6\": container with ID starting with 1676a9ec909fdd11894cdf18ab1500259cbbb7eb1394f4717fe5cd66cb86f5d6 not found: ID does not exist" containerID="1676a9ec909fdd11894cdf18ab1500259cbbb7eb1394f4717fe5cd66cb86f5d6" Oct 30 16:52:12.394177 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:12.393957 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1676a9ec909fdd11894cdf18ab1500259cbbb7eb1394f4717fe5cd66cb86f5d6"} err="failed to get container status \"1676a9ec909fdd11894cdf18ab1500259cbbb7eb1394f4717fe5cd66cb86f5d6\": rpc error: code = NotFound desc = could not find container \"1676a9ec909fdd11894cdf18ab1500259cbbb7eb1394f4717fe5cd66cb86f5d6\": container with ID starting with 1676a9ec909fdd11894cdf18ab1500259cbbb7eb1394f4717fe5cd66cb86f5d6 not found: ID does not exist" Oct 30 16:52:12.436254 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:12.436221 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-7lh24"] Oct 30 16:52:12.450505 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:12.450471 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-7lh24"] Oct 30 16:52:12.496909 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:12.496884 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-17f60f-5fbf6b59c9-vvk5s"] Oct 30 16:52:12.497420 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:52:12.497397 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod141c2e28_1616_44fb_8e0b_275678b894b0.slice/crio-d348ccc842cf3206d8ce42f6283a6d19759a460cf0c4736e43c9a65b24705190 WatchSource:0}: Error finding container d348ccc842cf3206d8ce42f6283a6d19759a460cf0c4736e43c9a65b24705190: Status 404 returned error can't find the container with id d348ccc842cf3206d8ce42f6283a6d19759a460cf0c4736e43c9a65b24705190 Oct 30 16:52:12.599687 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:12.599663 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e461432-f429-4153-9e32-954f2ca3fb61" path="/var/lib/kubelet/pods/0e461432-f429-4153-9e32-954f2ca3fb61/volumes" Oct 30 16:52:13.375417 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:13.375377 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-17f60f-5fbf6b59c9-vvk5s" event={"ID":"141c2e28-1616-44fb-8e0b-275678b894b0","Type":"ContainerStarted","Data":"a3802794e114812ee6b30885078114d3cc867ea9cf7d613aa2c08b2c678c5155"} Oct 30 16:52:13.375917 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:13.375466 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-17f60f-5fbf6b59c9-vvk5s" event={"ID":"141c2e28-1616-44fb-8e0b-275678b894b0","Type":"ContainerStarted","Data":"d348ccc842cf3206d8ce42f6283a6d19759a460cf0c4736e43c9a65b24705190"} Oct 30 16:52:13.375917 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:13.375624 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-17f60f-5fbf6b59c9-vvk5s" Oct 30 16:52:13.419142 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:13.419094 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-17f60f-5fbf6b59c9-vvk5s" podStartSLOduration=2.419081247 podStartE2EDuration="2.419081247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:52:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:52:13.418023943 +0000 UTC m=+1888.945359161" watchObservedRunningTime="2026-10-30 16:52:13.419081247 +0000 UTC m=+1888.946416502" Oct 30 16:52:18.845733 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:52:18.845679 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 16:52:22.369454 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:22.369425 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9bfcac-86969d48d7-j9dgf" Oct 30 16:52:24.402745 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:24.402716 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-17f60f-5fbf6b59c9-vvk5s" Oct 30 16:52:25.469451 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:25.469417 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3ed10a-6dcb868f5-2xwbx"] Oct 30 16:52:25.470431 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:25.470406 5052 cpu_manager.go:414] "containerMap: removing container" podUID="0e461432-f429-4153-9e32-954f2ca3fb61" containerName="istio-proxy" Oct 30 16:52:25.470431 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:25.470428 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e461432-f429-4153-9e32-954f2ca3fb61" containerName="istio-proxy" Oct 30 16:52:25.470733 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:25.470715 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e461432-f429-4153-9e32-954f2ca3fb61" containerName="istio-proxy" Oct 30 16:52:25.474751 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:25.474731 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3ed10a-6dcb868f5-2xwbx" Oct 30 16:52:25.499074 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:25.499044 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3ed10a-6dcb868f5-2xwbx"] Oct 30 16:52:25.603609 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:25.579112 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-3ed10a\"" Oct 30 16:52:25.603609 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:25.579291 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-3ed10a-dockercfg-m5r2m\"" Oct 30 16:52:25.622604 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:25.617103 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4d2755a-efbb-41f5-983f-309ee1cff113-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-3ed10a-6dcb868f5-2xwbx\" (UID: \"b4d2755a-efbb-41f5-983f-309ee1cff113\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3ed10a-6dcb868f5-2xwbx" Oct 30 16:52:25.622604 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:25.617169 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whqpx\" (UniqueName: \"kubernetes.io/projected/b4d2755a-efbb-41f5-983f-309ee1cff113-kube-api-access-whqpx\") pod \"payload-pre-processing-e2e-shared-a-w4-3ed10a-6dcb868f5-2xwbx\" (UID: \"b4d2755a-efbb-41f5-983f-309ee1cff113\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3ed10a-6dcb868f5-2xwbx" Oct 30 16:52:25.852553 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:25.852466 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4d2755a-efbb-41f5-983f-309ee1cff113-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-3ed10a-6dcb868f5-2xwbx\" (UID: \"b4d2755a-efbb-41f5-983f-309ee1cff113\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3ed10a-6dcb868f5-2xwbx" Oct 30 16:52:25.852553 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:25.852530 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whqpx\" (UniqueName: \"kubernetes.io/projected/b4d2755a-efbb-41f5-983f-309ee1cff113-kube-api-access-whqpx\") pod \"payload-pre-processing-e2e-shared-a-w4-3ed10a-6dcb868f5-2xwbx\" (UID: \"b4d2755a-efbb-41f5-983f-309ee1cff113\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3ed10a-6dcb868f5-2xwbx" Oct 30 16:52:25.853635 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:25.853614 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4d2755a-efbb-41f5-983f-309ee1cff113-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-3ed10a-6dcb868f5-2xwbx\" (UID: \"b4d2755a-efbb-41f5-983f-309ee1cff113\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3ed10a-6dcb868f5-2xwbx" Oct 30 16:52:25.877796 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:25.877771 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whqpx\" (UniqueName: \"kubernetes.io/projected/b4d2755a-efbb-41f5-983f-309ee1cff113-kube-api-access-whqpx\") pod \"payload-pre-processing-e2e-shared-a-w4-3ed10a-6dcb868f5-2xwbx\" (UID: \"b4d2755a-efbb-41f5-983f-309ee1cff113\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3ed10a-6dcb868f5-2xwbx" Oct 30 16:52:26.090093 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:26.090063 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3ed10a-6dcb868f5-2xwbx" Oct 30 16:52:26.466846 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:26.466813 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3ed10a-6dcb868f5-2xwbx"] Oct 30 16:52:26.467630 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:52:26.467603 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4d2755a_efbb_41f5_983f_309ee1cff113.slice/crio-da93b69d4ad31ee5ff02294a691e0d2187b1e74bcf1b71dc54b58756b5e8dc89 WatchSource:0}: Error finding container da93b69d4ad31ee5ff02294a691e0d2187b1e74bcf1b71dc54b58756b5e8dc89: Status 404 returned error can't find the container with id da93b69d4ad31ee5ff02294a691e0d2187b1e74bcf1b71dc54b58756b5e8dc89 Oct 30 16:52:26.623999 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:26.623968 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3ed10a-6dcb868f5-2xwbx" event={"ID":"b4d2755a-efbb-41f5-983f-309ee1cff113","Type":"ContainerStarted","Data":"510c0776948b928bb418a1d124f9c173d33a14c4345219c0fc13a1b771033a89"} Oct 30 16:52:26.624368 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:26.624043 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3ed10a-6dcb868f5-2xwbx" Oct 30 16:52:26.624368 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:26.624068 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3ed10a-6dcb868f5-2xwbx" event={"ID":"b4d2755a-efbb-41f5-983f-309ee1cff113","Type":"ContainerStarted","Data":"da93b69d4ad31ee5ff02294a691e0d2187b1e74bcf1b71dc54b58756b5e8dc89"} Oct 30 16:52:26.670444 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:26.670384 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3ed10a-6dcb868f5-2xwbx" podStartSLOduration=1.670365927 podStartE2EDuration="1.670365927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:52:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:52:26.666506851 +0000 UTC m=+1902.193842073" watchObservedRunningTime="2026-10-30 16:52:26.670365927 +0000 UTC m=+1902.197701146" Oct 30 16:52:37.698671 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:37.688199 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3ed10a-6dcb868f5-2xwbx" Oct 30 16:52:38.992106 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:38.992069 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-407eab-86959ffcb8-dwfh8"] Oct 30 16:52:38.992549 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:38.992331 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-407eab-86959ffcb8-dwfh8" podUID="bf584334-297c-4316-a308-fdbd217e5011" containerName="maas-api" containerID="cri-o://a0fe8427ce43831b4cf8763a54c1a543266c101e8907d3659baa17426dbab54f" gracePeriod=30 Oct 30 16:52:39.657921 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:39.657888 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-407eab-86959ffcb8-dwfh8" Oct 30 16:52:39.765133 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:39.765100 5052 generic.go:356] "Generic (PLEG): container finished" podID="bf584334-297c-4316-a308-fdbd217e5011" containerID="a0fe8427ce43831b4cf8763a54c1a543266c101e8907d3659baa17426dbab54f" exitCode=0 Oct 30 16:52:39.765397 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:39.765376 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-407eab-86959ffcb8-dwfh8" event={"ID":"bf584334-297c-4316-a308-fdbd217e5011","Type":"ContainerDied","Data":"a0fe8427ce43831b4cf8763a54c1a543266c101e8907d3659baa17426dbab54f"} Oct 30 16:52:39.765492 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:39.765434 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-407eab-86959ffcb8-dwfh8" event={"ID":"bf584334-297c-4316-a308-fdbd217e5011","Type":"ContainerDied","Data":"2b33847dc0e8d96a5b38a44e1c903b9298d72a477dd57244401ccead6b3bf59c"} Oct 30 16:52:39.765492 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:39.765466 5052 scope.go:122] "RemoveContainer" containerID="a0fe8427ce43831b4cf8763a54c1a543266c101e8907d3659baa17426dbab54f" Oct 30 16:52:39.765685 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:39.765658 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-407eab-86959ffcb8-dwfh8" Oct 30 16:52:39.776173 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:39.776147 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf584334-297c-4316-a308-fdbd217e5011-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf584334-297c-4316-a308-fdbd217e5011-metrics-certs\") pod \"bf584334-297c-4316-a308-fdbd217e5011\" (UID: \"bf584334-297c-4316-a308-fdbd217e5011\") " Oct 30 16:52:39.776297 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:39.776189 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf584334-297c-4316-a308-fdbd217e5011-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf584334-297c-4316-a308-fdbd217e5011-maas-api-tls\") pod \"bf584334-297c-4316-a308-fdbd217e5011\" (UID: \"bf584334-297c-4316-a308-fdbd217e5011\") " Oct 30 16:52:39.776297 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:39.776284 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf584334-297c-4316-a308-fdbd217e5011-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf584334-297c-4316-a308-fdbd217e5011-openshift-service-ca\") pod \"bf584334-297c-4316-a308-fdbd217e5011\" (UID: \"bf584334-297c-4316-a308-fdbd217e5011\") " Oct 30 16:52:39.776414 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:39.776390 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf584334-297c-4316-a308-fdbd217e5011-kube-api-access-js9fp\" (UniqueName: \"kubernetes.io/projected/bf584334-297c-4316-a308-fdbd217e5011-kube-api-access-js9fp\") pod \"bf584334-297c-4316-a308-fdbd217e5011\" (UID: \"bf584334-297c-4316-a308-fdbd217e5011\") " Oct 30 16:52:39.780630 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:39.780542 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf584334-297c-4316-a308-fdbd217e5011-kube-api-access-js9fp" pod "bf584334-297c-4316-a308-fdbd217e5011" (UID: "bf584334-297c-4316-a308-fdbd217e5011"). InnerVolumeSpecName "kube-api-access-js9fp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:52:39.784508 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:39.784462 5052 scope.go:122] "RemoveContainer" containerID="a0fe8427ce43831b4cf8763a54c1a543266c101e8907d3659baa17426dbab54f" Oct 30 16:52:39.785007 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:39.784764 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf584334-297c-4316-a308-fdbd217e5011-openshift-service-ca" pod "bf584334-297c-4316-a308-fdbd217e5011" (UID: "bf584334-297c-4316-a308-fdbd217e5011"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:52:39.785007 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:52:39.784817 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0fe8427ce43831b4cf8763a54c1a543266c101e8907d3659baa17426dbab54f\": container with ID starting with a0fe8427ce43831b4cf8763a54c1a543266c101e8907d3659baa17426dbab54f not found: ID does not exist" containerID="a0fe8427ce43831b4cf8763a54c1a543266c101e8907d3659baa17426dbab54f" Oct 30 16:52:39.785007 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:39.784850 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0fe8427ce43831b4cf8763a54c1a543266c101e8907d3659baa17426dbab54f"} err="failed to get container status \"a0fe8427ce43831b4cf8763a54c1a543266c101e8907d3659baa17426dbab54f\": rpc error: code = NotFound desc = could not find container \"a0fe8427ce43831b4cf8763a54c1a543266c101e8907d3659baa17426dbab54f\": container with ID starting with a0fe8427ce43831b4cf8763a54c1a543266c101e8907d3659baa17426dbab54f not found: ID does not exist" Oct 30 16:52:39.785261 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:39.785018 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf584334-297c-4316-a308-fdbd217e5011-maas-api-tls" pod "bf584334-297c-4316-a308-fdbd217e5011" (UID: "bf584334-297c-4316-a308-fdbd217e5011"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:52:39.786489 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:39.786455 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf584334-297c-4316-a308-fdbd217e5011-metrics-certs" pod "bf584334-297c-4316-a308-fdbd217e5011" (UID: "bf584334-297c-4316-a308-fdbd217e5011"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:52:39.878282 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:39.878251 5052 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf584334-297c-4316-a308-fdbd217e5011-maas-api-tls\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:52:39.878282 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:39.878278 5052 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf584334-297c-4316-a308-fdbd217e5011-openshift-service-ca\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:52:39.878282 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:39.878288 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-js9fp\" (UniqueName: \"kubernetes.io/projected/bf584334-297c-4316-a308-fdbd217e5011-kube-api-access-js9fp\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:52:39.878528 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:39.878298 5052 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf584334-297c-4316-a308-fdbd217e5011-metrics-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:52:40.160888 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:40.160856 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-407eab-86959ffcb8-dwfh8"] Oct 30 16:52:40.165904 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:40.165875 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-407eab-86959ffcb8-dwfh8"] Oct 30 16:52:40.599796 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:40.599754 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf584334-297c-4316-a308-fdbd217e5011" path="/var/lib/kubelet/pods/bf584334-297c-4316-a308-fdbd217e5011/volumes" Oct 30 16:52:45.654536 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:45.654451 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9bfcac-6bb4c44994-95lp4"] Oct 30 16:52:45.655297 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:45.655280 5052 cpu_manager.go:414] "containerMap: removing container" podUID="bf584334-297c-4316-a308-fdbd217e5011" containerName="maas-api" Oct 30 16:52:45.655345 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:45.655301 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf584334-297c-4316-a308-fdbd217e5011" containerName="maas-api" Oct 30 16:52:45.655433 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:45.655424 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf584334-297c-4316-a308-fdbd217e5011" containerName="maas-api" Oct 30 16:52:45.660896 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:45.660872 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9bfcac-6bb4c44994-95lp4" Oct 30 16:52:45.708625 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:45.702508 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9bfcac-6bb4c44994-95lp4"] Oct 30 16:52:45.714747 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:45.714718 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-9bfcac-serving-cert\"" Oct 30 16:52:45.774220 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:45.774190 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5898d00-6f84-40ef-890f-a476895b9690-maas-api-tls\") pod \"maas-api-e2e-inf-b-9bfcac-6bb4c44994-95lp4\" (UID: \"c5898d00-6f84-40ef-890f-a476895b9690\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9bfcac-6bb4c44994-95lp4" Oct 30 16:52:45.774412 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:45.774251 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7q468\" (UniqueName: \"kubernetes.io/projected/c5898d00-6f84-40ef-890f-a476895b9690-kube-api-access-7q468\") pod \"maas-api-e2e-inf-b-9bfcac-6bb4c44994-95lp4\" (UID: \"c5898d00-6f84-40ef-890f-a476895b9690\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9bfcac-6bb4c44994-95lp4" Oct 30 16:52:45.774576 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:45.774555 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5898d00-6f84-40ef-890f-a476895b9690-openshift-service-ca\") pod \"maas-api-e2e-inf-b-9bfcac-6bb4c44994-95lp4\" (UID: \"c5898d00-6f84-40ef-890f-a476895b9690\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9bfcac-6bb4c44994-95lp4" Oct 30 16:52:45.774718 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:45.774699 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5898d00-6f84-40ef-890f-a476895b9690-metrics-certs\") pod \"maas-api-e2e-inf-b-9bfcac-6bb4c44994-95lp4\" (UID: \"c5898d00-6f84-40ef-890f-a476895b9690\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9bfcac-6bb4c44994-95lp4" Oct 30 16:52:45.890348 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:45.888725 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5898d00-6f84-40ef-890f-a476895b9690-maas-api-tls\") pod \"maas-api-e2e-inf-b-9bfcac-6bb4c44994-95lp4\" (UID: \"c5898d00-6f84-40ef-890f-a476895b9690\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9bfcac-6bb4c44994-95lp4" Oct 30 16:52:45.890348 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:45.888874 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7q468\" (UniqueName: \"kubernetes.io/projected/c5898d00-6f84-40ef-890f-a476895b9690-kube-api-access-7q468\") pod \"maas-api-e2e-inf-b-9bfcac-6bb4c44994-95lp4\" (UID: \"c5898d00-6f84-40ef-890f-a476895b9690\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9bfcac-6bb4c44994-95lp4" Oct 30 16:52:45.890348 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:45.889117 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5898d00-6f84-40ef-890f-a476895b9690-openshift-service-ca\") pod \"maas-api-e2e-inf-b-9bfcac-6bb4c44994-95lp4\" (UID: \"c5898d00-6f84-40ef-890f-a476895b9690\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9bfcac-6bb4c44994-95lp4" Oct 30 16:52:45.890348 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:45.889187 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5898d00-6f84-40ef-890f-a476895b9690-metrics-certs\") pod \"maas-api-e2e-inf-b-9bfcac-6bb4c44994-95lp4\" (UID: \"c5898d00-6f84-40ef-890f-a476895b9690\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9bfcac-6bb4c44994-95lp4" Oct 30 16:52:45.895600 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:45.895330 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5898d00-6f84-40ef-890f-a476895b9690-openshift-service-ca\") pod \"maas-api-e2e-inf-b-9bfcac-6bb4c44994-95lp4\" (UID: \"c5898d00-6f84-40ef-890f-a476895b9690\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9bfcac-6bb4c44994-95lp4" Oct 30 16:52:45.900913 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:45.900435 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5898d00-6f84-40ef-890f-a476895b9690-metrics-certs\") pod \"maas-api-e2e-inf-b-9bfcac-6bb4c44994-95lp4\" (UID: \"c5898d00-6f84-40ef-890f-a476895b9690\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9bfcac-6bb4c44994-95lp4" Oct 30 16:52:45.910433 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:45.907497 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5898d00-6f84-40ef-890f-a476895b9690-maas-api-tls\") pod \"maas-api-e2e-inf-b-9bfcac-6bb4c44994-95lp4\" (UID: \"c5898d00-6f84-40ef-890f-a476895b9690\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9bfcac-6bb4c44994-95lp4" Oct 30 16:52:45.939603 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:45.928795 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7q468\" (UniqueName: \"kubernetes.io/projected/c5898d00-6f84-40ef-890f-a476895b9690-kube-api-access-7q468\") pod \"maas-api-e2e-inf-b-9bfcac-6bb4c44994-95lp4\" (UID: \"c5898d00-6f84-40ef-890f-a476895b9690\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9bfcac-6bb4c44994-95lp4" Oct 30 16:52:45.985569 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:45.979898 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9bfcac-6bb4c44994-95lp4" Oct 30 16:52:46.328108 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:46.328078 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9bfcac-6bb4c44994-95lp4"] Oct 30 16:52:46.328778 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:52:46.328753 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5898d00_6f84_40ef_890f_a476895b9690.slice/crio-3e21c21e037e4a1f26036b1a37b21d3bee2ee0960b4fc94e36f83a0b30388851 WatchSource:0}: Error finding container 3e21c21e037e4a1f26036b1a37b21d3bee2ee0960b4fc94e36f83a0b30388851: Status 404 returned error can't find the container with id 3e21c21e037e4a1f26036b1a37b21d3bee2ee0960b4fc94e36f83a0b30388851 Oct 30 16:52:46.330299 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:46.330281 5052 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 16:52:46.831750 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:46.831714 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9bfcac-6bb4c44994-95lp4" event={"ID":"c5898d00-6f84-40ef-890f-a476895b9690","Type":"ContainerStarted","Data":"2f92983ee0985e683975e6113285cb593685a38d154642a58581a572efc9c673"} Oct 30 16:52:46.832216 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:46.831773 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9bfcac-6bb4c44994-95lp4" event={"ID":"c5898d00-6f84-40ef-890f-a476895b9690","Type":"ContainerStarted","Data":"3e21c21e037e4a1f26036b1a37b21d3bee2ee0960b4fc94e36f83a0b30388851"} Oct 30 16:52:46.832216 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:46.831815 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9bfcac-6bb4c44994-95lp4" Oct 30 16:52:46.892933 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:46.892874 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9bfcac-6bb4c44994-95lp4" podStartSLOduration=1.513572259 podStartE2EDuration="1.89285597s" totalImagesPullingTime="379.283711ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:52:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:52:46.888652638 +0000 UTC m=+1922.415987854" watchObservedRunningTime="2026-10-30 16:52:46.89285597 +0000 UTC m=+1922.420191190" Oct 30 16:52:48.841826 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:52:48.841785 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 16:52:49.538972 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:49.538939 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-407eab-openshift-default-85f774ff9c-nldmk"] Oct 30 16:52:49.539226 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:49.539198 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-407eab-openshift-default-85f774ff9c-nldmk" podUID="218cbb21-a450-4352-b4b3-f7e8558688bb" containerName="istio-proxy" containerID="cri-o://cf98a641a60e69324dd36c184f61b43dc9e0ef04ff6c274019a05a026b4dab99" gracePeriod=30 Oct 30 16:52:52.845381 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:52.845351 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9bfcac-6bb4c44994-95lp4" Oct 30 16:52:54.807982 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:54.807918 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-407eab-openshift-default-85f774ff9c-nldmk" Oct 30 16:52:54.924346 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:54.924311 5052 generic.go:356] "Generic (PLEG): container finished" podID="218cbb21-a450-4352-b4b3-f7e8558688bb" containerID="cf98a641a60e69324dd36c184f61b43dc9e0ef04ff6c274019a05a026b4dab99" exitCode=0 Oct 30 16:52:54.924544 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:54.924389 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-407eab-openshift-default-85f774ff9c-nldmk" Oct 30 16:52:54.924544 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:54.924408 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-407eab-openshift-default-85f774ff9c-nldmk" event={"ID":"218cbb21-a450-4352-b4b3-f7e8558688bb","Type":"ContainerDied","Data":"cf98a641a60e69324dd36c184f61b43dc9e0ef04ff6c274019a05a026b4dab99"} Oct 30 16:52:54.924544 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:54.924476 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-407eab-openshift-default-85f774ff9c-nldmk" event={"ID":"218cbb21-a450-4352-b4b3-f7e8558688bb","Type":"ContainerDied","Data":"c222361e8b16ea70109b930ef59793198858cbd634c16172dbc547c11ff32bcb"} Oct 30 16:52:54.924544 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:54.924510 5052 scope.go:122] "RemoveContainer" containerID="cf98a641a60e69324dd36c184f61b43dc9e0ef04ff6c274019a05a026b4dab99" Oct 30 16:52:54.939536 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:54.939511 5052 scope.go:122] "RemoveContainer" containerID="5533cf1b92cd792dfcb6a1953f72d0abedaee4cb23f48905192a29a1642e75f3" Oct 30 16:52:54.946315 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:54.946285 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/218cbb21-a450-4352-b4b3-f7e8558688bb-istio-token\" (UniqueName: \"kubernetes.io/projected/218cbb21-a450-4352-b4b3-f7e8558688bb-istio-token\") pod \"218cbb21-a450-4352-b4b3-f7e8558688bb\" (UID: \"218cbb21-a450-4352-b4b3-f7e8558688bb\") " Oct 30 16:52:54.946450 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:54.946431 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/218cbb21-a450-4352-b4b3-f7e8558688bb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/218cbb21-a450-4352-b4b3-f7e8558688bb-istio-podinfo\") pod \"218cbb21-a450-4352-b4b3-f7e8558688bb\" (UID: \"218cbb21-a450-4352-b4b3-f7e8558688bb\") " Oct 30 16:52:54.946533 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:54.946503 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/218cbb21-a450-4352-b4b3-f7e8558688bb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/218cbb21-a450-4352-b4b3-f7e8558688bb-istiod-ca-cert\") pod \"218cbb21-a450-4352-b4b3-f7e8558688bb\" (UID: \"218cbb21-a450-4352-b4b3-f7e8558688bb\") " Oct 30 16:52:54.946636 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:54.946612 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/218cbb21-a450-4352-b4b3-f7e8558688bb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/218cbb21-a450-4352-b4b3-f7e8558688bb-workload-socket\") pod \"218cbb21-a450-4352-b4b3-f7e8558688bb\" (UID: \"218cbb21-a450-4352-b4b3-f7e8558688bb\") " Oct 30 16:52:54.946692 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:54.946643 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/218cbb21-a450-4352-b4b3-f7e8558688bb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/218cbb21-a450-4352-b4b3-f7e8558688bb-istio-data\") pod \"218cbb21-a450-4352-b4b3-f7e8558688bb\" (UID: \"218cbb21-a450-4352-b4b3-f7e8558688bb\") " Oct 30 16:52:54.946750 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:54.946720 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/218cbb21-a450-4352-b4b3-f7e8558688bb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/218cbb21-a450-4352-b4b3-f7e8558688bb-istio-envoy\") pod \"218cbb21-a450-4352-b4b3-f7e8558688bb\" (UID: \"218cbb21-a450-4352-b4b3-f7e8558688bb\") " Oct 30 16:52:54.946812 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:54.946753 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/218cbb21-a450-4352-b4b3-f7e8558688bb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/218cbb21-a450-4352-b4b3-f7e8558688bb-credential-socket\") pod \"218cbb21-a450-4352-b4b3-f7e8558688bb\" (UID: \"218cbb21-a450-4352-b4b3-f7e8558688bb\") " Oct 30 16:52:54.946870 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:54.946860 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/218cbb21-a450-4352-b4b3-f7e8558688bb-kube-api-access-2nztq\" (UniqueName: \"kubernetes.io/projected/218cbb21-a450-4352-b4b3-f7e8558688bb-kube-api-access-2nztq\") pod \"218cbb21-a450-4352-b4b3-f7e8558688bb\" (UID: \"218cbb21-a450-4352-b4b3-f7e8558688bb\") " Oct 30 16:52:54.946930 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:54.946891 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/218cbb21-a450-4352-b4b3-f7e8558688bb-workload-socket" pod "218cbb21-a450-4352-b4b3-f7e8558688bb" (UID: "218cbb21-a450-4352-b4b3-f7e8558688bb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:52:54.946930 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:54.946910 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/218cbb21-a450-4352-b4b3-f7e8558688bb-istiod-ca-cert" pod "218cbb21-a450-4352-b4b3-f7e8558688bb" (UID: "218cbb21-a450-4352-b4b3-f7e8558688bb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:52:54.947037 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:54.946996 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/218cbb21-a450-4352-b4b3-f7e8558688bb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/218cbb21-a450-4352-b4b3-f7e8558688bb-workload-certs\") pod \"218cbb21-a450-4352-b4b3-f7e8558688bb\" (UID: \"218cbb21-a450-4352-b4b3-f7e8558688bb\") " Oct 30 16:52:54.947037 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:54.947010 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/218cbb21-a450-4352-b4b3-f7e8558688bb-istio-data" pod "218cbb21-a450-4352-b4b3-f7e8558688bb" (UID: "218cbb21-a450-4352-b4b3-f7e8558688bb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:52:54.947173 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:54.947151 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/218cbb21-a450-4352-b4b3-f7e8558688bb-credential-socket" pod "218cbb21-a450-4352-b4b3-f7e8558688bb" (UID: "218cbb21-a450-4352-b4b3-f7e8558688bb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:52:54.947310 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:54.947290 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/218cbb21-a450-4352-b4b3-f7e8558688bb-workload-certs" pod "218cbb21-a450-4352-b4b3-f7e8558688bb" (UID: "218cbb21-a450-4352-b4b3-f7e8558688bb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:52:54.947887 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:54.947863 5052 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/218cbb21-a450-4352-b4b3-f7e8558688bb-workload-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:52:54.947987 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:54.947895 5052 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/218cbb21-a450-4352-b4b3-f7e8558688bb-istiod-ca-cert\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:52:54.947987 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:54.947911 5052 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/218cbb21-a450-4352-b4b3-f7e8558688bb-workload-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:52:54.947987 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:54.947926 5052 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/218cbb21-a450-4352-b4b3-f7e8558688bb-istio-data\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:52:54.947987 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:54.947942 5052 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/218cbb21-a450-4352-b4b3-f7e8558688bb-credential-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:52:54.948849 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:54.948824 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/218cbb21-a450-4352-b4b3-f7e8558688bb-istio-token" pod "218cbb21-a450-4352-b4b3-f7e8558688bb" (UID: "218cbb21-a450-4352-b4b3-f7e8558688bb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:52:54.949116 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:54.949091 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/218cbb21-a450-4352-b4b3-f7e8558688bb-istio-podinfo" pod "218cbb21-a450-4352-b4b3-f7e8558688bb" (UID: "218cbb21-a450-4352-b4b3-f7e8558688bb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:52:54.949218 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:54.949087 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/218cbb21-a450-4352-b4b3-f7e8558688bb-istio-envoy" pod "218cbb21-a450-4352-b4b3-f7e8558688bb" (UID: "218cbb21-a450-4352-b4b3-f7e8558688bb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:52:54.949337 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:54.949323 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/218cbb21-a450-4352-b4b3-f7e8558688bb-kube-api-access-2nztq" pod "218cbb21-a450-4352-b4b3-f7e8558688bb" (UID: "218cbb21-a450-4352-b4b3-f7e8558688bb"). InnerVolumeSpecName "kube-api-access-2nztq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:52:54.952600 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:54.952568 5052 scope.go:122] "RemoveContainer" containerID="cf98a641a60e69324dd36c184f61b43dc9e0ef04ff6c274019a05a026b4dab99" Oct 30 16:52:54.952890 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:52:54.952871 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf98a641a60e69324dd36c184f61b43dc9e0ef04ff6c274019a05a026b4dab99\": container with ID starting with cf98a641a60e69324dd36c184f61b43dc9e0ef04ff6c274019a05a026b4dab99 not found: ID does not exist" containerID="cf98a641a60e69324dd36c184f61b43dc9e0ef04ff6c274019a05a026b4dab99" Oct 30 16:52:54.952951 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:54.952902 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf98a641a60e69324dd36c184f61b43dc9e0ef04ff6c274019a05a026b4dab99"} err="failed to get container status \"cf98a641a60e69324dd36c184f61b43dc9e0ef04ff6c274019a05a026b4dab99\": rpc error: code = NotFound desc = could not find container \"cf98a641a60e69324dd36c184f61b43dc9e0ef04ff6c274019a05a026b4dab99\": container with ID starting with cf98a641a60e69324dd36c184f61b43dc9e0ef04ff6c274019a05a026b4dab99 not found: ID does not exist" Oct 30 16:52:54.952951 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:54.952924 5052 scope.go:122] "RemoveContainer" containerID="5533cf1b92cd792dfcb6a1953f72d0abedaee4cb23f48905192a29a1642e75f3" Oct 30 16:52:54.953205 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:52:54.953188 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5533cf1b92cd792dfcb6a1953f72d0abedaee4cb23f48905192a29a1642e75f3\": container with ID starting with 5533cf1b92cd792dfcb6a1953f72d0abedaee4cb23f48905192a29a1642e75f3 not found: ID does not exist" containerID="5533cf1b92cd792dfcb6a1953f72d0abedaee4cb23f48905192a29a1642e75f3" Oct 30 16:52:54.953258 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:54.953208 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5533cf1b92cd792dfcb6a1953f72d0abedaee4cb23f48905192a29a1642e75f3"} err="failed to get container status \"5533cf1b92cd792dfcb6a1953f72d0abedaee4cb23f48905192a29a1642e75f3\": rpc error: code = NotFound desc = could not find container \"5533cf1b92cd792dfcb6a1953f72d0abedaee4cb23f48905192a29a1642e75f3\": container with ID starting with 5533cf1b92cd792dfcb6a1953f72d0abedaee4cb23f48905192a29a1642e75f3 not found: ID does not exist" Oct 30 16:52:55.049023 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:55.048984 5052 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/218cbb21-a450-4352-b4b3-f7e8558688bb-istio-podinfo\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:52:55.049023 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:55.049012 5052 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/218cbb21-a450-4352-b4b3-f7e8558688bb-istio-envoy\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:52:55.049023 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:55.049023 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nztq\" (UniqueName: \"kubernetes.io/projected/218cbb21-a450-4352-b4b3-f7e8558688bb-kube-api-access-2nztq\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:52:55.049023 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:55.049032 5052 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/218cbb21-a450-4352-b4b3-f7e8558688bb-istio-token\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:52:55.276434 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:55.276398 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-407eab-openshift-default-85f774ff9c-nldmk"] Oct 30 16:52:55.284819 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:55.284789 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-407eab-openshift-default-85f774ff9c-nldmk"] Oct 30 16:52:56.023023 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:56.022986 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-ff1189/test-model-ff1189-kserve-84bc7d6c57-5f475"] Oct 30 16:52:56.024189 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:56.024168 5052 cpu_manager.go:414] "containerMap: removing container" podUID="218cbb21-a450-4352-b4b3-f7e8558688bb" containerName="istio-proxy" Oct 30 16:52:56.024268 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:56.024193 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="218cbb21-a450-4352-b4b3-f7e8558688bb" containerName="istio-proxy" Oct 30 16:52:56.024509 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:56.024485 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="218cbb21-a450-4352-b4b3-f7e8558688bb" containerName="istio-proxy" Oct 30 16:52:56.024722 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:56.024700 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="218cbb21-a450-4352-b4b3-f7e8558688bb" containerName="istio-proxy" Oct 30 16:52:56.030724 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:56.030704 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ff1189/test-model-ff1189-kserve-84bc7d6c57-5f475" Oct 30 16:52:56.055353 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:56.055327 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-ff1189/test-model-ff1189-kserve-84bc7d6c57-5f475"] Oct 30 16:52:56.066642 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:56.066605 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-ff1189\"/\"kube-root-ca.crt\"" Oct 30 16:52:56.067003 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:56.066982 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-ff1189\"/\"default-dockercfg-gb858\"" Oct 30 16:52:56.069454 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:56.067318 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-ff1189\"/\"test-model-ff1189-kserve-self-signed-certs\"" Oct 30 16:52:56.069454 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:56.068441 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4e168d8-c482-4fdf-9d3c-aa41fd252c90-home\") pod \"test-model-ff1189-kserve-84bc7d6c57-5f475\" (UID: \"a4e168d8-c482-4fdf-9d3c-aa41fd252c90\") " pod="ai-tenant-e2e-inf-a-ff1189/test-model-ff1189-kserve-84bc7d6c57-5f475" Oct 30 16:52:56.069454 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:56.068533 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbnc6\" (UniqueName: \"kubernetes.io/projected/a4e168d8-c482-4fdf-9d3c-aa41fd252c90-kube-api-access-hbnc6\") pod \"test-model-ff1189-kserve-84bc7d6c57-5f475\" (UID: \"a4e168d8-c482-4fdf-9d3c-aa41fd252c90\") " pod="ai-tenant-e2e-inf-a-ff1189/test-model-ff1189-kserve-84bc7d6c57-5f475" Oct 30 16:52:56.069454 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:56.068630 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4e168d8-c482-4fdf-9d3c-aa41fd252c90-tls-certs\") pod \"test-model-ff1189-kserve-84bc7d6c57-5f475\" (UID: \"a4e168d8-c482-4fdf-9d3c-aa41fd252c90\") " pod="ai-tenant-e2e-inf-a-ff1189/test-model-ff1189-kserve-84bc7d6c57-5f475" Oct 30 16:52:56.069454 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:56.068690 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4e168d8-c482-4fdf-9d3c-aa41fd252c90-model-cache\") pod \"test-model-ff1189-kserve-84bc7d6c57-5f475\" (UID: \"a4e168d8-c482-4fdf-9d3c-aa41fd252c90\") " pod="ai-tenant-e2e-inf-a-ff1189/test-model-ff1189-kserve-84bc7d6c57-5f475" Oct 30 16:52:56.069454 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:56.068770 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4e168d8-c482-4fdf-9d3c-aa41fd252c90-tmp-dir\") pod \"test-model-ff1189-kserve-84bc7d6c57-5f475\" (UID: \"a4e168d8-c482-4fdf-9d3c-aa41fd252c90\") " pod="ai-tenant-e2e-inf-a-ff1189/test-model-ff1189-kserve-84bc7d6c57-5f475" Oct 30 16:52:56.069454 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:56.068883 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4e168d8-c482-4fdf-9d3c-aa41fd252c90-dshm\") pod \"test-model-ff1189-kserve-84bc7d6c57-5f475\" (UID: \"a4e168d8-c482-4fdf-9d3c-aa41fd252c90\") " pod="ai-tenant-e2e-inf-a-ff1189/test-model-ff1189-kserve-84bc7d6c57-5f475" Oct 30 16:52:56.080128 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:56.080109 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-ff1189\"/\"openshift-service-ca.crt\"" Oct 30 16:52:56.170373 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:56.170344 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbnc6\" (UniqueName: \"kubernetes.io/projected/a4e168d8-c482-4fdf-9d3c-aa41fd252c90-kube-api-access-hbnc6\") pod \"test-model-ff1189-kserve-84bc7d6c57-5f475\" (UID: \"a4e168d8-c482-4fdf-9d3c-aa41fd252c90\") " pod="ai-tenant-e2e-inf-a-ff1189/test-model-ff1189-kserve-84bc7d6c57-5f475" Oct 30 16:52:56.170555 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:56.170409 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4e168d8-c482-4fdf-9d3c-aa41fd252c90-tls-certs\") pod \"test-model-ff1189-kserve-84bc7d6c57-5f475\" (UID: \"a4e168d8-c482-4fdf-9d3c-aa41fd252c90\") " pod="ai-tenant-e2e-inf-a-ff1189/test-model-ff1189-kserve-84bc7d6c57-5f475" Oct 30 16:52:56.170555 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:56.170473 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4e168d8-c482-4fdf-9d3c-aa41fd252c90-model-cache\") pod \"test-model-ff1189-kserve-84bc7d6c57-5f475\" (UID: \"a4e168d8-c482-4fdf-9d3c-aa41fd252c90\") " pod="ai-tenant-e2e-inf-a-ff1189/test-model-ff1189-kserve-84bc7d6c57-5f475" Oct 30 16:52:56.170711 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:56.170572 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4e168d8-c482-4fdf-9d3c-aa41fd252c90-tmp-dir\") pod \"test-model-ff1189-kserve-84bc7d6c57-5f475\" (UID: \"a4e168d8-c482-4fdf-9d3c-aa41fd252c90\") " pod="ai-tenant-e2e-inf-a-ff1189/test-model-ff1189-kserve-84bc7d6c57-5f475" Oct 30 16:52:56.170749 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:56.170723 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4e168d8-c482-4fdf-9d3c-aa41fd252c90-dshm\") pod \"test-model-ff1189-kserve-84bc7d6c57-5f475\" (UID: \"a4e168d8-c482-4fdf-9d3c-aa41fd252c90\") " pod="ai-tenant-e2e-inf-a-ff1189/test-model-ff1189-kserve-84bc7d6c57-5f475" Oct 30 16:52:56.170863 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:56.170848 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4e168d8-c482-4fdf-9d3c-aa41fd252c90-home\") pod \"test-model-ff1189-kserve-84bc7d6c57-5f475\" (UID: \"a4e168d8-c482-4fdf-9d3c-aa41fd252c90\") " pod="ai-tenant-e2e-inf-a-ff1189/test-model-ff1189-kserve-84bc7d6c57-5f475" Oct 30 16:52:56.170969 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:56.170945 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4e168d8-c482-4fdf-9d3c-aa41fd252c90-tmp-dir\") pod \"test-model-ff1189-kserve-84bc7d6c57-5f475\" (UID: \"a4e168d8-c482-4fdf-9d3c-aa41fd252c90\") " pod="ai-tenant-e2e-inf-a-ff1189/test-model-ff1189-kserve-84bc7d6c57-5f475" Oct 30 16:52:56.171034 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:56.170997 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4e168d8-c482-4fdf-9d3c-aa41fd252c90-model-cache\") pod \"test-model-ff1189-kserve-84bc7d6c57-5f475\" (UID: \"a4e168d8-c482-4fdf-9d3c-aa41fd252c90\") " pod="ai-tenant-e2e-inf-a-ff1189/test-model-ff1189-kserve-84bc7d6c57-5f475" Oct 30 16:52:56.171139 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:56.171123 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4e168d8-c482-4fdf-9d3c-aa41fd252c90-home\") pod \"test-model-ff1189-kserve-84bc7d6c57-5f475\" (UID: \"a4e168d8-c482-4fdf-9d3c-aa41fd252c90\") " pod="ai-tenant-e2e-inf-a-ff1189/test-model-ff1189-kserve-84bc7d6c57-5f475" Oct 30 16:52:56.172874 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:56.172851 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4e168d8-c482-4fdf-9d3c-aa41fd252c90-dshm\") pod \"test-model-ff1189-kserve-84bc7d6c57-5f475\" (UID: \"a4e168d8-c482-4fdf-9d3c-aa41fd252c90\") " pod="ai-tenant-e2e-inf-a-ff1189/test-model-ff1189-kserve-84bc7d6c57-5f475" Oct 30 16:52:56.173178 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:56.173162 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4e168d8-c482-4fdf-9d3c-aa41fd252c90-tls-certs\") pod \"test-model-ff1189-kserve-84bc7d6c57-5f475\" (UID: \"a4e168d8-c482-4fdf-9d3c-aa41fd252c90\") " pod="ai-tenant-e2e-inf-a-ff1189/test-model-ff1189-kserve-84bc7d6c57-5f475" Oct 30 16:52:56.198431 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:56.198408 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbnc6\" (UniqueName: \"kubernetes.io/projected/a4e168d8-c482-4fdf-9d3c-aa41fd252c90-kube-api-access-hbnc6\") pod \"test-model-ff1189-kserve-84bc7d6c57-5f475\" (UID: \"a4e168d8-c482-4fdf-9d3c-aa41fd252c90\") " pod="ai-tenant-e2e-inf-a-ff1189/test-model-ff1189-kserve-84bc7d6c57-5f475" Oct 30 16:52:56.349625 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:56.349520 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ff1189/test-model-ff1189-kserve-84bc7d6c57-5f475" Oct 30 16:52:56.604660 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:56.603693 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="218cbb21-a450-4352-b4b3-f7e8558688bb" path="/var/lib/kubelet/pods/218cbb21-a450-4352-b4b3-f7e8558688bb/volumes" Oct 30 16:52:56.742096 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:56.742066 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-ff1189/test-model-ff1189-kserve-84bc7d6c57-5f475"] Oct 30 16:52:56.742096 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:52:56.742080 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda4e168d8_c482_4fdf_9d3c_aa41fd252c90.slice/crio-2811e531d15763581a8f3e6ea61540fddd5640082e8bc34fa814c4b45d07dfcf WatchSource:0}: Error finding container 2811e531d15763581a8f3e6ea61540fddd5640082e8bc34fa814c4b45d07dfcf: Status 404 returned error can't find the container with id 2811e531d15763581a8f3e6ea61540fddd5640082e8bc34fa814c4b45d07dfcf Oct 30 16:52:56.943621 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:56.943539 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ff1189/test-model-ff1189-kserve-84bc7d6c57-5f475" event={"ID":"a4e168d8-c482-4fdf-9d3c-aa41fd252c90","Type":"ContainerStarted","Data":"139cd2aa8b19eba534fd6f80e60264580164404115226f981d0b2ecf3b351b8b"} Oct 30 16:52:56.943829 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:56.943636 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ff1189/test-model-ff1189-kserve-84bc7d6c57-5f475" event={"ID":"a4e168d8-c482-4fdf-9d3c-aa41fd252c90","Type":"ContainerStarted","Data":"2811e531d15763581a8f3e6ea61540fddd5640082e8bc34fa814c4b45d07dfcf"} Oct 30 16:52:56.997211 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:56.997150 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-ff1189/test-model-ff1189-kserve-84bc7d6c57-5f475" podStartSLOduration=1.9971361399999998 podStartE2EDuration="1.99713614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:52:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:52:56.994636149 +0000 UTC m=+1932.521971369" watchObservedRunningTime="2026-10-30 16:52:56.99713614 +0000 UTC m=+1932.524471359" Oct 30 16:52:57.350253 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:57.350169 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-ff1189/test-model-ff1189-kserve-84bc7d6c57-5f475" Oct 30 16:52:58.117451 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:58.117410 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ea77d8-6d444d798d-29glb"] Oct 30 16:52:58.118365 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:58.118336 5052 cpu_manager.go:414] "containerMap: removing container" podUID="218cbb21-a450-4352-b4b3-f7e8558688bb" containerName="istio-proxy" Oct 30 16:52:58.118365 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:58.118361 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="218cbb21-a450-4352-b4b3-f7e8558688bb" containerName="istio-proxy" Oct 30 16:52:58.122163 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:58.122144 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ea77d8-6d444d798d-29glb" Oct 30 16:52:58.142972 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:58.142946 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ea77d8-6d444d798d-29glb"] Oct 30 16:52:58.149982 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:58.149959 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-ea77d8\"" Oct 30 16:52:58.177833 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:58.177806 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ea77d8-584f58fcf-k88cs"] Oct 30 16:52:58.182788 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:58.182768 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ea77d8-584f58fcf-k88cs" Oct 30 16:52:58.200445 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:58.200419 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-ea77d8-dockercfg-vxl7q\"" Oct 30 16:52:58.203521 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:58.202797 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ea77d8-584f58fcf-k88cs"] Oct 30 16:52:58.268604 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:58.266740 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zms84\" (UniqueName: \"kubernetes.io/projected/2b967416-660d-4e1e-be9a-963007619616-kube-api-access-zms84\") pod \"payload-pre-processing-e2e-inf-a-ea77d8-584f58fcf-k88cs\" (UID: \"2b967416-660d-4e1e-be9a-963007619616\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ea77d8-584f58fcf-k88cs" Oct 30 16:52:58.268604 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:58.266953 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b967416-660d-4e1e-be9a-963007619616-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-ea77d8-584f58fcf-k88cs\" (UID: \"2b967416-660d-4e1e-be9a-963007619616\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ea77d8-584f58fcf-k88cs" Oct 30 16:52:58.268604 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:58.267008 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2715b96-2ae8-4862-a787-5df13822387c-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-ea77d8-6d444d798d-29glb\" (UID: \"a2715b96-2ae8-4862-a787-5df13822387c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ea77d8-6d444d798d-29glb" Oct 30 16:52:58.268604 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:58.267139 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2tg2\" (UniqueName: \"kubernetes.io/projected/a2715b96-2ae8-4862-a787-5df13822387c-kube-api-access-t2tg2\") pod \"payload-processing-e2e-inf-a-ea77d8-6d444d798d-29glb\" (UID: \"a2715b96-2ae8-4862-a787-5df13822387c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ea77d8-6d444d798d-29glb" Oct 30 16:52:58.459634 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:58.457340 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2715b96-2ae8-4862-a787-5df13822387c-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-ea77d8-6d444d798d-29glb\" (UID: \"a2715b96-2ae8-4862-a787-5df13822387c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ea77d8-6d444d798d-29glb" Oct 30 16:52:58.459634 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:58.457489 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2tg2\" (UniqueName: \"kubernetes.io/projected/a2715b96-2ae8-4862-a787-5df13822387c-kube-api-access-t2tg2\") pod \"payload-processing-e2e-inf-a-ea77d8-6d444d798d-29glb\" (UID: \"a2715b96-2ae8-4862-a787-5df13822387c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ea77d8-6d444d798d-29glb" Oct 30 16:52:58.459634 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:58.457704 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zms84\" (UniqueName: \"kubernetes.io/projected/2b967416-660d-4e1e-be9a-963007619616-kube-api-access-zms84\") pod \"payload-pre-processing-e2e-inf-a-ea77d8-584f58fcf-k88cs\" (UID: \"2b967416-660d-4e1e-be9a-963007619616\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ea77d8-584f58fcf-k88cs" Oct 30 16:52:58.459634 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:58.457861 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b967416-660d-4e1e-be9a-963007619616-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-ea77d8-584f58fcf-k88cs\" (UID: \"2b967416-660d-4e1e-be9a-963007619616\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ea77d8-584f58fcf-k88cs" Oct 30 16:52:58.520602 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:58.497554 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2715b96-2ae8-4862-a787-5df13822387c-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-ea77d8-6d444d798d-29glb\" (UID: \"a2715b96-2ae8-4862-a787-5df13822387c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ea77d8-6d444d798d-29glb" Oct 30 16:52:58.529096 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:58.529051 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b967416-660d-4e1e-be9a-963007619616-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-ea77d8-584f58fcf-k88cs\" (UID: \"2b967416-660d-4e1e-be9a-963007619616\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ea77d8-584f58fcf-k88cs" Oct 30 16:52:58.542497 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:58.542469 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zms84\" (UniqueName: \"kubernetes.io/projected/2b967416-660d-4e1e-be9a-963007619616-kube-api-access-zms84\") pod \"payload-pre-processing-e2e-inf-a-ea77d8-584f58fcf-k88cs\" (UID: \"2b967416-660d-4e1e-be9a-963007619616\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ea77d8-584f58fcf-k88cs" Oct 30 16:52:58.546595 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:58.546556 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2tg2\" (UniqueName: \"kubernetes.io/projected/a2715b96-2ae8-4862-a787-5df13822387c-kube-api-access-t2tg2\") pod \"payload-processing-e2e-inf-a-ea77d8-6d444d798d-29glb\" (UID: \"a2715b96-2ae8-4862-a787-5df13822387c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ea77d8-6d444d798d-29glb" Oct 30 16:52:58.738205 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:58.738116 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ea77d8-6d444d798d-29glb" Oct 30 16:52:58.809716 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:58.808568 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ea77d8-584f58fcf-k88cs" Oct 30 16:52:59.055387 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:59.055359 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ea77d8-6d444d798d-29glb"] Oct 30 16:52:59.055976 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:52:59.055946 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2715b96_2ae8_4862_a787_5df13822387c.slice/crio-4acfeb0297631c1e95a0d0389cd96430ee26f37c4189ae4cf05037d4f08d0c58 WatchSource:0}: Error finding container 4acfeb0297631c1e95a0d0389cd96430ee26f37c4189ae4cf05037d4f08d0c58: Status 404 returned error can't find the container with id 4acfeb0297631c1e95a0d0389cd96430ee26f37c4189ae4cf05037d4f08d0c58 Oct 30 16:52:59.178726 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:52:59.178693 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b967416_660d_4e1e_be9a_963007619616.slice/crio-37cbf0655ebf7518ddd061ca7873257f0d00d995b60fe275304e633ee30bd8dc WatchSource:0}: Error finding container 37cbf0655ebf7518ddd061ca7873257f0d00d995b60fe275304e633ee30bd8dc: Status 404 returned error can't find the container with id 37cbf0655ebf7518ddd061ca7873257f0d00d995b60fe275304e633ee30bd8dc Oct 30 16:52:59.178861 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:59.178785 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ea77d8-584f58fcf-k88cs"] Oct 30 16:52:59.972901 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:59.972857 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ea77d8-6d444d798d-29glb" event={"ID":"a2715b96-2ae8-4862-a787-5df13822387c","Type":"ContainerStarted","Data":"fe01afa52c0e2d1956ceaa330da7faf621e1036310ddfce36615623ad17e840a"} Oct 30 16:52:59.973319 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:59.972931 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ea77d8-6d444d798d-29glb" event={"ID":"a2715b96-2ae8-4862-a787-5df13822387c","Type":"ContainerStarted","Data":"4acfeb0297631c1e95a0d0389cd96430ee26f37c4189ae4cf05037d4f08d0c58"} Oct 30 16:52:59.973319 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:59.973077 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-ea77d8-6d444d798d-29glb" Oct 30 16:52:59.974755 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:59.974731 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ea77d8-584f58fcf-k88cs" event={"ID":"2b967416-660d-4e1e-be9a-963007619616","Type":"ContainerStarted","Data":"a638ce3ad204fcc6ef0208a71e2f98b04b7a9ff9de546fa1d7867f0297fa296e"} Oct 30 16:52:59.974864 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:59.974772 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ea77d8-584f58fcf-k88cs" event={"ID":"2b967416-660d-4e1e-be9a-963007619616","Type":"ContainerStarted","Data":"37cbf0655ebf7518ddd061ca7873257f0d00d995b60fe275304e633ee30bd8dc"} Oct 30 16:52:59.974922 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:52:59.974873 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ea77d8-584f58fcf-k88cs" Oct 30 16:53:00.019248 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:53:00.019198 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-ea77d8-6d444d798d-29glb" podStartSLOduration=2.019183684 podStartE2EDuration="2.019183684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:52:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:53:00.015206472 +0000 UTC m=+1935.542541691" watchObservedRunningTime="2026-10-30 16:53:00.019183684 +0000 UTC m=+1935.546518927" Oct 30 16:53:00.059092 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:53:00.059035 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ea77d8-584f58fcf-k88cs" podStartSLOduration=2.05901927 podStartE2EDuration="2.05901927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:52:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:53:00.057501965 +0000 UTC m=+1935.584837178" watchObservedRunningTime="2026-10-30 16:53:00.05901927 +0000 UTC m=+1935.586354488" Oct 30 16:53:05.525201 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:53:05.525160 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cdwcj"] Oct 30 16:53:06.350910 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:53:06.350873 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-ff1189/test-model-ff1189-kserve-84bc7d6c57-5f475" Oct 30 16:53:06.363463 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:53:06.363434 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-ff1189/test-model-ff1189-kserve-84bc7d6c57-5f475" Oct 30 16:53:07.340922 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:53:07.340890 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-ff1189/test-model-ff1189-kserve-84bc7d6c57-5f475" Oct 30 16:53:10.984565 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:53:10.984532 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ea77d8-584f58fcf-k88cs" Oct 30 16:53:10.985101 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:53:10.984662 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-ea77d8-6d444d798d-29glb" Oct 30 16:53:17.252740 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:53:17.250169 5052 patch_prober.go:29] interesting pod/e2e-inf-a-a59502-openshift-default-74889b7c5f-9m8g5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": read tcp 10.132.0.2:53336->10.132.0.148:15021: read: connection reset by peer" start-of-body= Oct 30 16:53:17.252740 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:53:17.250235 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-a59502-openshift-default-74889b7c5f-9m8g5" podUID="d7f77c25-893f-40a2-a406-1891790e4cb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": read tcp 10.132.0.2:53336->10.132.0.148:15021: read: connection reset by peer" Oct 30 16:53:18.166271 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:53:18.166236 5052 generic.go:356] "Generic (PLEG): container finished" podID="d7f77c25-893f-40a2-a406-1891790e4cb5" containerID="bede44d865571edbe12a4667fdc6a7856a8ffd6bebc876fb78a0e14ca5361bbe" exitCode=0 Oct 30 16:53:18.166450 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:53:18.166308 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a59502-openshift-default-74889b7c5f-9m8g5" event={"ID":"d7f77c25-893f-40a2-a406-1891790e4cb5","Type":"ContainerDied","Data":"bede44d865571edbe12a4667fdc6a7856a8ffd6bebc876fb78a0e14ca5361bbe"} Oct 30 16:53:18.166842 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:53:18.166821 5052 scope.go:122] "RemoveContainer" containerID="bede44d865571edbe12a4667fdc6a7856a8ffd6bebc876fb78a0e14ca5361bbe" Oct 30 16:53:18.168715 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:53:18.168685 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:53:18.168833 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:53:18.168772 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:53:18.168833 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:53:18.168824 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:53:18.843425 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:53:18.843390 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 16:53:19.175866 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:53:19.175829 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a59502-openshift-default-74889b7c5f-9m8g5" event={"ID":"d7f77c25-893f-40a2-a406-1891790e4cb5","Type":"ContainerStarted","Data":"360a67d2a2b0835a961c6a60c20a1225eae3dddaf4be89423b1fa3345dad7cfa"} Oct 30 16:53:20.086063 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:53:20.086023 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-a59502-openshift-default-74889b7c5f-9m8g5" Oct 30 16:53:20.086490 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:53:20.086216 5052 patch_prober.go:29] interesting pod/e2e-inf-a-a59502-openshift-default-74889b7c5f-9m8g5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Oct 30 16:53:20.086490 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:53:20.086263 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a59502-openshift-default-74889b7c5f-9m8g5" podUID="d7f77c25-893f-40a2-a406-1891790e4cb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Oct 30 16:53:21.086100 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:53:21.086059 5052 patch_prober.go:29] interesting pod/e2e-inf-a-a59502-openshift-default-74889b7c5f-9m8g5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Oct 30 16:53:21.086638 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:53:21.086123 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a59502-openshift-default-74889b7c5f-9m8g5" podUID="d7f77c25-893f-40a2-a406-1891790e4cb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Oct 30 16:53:22.086884 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:53:22.086847 5052 patch_prober.go:29] interesting pod/e2e-inf-a-a59502-openshift-default-74889b7c5f-9m8g5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Oct 30 16:53:22.087328 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:53:22.086900 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a59502-openshift-default-74889b7c5f-9m8g5" podUID="d7f77c25-893f-40a2-a406-1891790e4cb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Oct 30 16:53:23.086787 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:53:23.086752 5052 patch_prober.go:29] interesting pod/e2e-inf-a-a59502-openshift-default-74889b7c5f-9m8g5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Oct 30 16:53:23.086982 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:53:23.086806 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a59502-openshift-default-74889b7c5f-9m8g5" podUID="d7f77c25-893f-40a2-a406-1891790e4cb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Oct 30 16:53:24.086068 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:53:24.086033 5052 patch_prober.go:29] interesting pod/e2e-inf-a-a59502-openshift-default-74889b7c5f-9m8g5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Oct 30 16:53:24.086270 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:53:24.086090 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a59502-openshift-default-74889b7c5f-9m8g5" podUID="d7f77c25-893f-40a2-a406-1891790e4cb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Oct 30 16:53:25.455281 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:53:25.455236 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-a59502-openshift-default-74889b7c5f-9m8g5" Oct 30 16:53:25.456236 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:53:25.456208 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a59502-openshift-default-74889b7c5f-9m8g5" Oct 30 16:53:25.457206 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:53:25.457184 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-a59502-openshift-default-74889b7c5f-9m8g5" Oct 30 16:53:38.424988 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:53:38.424948 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ea77d8-64c6656497-w87sb"] Oct 30 16:53:38.431145 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:53:38.431122 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ea77d8-64c6656497-w87sb" Oct 30 16:53:38.461849 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:53:38.461793 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-ea77d8-serving-cert\"" Oct 30 16:53:38.471318 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:53:38.471291 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ea77d8-64c6656497-w87sb"] Oct 30 16:53:38.576034 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:53:38.575993 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0867eae6-ab5d-495b-8822-5e64be91d7f1-maas-api-tls\") pod \"maas-api-e2e-inf-a-ea77d8-64c6656497-w87sb\" (UID: \"0867eae6-ab5d-495b-8822-5e64be91d7f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ea77d8-64c6656497-w87sb" Oct 30 16:53:38.576254 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:53:38.576226 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vx54p\" (UniqueName: \"kubernetes.io/projected/0867eae6-ab5d-495b-8822-5e64be91d7f1-kube-api-access-vx54p\") pod \"maas-api-e2e-inf-a-ea77d8-64c6656497-w87sb\" (UID: \"0867eae6-ab5d-495b-8822-5e64be91d7f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ea77d8-64c6656497-w87sb" Oct 30 16:53:38.576345 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:53:38.576273 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0867eae6-ab5d-495b-8822-5e64be91d7f1-metrics-certs\") pod \"maas-api-e2e-inf-a-ea77d8-64c6656497-w87sb\" (UID: \"0867eae6-ab5d-495b-8822-5e64be91d7f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ea77d8-64c6656497-w87sb" Oct 30 16:53:38.576535 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:53:38.576514 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0867eae6-ab5d-495b-8822-5e64be91d7f1-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ea77d8-64c6656497-w87sb\" (UID: \"0867eae6-ab5d-495b-8822-5e64be91d7f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ea77d8-64c6656497-w87sb" Oct 30 16:53:38.677681 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:53:38.677550 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vx54p\" (UniqueName: \"kubernetes.io/projected/0867eae6-ab5d-495b-8822-5e64be91d7f1-kube-api-access-vx54p\") pod \"maas-api-e2e-inf-a-ea77d8-64c6656497-w87sb\" (UID: \"0867eae6-ab5d-495b-8822-5e64be91d7f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ea77d8-64c6656497-w87sb" Oct 30 16:53:38.677857 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:53:38.677804 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0867eae6-ab5d-495b-8822-5e64be91d7f1-metrics-certs\") pod \"maas-api-e2e-inf-a-ea77d8-64c6656497-w87sb\" (UID: \"0867eae6-ab5d-495b-8822-5e64be91d7f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ea77d8-64c6656497-w87sb" Oct 30 16:53:38.678287 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:53:38.678259 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0867eae6-ab5d-495b-8822-5e64be91d7f1-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ea77d8-64c6656497-w87sb\" (UID: \"0867eae6-ab5d-495b-8822-5e64be91d7f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ea77d8-64c6656497-w87sb" Oct 30 16:53:38.678559 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:53:38.678538 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0867eae6-ab5d-495b-8822-5e64be91d7f1-maas-api-tls\") pod \"maas-api-e2e-inf-a-ea77d8-64c6656497-w87sb\" (UID: \"0867eae6-ab5d-495b-8822-5e64be91d7f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ea77d8-64c6656497-w87sb" Oct 30 16:53:38.679128 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:53:38.679102 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0867eae6-ab5d-495b-8822-5e64be91d7f1-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ea77d8-64c6656497-w87sb\" (UID: \"0867eae6-ab5d-495b-8822-5e64be91d7f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ea77d8-64c6656497-w87sb" Oct 30 16:53:38.681432 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:53:38.681412 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0867eae6-ab5d-495b-8822-5e64be91d7f1-maas-api-tls\") pod \"maas-api-e2e-inf-a-ea77d8-64c6656497-w87sb\" (UID: \"0867eae6-ab5d-495b-8822-5e64be91d7f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ea77d8-64c6656497-w87sb" Oct 30 16:53:38.681549 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:53:38.681527 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0867eae6-ab5d-495b-8822-5e64be91d7f1-metrics-certs\") pod \"maas-api-e2e-inf-a-ea77d8-64c6656497-w87sb\" (UID: \"0867eae6-ab5d-495b-8822-5e64be91d7f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ea77d8-64c6656497-w87sb" Oct 30 16:53:38.702774 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:53:38.702746 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vx54p\" (UniqueName: \"kubernetes.io/projected/0867eae6-ab5d-495b-8822-5e64be91d7f1-kube-api-access-vx54p\") pod \"maas-api-e2e-inf-a-ea77d8-64c6656497-w87sb\" (UID: \"0867eae6-ab5d-495b-8822-5e64be91d7f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ea77d8-64c6656497-w87sb" Oct 30 16:53:38.766604 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:53:38.766270 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ea77d8-64c6656497-w87sb" Oct 30 16:53:39.177486 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:53:39.177455 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ea77d8-64c6656497-w87sb"] Oct 30 16:53:39.178546 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:53:39.178519 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0867eae6_ab5d_495b_8822_5e64be91d7f1.slice/crio-bbba5d510ec24e0a3dd7f5bd5b9902fb48372f83e075a07d0451192346f5c73f WatchSource:0}: Error finding container bbba5d510ec24e0a3dd7f5bd5b9902fb48372f83e075a07d0451192346f5c73f: Status 404 returned error can't find the container with id bbba5d510ec24e0a3dd7f5bd5b9902fb48372f83e075a07d0451192346f5c73f Oct 30 16:53:39.444952 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:53:39.444911 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ea77d8-64c6656497-w87sb" event={"ID":"0867eae6-ab5d-495b-8822-5e64be91d7f1","Type":"ContainerStarted","Data":"bbba5d510ec24e0a3dd7f5bd5b9902fb48372f83e075a07d0451192346f5c73f"} Oct 30 16:53:40.456414 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:53:40.456286 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ea77d8-64c6656497-w87sb" event={"ID":"0867eae6-ab5d-495b-8822-5e64be91d7f1","Type":"ContainerStarted","Data":"3b377054dfa0fc597c707de84836e193a2ab2f5eab1ae2b77c6bc2de89ce11cd"} Oct 30 16:53:40.456414 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:53:40.456358 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ea77d8-64c6656497-w87sb" Oct 30 16:53:40.504615 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:53:40.504529 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ea77d8-64c6656497-w87sb" podStartSLOduration=2.146722604 podStartE2EDuration="2.504515288s" totalImagesPullingTime="357.792684ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:53:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:53:40.502451991 +0000 UTC m=+1976.029787210" watchObservedRunningTime="2026-10-30 16:53:40.504515288 +0000 UTC m=+1976.031850506" Oct 30 16:53:42.707398 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:53:42.707361 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9c1e26-5f76c9f698-gvxqs"] Oct 30 16:53:42.725910 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:53:42.725879 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9c1e26-5f76c9f698-gvxqs"] Oct 30 16:53:42.730926 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:53:42.730806 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-9c1e26-5f76c9f698-gvxqs" Oct 30 16:53:42.778542 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:53:42.778518 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-9c1e26\"" Oct 30 16:53:42.778725 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:53:42.778624 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-9c1e26-dockercfg-b7b4m\"" Oct 30 16:53:42.784608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:53:42.784394 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sl5s\" (UniqueName: \"kubernetes.io/projected/9e7b9aff-3301-4350-b511-262ff5088c3b-kube-api-access-4sl5s\") pod \"payload-processing-e2e-inf-b-9c1e26-5f76c9f698-gvxqs\" (UID: \"9e7b9aff-3301-4350-b511-262ff5088c3b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9c1e26-5f76c9f698-gvxqs" Oct 30 16:53:42.785647 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:53:42.785574 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e7b9aff-3301-4350-b511-262ff5088c3b-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-9c1e26-5f76c9f698-gvxqs\" (UID: \"9e7b9aff-3301-4350-b511-262ff5088c3b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9c1e26-5f76c9f698-gvxqs" Oct 30 16:53:42.888116 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:53:42.888078 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e7b9aff-3301-4350-b511-262ff5088c3b-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-9c1e26-5f76c9f698-gvxqs\" (UID: \"9e7b9aff-3301-4350-b511-262ff5088c3b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9c1e26-5f76c9f698-gvxqs" Oct 30 16:53:42.888378 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:53:42.888355 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sl5s\" (UniqueName: \"kubernetes.io/projected/9e7b9aff-3301-4350-b511-262ff5088c3b-kube-api-access-4sl5s\") pod \"payload-processing-e2e-inf-b-9c1e26-5f76c9f698-gvxqs\" (UID: \"9e7b9aff-3301-4350-b511-262ff5088c3b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9c1e26-5f76c9f698-gvxqs" Oct 30 16:53:42.888934 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:53:42.888906 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e7b9aff-3301-4350-b511-262ff5088c3b-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-9c1e26-5f76c9f698-gvxqs\" (UID: \"9e7b9aff-3301-4350-b511-262ff5088c3b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9c1e26-5f76c9f698-gvxqs" Oct 30 16:53:42.913345 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:53:42.913318 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sl5s\" (UniqueName: \"kubernetes.io/projected/9e7b9aff-3301-4350-b511-262ff5088c3b-kube-api-access-4sl5s\") pod \"payload-processing-e2e-inf-b-9c1e26-5f76c9f698-gvxqs\" (UID: \"9e7b9aff-3301-4350-b511-262ff5088c3b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9c1e26-5f76c9f698-gvxqs" Oct 30 16:53:43.103407 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:53:43.103323 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-9c1e26-5f76c9f698-gvxqs" Oct 30 16:53:43.413369 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:53:43.413340 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e7b9aff_3301_4350_b511_262ff5088c3b.slice/crio-7eef620115790e7f43c12a59f2a63f0feed9fc3c66d9296257c8f6ca1ae4296f WatchSource:0}: Error finding container 7eef620115790e7f43c12a59f2a63f0feed9fc3c66d9296257c8f6ca1ae4296f: Status 404 returned error can't find the container with id 7eef620115790e7f43c12a59f2a63f0feed9fc3c66d9296257c8f6ca1ae4296f Oct 30 16:53:43.413986 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:53:43.413942 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9c1e26-5f76c9f698-gvxqs"] Oct 30 16:53:43.655437 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:53:43.655404 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-9c1e26-5f76c9f698-gvxqs" event={"ID":"9e7b9aff-3301-4350-b511-262ff5088c3b","Type":"ContainerStarted","Data":"3077a6727c59171224522b4dd4c2b03c017d1dd8ce9611468dad0b30654fb136"} Oct 30 16:53:43.655660 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:53:43.655461 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-9c1e26-5f76c9f698-gvxqs" Oct 30 16:53:43.655660 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:53:43.655479 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-9c1e26-5f76c9f698-gvxqs" event={"ID":"9e7b9aff-3301-4350-b511-262ff5088c3b","Type":"ContainerStarted","Data":"7eef620115790e7f43c12a59f2a63f0feed9fc3c66d9296257c8f6ca1ae4296f"} Oct 30 16:53:43.701615 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:53:43.701499 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-9c1e26-5f76c9f698-gvxqs" podStartSLOduration=1.70148283 podStartE2EDuration="1.70148283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:53:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:53:43.698439845 +0000 UTC m=+1979.225775083" watchObservedRunningTime="2026-10-30 16:53:43.70148283 +0000 UTC m=+1979.228818050" Oct 30 16:53:46.085366 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:53:46.085336 5052 scope.go:122] "RemoveContainer" containerID="75373b1db1fd33a998a1ffb62471e48229104d78c1378ea07fc83a518f4b9447" Oct 30 16:53:46.469363 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:53:46.469331 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ea77d8-64c6656497-w87sb" Oct 30 16:53:48.842655 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:53:48.842625 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 16:53:49.660502 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:53:49.660464 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cdwcj"] Oct 30 16:53:54.666539 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:53:54.666507 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-9c1e26-5f76c9f698-gvxqs" Oct 30 16:53:55.472604 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:53:55.471655 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-689899-84d4f48f6d-48std"] Oct 30 16:53:55.483288 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:53:55.483260 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-689899-84d4f48f6d-48std"] Oct 30 16:53:55.483669 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:53:55.483647 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-689899-84d4f48f6d-48std" Oct 30 16:53:55.511081 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:53:55.511053 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-689899\"" Oct 30 16:53:55.511609 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:53:55.511574 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-689899-7c7b96964c-ltxh4"] Oct 30 16:53:55.516176 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:53:55.516160 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-689899-7c7b96964c-ltxh4" Oct 30 16:53:55.556604 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:53:55.543969 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-689899-dockercfg-dfkjx\"" Oct 30 16:53:55.643605 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:53:55.643077 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50a076fb-ca0f-4c64-8c69-38b4ab22b088-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-689899-7c7b96964c-ltxh4\" (UID: \"50a076fb-ca0f-4c64-8c69-38b4ab22b088\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-689899-7c7b96964c-ltxh4" Oct 30 16:53:55.643605 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:53:55.643157 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bv9qz\" (UniqueName: \"kubernetes.io/projected/7301ced5-39dd-4e54-bff6-5262a04a0ebf-kube-api-access-bv9qz\") pod \"payload-processing-e2e-inf-b-689899-84d4f48f6d-48std\" (UID: \"7301ced5-39dd-4e54-bff6-5262a04a0ebf\") " pod="openshift-ingress/payload-processing-e2e-inf-b-689899-84d4f48f6d-48std" Oct 30 16:53:55.643605 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:53:55.643269 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7k49d\" (UniqueName: \"kubernetes.io/projected/50a076fb-ca0f-4c64-8c69-38b4ab22b088-kube-api-access-7k49d\") pod \"payload-pre-processing-e2e-inf-b-689899-7c7b96964c-ltxh4\" (UID: \"50a076fb-ca0f-4c64-8c69-38b4ab22b088\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-689899-7c7b96964c-ltxh4" Oct 30 16:53:55.643605 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:53:55.643448 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7301ced5-39dd-4e54-bff6-5262a04a0ebf-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-689899-84d4f48f6d-48std\" (UID: \"7301ced5-39dd-4e54-bff6-5262a04a0ebf\") " pod="openshift-ingress/payload-processing-e2e-inf-b-689899-84d4f48f6d-48std" Oct 30 16:53:55.754663 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:53:55.753807 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50a076fb-ca0f-4c64-8c69-38b4ab22b088-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-689899-7c7b96964c-ltxh4\" (UID: \"50a076fb-ca0f-4c64-8c69-38b4ab22b088\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-689899-7c7b96964c-ltxh4" Oct 30 16:53:55.754663 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:53:55.753885 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bv9qz\" (UniqueName: \"kubernetes.io/projected/7301ced5-39dd-4e54-bff6-5262a04a0ebf-kube-api-access-bv9qz\") pod \"payload-processing-e2e-inf-b-689899-84d4f48f6d-48std\" (UID: \"7301ced5-39dd-4e54-bff6-5262a04a0ebf\") " pod="openshift-ingress/payload-processing-e2e-inf-b-689899-84d4f48f6d-48std" Oct 30 16:53:55.754663 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:53:55.753999 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7k49d\" (UniqueName: \"kubernetes.io/projected/50a076fb-ca0f-4c64-8c69-38b4ab22b088-kube-api-access-7k49d\") pod \"payload-pre-processing-e2e-inf-b-689899-7c7b96964c-ltxh4\" (UID: \"50a076fb-ca0f-4c64-8c69-38b4ab22b088\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-689899-7c7b96964c-ltxh4" Oct 30 16:53:55.754663 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:53:55.754191 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7301ced5-39dd-4e54-bff6-5262a04a0ebf-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-689899-84d4f48f6d-48std\" (UID: \"7301ced5-39dd-4e54-bff6-5262a04a0ebf\") " pod="openshift-ingress/payload-processing-e2e-inf-b-689899-84d4f48f6d-48std" Oct 30 16:53:55.940659 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:53:55.938797 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7301ced5-39dd-4e54-bff6-5262a04a0ebf-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-689899-84d4f48f6d-48std\" (UID: \"7301ced5-39dd-4e54-bff6-5262a04a0ebf\") " pod="openshift-ingress/payload-processing-e2e-inf-b-689899-84d4f48f6d-48std" Oct 30 16:53:55.982964 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:53:55.982901 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7k49d\" (UniqueName: \"kubernetes.io/projected/50a076fb-ca0f-4c64-8c69-38b4ab22b088-kube-api-access-7k49d\") pod \"payload-pre-processing-e2e-inf-b-689899-7c7b96964c-ltxh4\" (UID: \"50a076fb-ca0f-4c64-8c69-38b4ab22b088\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-689899-7c7b96964c-ltxh4" Oct 30 16:53:55.986663 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:53:55.986542 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bv9qz\" (UniqueName: \"kubernetes.io/projected/7301ced5-39dd-4e54-bff6-5262a04a0ebf-kube-api-access-bv9qz\") pod \"payload-processing-e2e-inf-b-689899-84d4f48f6d-48std\" (UID: \"7301ced5-39dd-4e54-bff6-5262a04a0ebf\") " pod="openshift-ingress/payload-processing-e2e-inf-b-689899-84d4f48f6d-48std" Oct 30 16:53:55.998088 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:53:55.998060 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50a076fb-ca0f-4c64-8c69-38b4ab22b088-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-689899-7c7b96964c-ltxh4\" (UID: \"50a076fb-ca0f-4c64-8c69-38b4ab22b088\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-689899-7c7b96964c-ltxh4" Oct 30 16:53:56.051658 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:53:56.044626 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-689899-7c7b96964c-ltxh4"] Oct 30 16:53:56.105699 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:53:56.105670 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-689899-84d4f48f6d-48std" Oct 30 16:53:56.258306 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:53:56.258014 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-689899-7c7b96964c-ltxh4" Oct 30 16:53:56.560704 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:53:56.560668 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7301ced5_39dd_4e54_bff6_5262a04a0ebf.slice/crio-4738ddd107b1aa04e293c47ca8ce9955336648ec3ed8dfc2ca85109bcee9bb2b WatchSource:0}: Error finding container 4738ddd107b1aa04e293c47ca8ce9955336648ec3ed8dfc2ca85109bcee9bb2b: Status 404 returned error can't find the container with id 4738ddd107b1aa04e293c47ca8ce9955336648ec3ed8dfc2ca85109bcee9bb2b Oct 30 16:53:56.560846 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:53:56.560699 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-689899-84d4f48f6d-48std"] Oct 30 16:53:56.657167 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:53:56.657133 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-689899-7c7b96964c-ltxh4"] Oct 30 16:53:56.658949 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:53:56.658922 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50a076fb_ca0f_4c64_8c69_38b4ab22b088.slice/crio-5d64ded69aaf942b97830458b8735faac244b19f71673d4bcbced1ccb32d0153 WatchSource:0}: Error finding container 5d64ded69aaf942b97830458b8735faac244b19f71673d4bcbced1ccb32d0153: Status 404 returned error can't find the container with id 5d64ded69aaf942b97830458b8735faac244b19f71673d4bcbced1ccb32d0153 Oct 30 16:53:57.190675 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:53:57.190399 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-689899-84d4f48f6d-48std" event={"ID":"7301ced5-39dd-4e54-bff6-5262a04a0ebf","Type":"ContainerStarted","Data":"933da0766f70f368f236f5e5eca0f5329edb418a88ce653b777215f1ade554db"} Oct 30 16:53:57.190675 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:53:57.190465 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-689899-84d4f48f6d-48std" event={"ID":"7301ced5-39dd-4e54-bff6-5262a04a0ebf","Type":"ContainerStarted","Data":"4738ddd107b1aa04e293c47ca8ce9955336648ec3ed8dfc2ca85109bcee9bb2b"} Oct 30 16:53:57.190675 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:53:57.190519 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-689899-84d4f48f6d-48std" Oct 30 16:53:57.199355 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:53:57.195694 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-689899-7c7b96964c-ltxh4" event={"ID":"50a076fb-ca0f-4c64-8c69-38b4ab22b088","Type":"ContainerStarted","Data":"5d64ded69aaf942b97830458b8735faac244b19f71673d4bcbced1ccb32d0153"} Oct 30 16:53:57.263616 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:53:57.262324 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-689899-84d4f48f6d-48std" podStartSLOduration=2.262309921 podStartE2EDuration="2.262309921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:53:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:53:57.247118909 +0000 UTC m=+1992.774454143" watchObservedRunningTime="2026-10-30 16:53:57.262309921 +0000 UTC m=+1992.789645163" Oct 30 16:53:58.205527 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:53:58.205488 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-689899-7c7b96964c-ltxh4" event={"ID":"50a076fb-ca0f-4c64-8c69-38b4ab22b088","Type":"ContainerStarted","Data":"8091e31f66ae35e784eab02725393affca71048ef30bacb31dcf2c92749538ab"} Oct 30 16:53:58.205942 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:53:58.205761 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-689899-7c7b96964c-ltxh4" Oct 30 16:53:58.251386 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:53:58.251326 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-689899-7c7b96964c-ltxh4" podStartSLOduration=3.251307958 podStartE2EDuration="3.251307958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:53:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:53:58.247894058 +0000 UTC m=+1993.775229280" watchObservedRunningTime="2026-10-30 16:53:58.251307958 +0000 UTC m=+1993.778643178" Oct 30 16:54:02.247691 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:02.247653 5052 generic.go:356] "Generic (PLEG): container finished" podID="dd7dff09-720a-411c-9c69-97bc38ac2178" containerID="d320e735460998219605a75ccf5c0b2679ce5f7943727b561b6bacccaf698dab" exitCode=0 Oct 30 16:54:02.248226 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:02.247735 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-c4z7h" event={"ID":"dd7dff09-720a-411c-9c69-97bc38ac2178","Type":"ContainerDied","Data":"d320e735460998219605a75ccf5c0b2679ce5f7943727b561b6bacccaf698dab"} Oct 30 16:54:02.248320 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:02.248264 5052 scope.go:122] "RemoveContainer" containerID="d320e735460998219605a75ccf5c0b2679ce5f7943727b561b6bacccaf698dab" Oct 30 16:54:02.250597 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:02.250557 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:54:02.250688 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:02.250659 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:54:02.250741 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:02.250694 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:54:03.257461 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:03.257429 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-c4z7h" event={"ID":"dd7dff09-720a-411c-9c69-97bc38ac2178","Type":"ContainerStarted","Data":"bd59317436f748983cc0bd1aaa1c88e2b5ae8b0d58b5b4797864cbb0566fa3ce"} Oct 30 16:54:03.512653 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:03.510806 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-c4z7h" Oct 30 16:54:03.547954 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:03.538717 5052 patch_prober.go:29] interesting pod/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-c4z7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Oct 30 16:54:03.547954 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:03.538805 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-c4z7h" podUID="dd7dff09-720a-411c-9c69-97bc38ac2178" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Oct 30 16:54:03.733454 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:03.728247 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-480a60-openshift-default-fb894cdbc-4zzvx"] Oct 30 16:54:03.762629 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:03.757733 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-480a60-openshift-default-fb894cdbc-4zzvx" Oct 30 16:54:03.763262 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:03.763196 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-480a60-openshift-default-fb894cdbc-4zzvx"] Oct 30 16:54:03.776239 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:03.776209 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-480a60-openshift-default-dockercfg-q85jg\"" Oct 30 16:54:03.872548 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:03.872506 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/957040df-7379-406f-8398-573f7ea3bde6-istio-envoy\") pod \"e2e-inf-b-480a60-openshift-default-fb894cdbc-4zzvx\" (UID: \"957040df-7379-406f-8398-573f7ea3bde6\") " pod="openshift-ingress/e2e-inf-b-480a60-openshift-default-fb894cdbc-4zzvx" Oct 30 16:54:03.872788 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:03.872553 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/957040df-7379-406f-8398-573f7ea3bde6-istiod-ca-cert\") pod \"e2e-inf-b-480a60-openshift-default-fb894cdbc-4zzvx\" (UID: \"957040df-7379-406f-8398-573f7ea3bde6\") " pod="openshift-ingress/e2e-inf-b-480a60-openshift-default-fb894cdbc-4zzvx" Oct 30 16:54:03.872788 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:03.872710 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/957040df-7379-406f-8398-573f7ea3bde6-workload-socket\") pod \"e2e-inf-b-480a60-openshift-default-fb894cdbc-4zzvx\" (UID: \"957040df-7379-406f-8398-573f7ea3bde6\") " pod="openshift-ingress/e2e-inf-b-480a60-openshift-default-fb894cdbc-4zzvx" Oct 30 16:54:03.872788 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:03.872744 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/957040df-7379-406f-8398-573f7ea3bde6-istio-podinfo\") pod \"e2e-inf-b-480a60-openshift-default-fb894cdbc-4zzvx\" (UID: \"957040df-7379-406f-8398-573f7ea3bde6\") " pod="openshift-ingress/e2e-inf-b-480a60-openshift-default-fb894cdbc-4zzvx" Oct 30 16:54:03.872980 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:03.872925 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/957040df-7379-406f-8398-573f7ea3bde6-istio-data\") pod \"e2e-inf-b-480a60-openshift-default-fb894cdbc-4zzvx\" (UID: \"957040df-7379-406f-8398-573f7ea3bde6\") " pod="openshift-ingress/e2e-inf-b-480a60-openshift-default-fb894cdbc-4zzvx" Oct 30 16:54:03.872980 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:03.872950 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/957040df-7379-406f-8398-573f7ea3bde6-credential-socket\") pod \"e2e-inf-b-480a60-openshift-default-fb894cdbc-4zzvx\" (UID: \"957040df-7379-406f-8398-573f7ea3bde6\") " pod="openshift-ingress/e2e-inf-b-480a60-openshift-default-fb894cdbc-4zzvx" Oct 30 16:54:03.872980 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:03.872969 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/957040df-7379-406f-8398-573f7ea3bde6-workload-certs\") pod \"e2e-inf-b-480a60-openshift-default-fb894cdbc-4zzvx\" (UID: \"957040df-7379-406f-8398-573f7ea3bde6\") " pod="openshift-ingress/e2e-inf-b-480a60-openshift-default-fb894cdbc-4zzvx" Oct 30 16:54:03.873134 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:03.873110 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/957040df-7379-406f-8398-573f7ea3bde6-istio-token\") pod \"e2e-inf-b-480a60-openshift-default-fb894cdbc-4zzvx\" (UID: \"957040df-7379-406f-8398-573f7ea3bde6\") " pod="openshift-ingress/e2e-inf-b-480a60-openshift-default-fb894cdbc-4zzvx" Oct 30 16:54:03.873263 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:03.873245 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfhq6\" (UniqueName: \"kubernetes.io/projected/957040df-7379-406f-8398-573f7ea3bde6-kube-api-access-gfhq6\") pod \"e2e-inf-b-480a60-openshift-default-fb894cdbc-4zzvx\" (UID: \"957040df-7379-406f-8398-573f7ea3bde6\") " pod="openshift-ingress/e2e-inf-b-480a60-openshift-default-fb894cdbc-4zzvx" Oct 30 16:54:03.974569 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:03.974533 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfhq6\" (UniqueName: \"kubernetes.io/projected/957040df-7379-406f-8398-573f7ea3bde6-kube-api-access-gfhq6\") pod \"e2e-inf-b-480a60-openshift-default-fb894cdbc-4zzvx\" (UID: \"957040df-7379-406f-8398-573f7ea3bde6\") " pod="openshift-ingress/e2e-inf-b-480a60-openshift-default-fb894cdbc-4zzvx" Oct 30 16:54:03.974783 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:03.974633 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/957040df-7379-406f-8398-573f7ea3bde6-istio-envoy\") pod \"e2e-inf-b-480a60-openshift-default-fb894cdbc-4zzvx\" (UID: \"957040df-7379-406f-8398-573f7ea3bde6\") " pod="openshift-ingress/e2e-inf-b-480a60-openshift-default-fb894cdbc-4zzvx" Oct 30 16:54:03.974783 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:03.974655 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/957040df-7379-406f-8398-573f7ea3bde6-istiod-ca-cert\") pod \"e2e-inf-b-480a60-openshift-default-fb894cdbc-4zzvx\" (UID: \"957040df-7379-406f-8398-573f7ea3bde6\") " pod="openshift-ingress/e2e-inf-b-480a60-openshift-default-fb894cdbc-4zzvx" Oct 30 16:54:03.974783 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:03.974689 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/957040df-7379-406f-8398-573f7ea3bde6-workload-socket\") pod \"e2e-inf-b-480a60-openshift-default-fb894cdbc-4zzvx\" (UID: \"957040df-7379-406f-8398-573f7ea3bde6\") " pod="openshift-ingress/e2e-inf-b-480a60-openshift-default-fb894cdbc-4zzvx" Oct 30 16:54:03.974894 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:03.974805 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/957040df-7379-406f-8398-573f7ea3bde6-istio-podinfo\") pod \"e2e-inf-b-480a60-openshift-default-fb894cdbc-4zzvx\" (UID: \"957040df-7379-406f-8398-573f7ea3bde6\") " pod="openshift-ingress/e2e-inf-b-480a60-openshift-default-fb894cdbc-4zzvx" Oct 30 16:54:03.975357 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:03.975051 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/957040df-7379-406f-8398-573f7ea3bde6-istio-data\") pod \"e2e-inf-b-480a60-openshift-default-fb894cdbc-4zzvx\" (UID: \"957040df-7379-406f-8398-573f7ea3bde6\") " pod="openshift-ingress/e2e-inf-b-480a60-openshift-default-fb894cdbc-4zzvx" Oct 30 16:54:03.975357 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:03.975100 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/957040df-7379-406f-8398-573f7ea3bde6-credential-socket\") pod \"e2e-inf-b-480a60-openshift-default-fb894cdbc-4zzvx\" (UID: \"957040df-7379-406f-8398-573f7ea3bde6\") " pod="openshift-ingress/e2e-inf-b-480a60-openshift-default-fb894cdbc-4zzvx" Oct 30 16:54:03.975357 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:03.975125 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/957040df-7379-406f-8398-573f7ea3bde6-workload-certs\") pod \"e2e-inf-b-480a60-openshift-default-fb894cdbc-4zzvx\" (UID: \"957040df-7379-406f-8398-573f7ea3bde6\") " pod="openshift-ingress/e2e-inf-b-480a60-openshift-default-fb894cdbc-4zzvx" Oct 30 16:54:03.975357 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:03.975179 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/957040df-7379-406f-8398-573f7ea3bde6-istio-token\") pod \"e2e-inf-b-480a60-openshift-default-fb894cdbc-4zzvx\" (UID: \"957040df-7379-406f-8398-573f7ea3bde6\") " pod="openshift-ingress/e2e-inf-b-480a60-openshift-default-fb894cdbc-4zzvx" Oct 30 16:54:03.975357 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:03.975175 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/957040df-7379-406f-8398-573f7ea3bde6-workload-socket\") pod \"e2e-inf-b-480a60-openshift-default-fb894cdbc-4zzvx\" (UID: \"957040df-7379-406f-8398-573f7ea3bde6\") " pod="openshift-ingress/e2e-inf-b-480a60-openshift-default-fb894cdbc-4zzvx" Oct 30 16:54:03.975357 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:03.975343 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/957040df-7379-406f-8398-573f7ea3bde6-istio-data\") pod \"e2e-inf-b-480a60-openshift-default-fb894cdbc-4zzvx\" (UID: \"957040df-7379-406f-8398-573f7ea3bde6\") " pod="openshift-ingress/e2e-inf-b-480a60-openshift-default-fb894cdbc-4zzvx" Oct 30 16:54:03.975898 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:03.975347 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/957040df-7379-406f-8398-573f7ea3bde6-istiod-ca-cert\") pod \"e2e-inf-b-480a60-openshift-default-fb894cdbc-4zzvx\" (UID: \"957040df-7379-406f-8398-573f7ea3bde6\") " pod="openshift-ingress/e2e-inf-b-480a60-openshift-default-fb894cdbc-4zzvx" Oct 30 16:54:03.975898 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:03.975427 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/957040df-7379-406f-8398-573f7ea3bde6-credential-socket\") pod \"e2e-inf-b-480a60-openshift-default-fb894cdbc-4zzvx\" (UID: \"957040df-7379-406f-8398-573f7ea3bde6\") " pod="openshift-ingress/e2e-inf-b-480a60-openshift-default-fb894cdbc-4zzvx" Oct 30 16:54:03.975898 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:03.975501 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/957040df-7379-406f-8398-573f7ea3bde6-workload-certs\") pod \"e2e-inf-b-480a60-openshift-default-fb894cdbc-4zzvx\" (UID: \"957040df-7379-406f-8398-573f7ea3bde6\") " pod="openshift-ingress/e2e-inf-b-480a60-openshift-default-fb894cdbc-4zzvx" Oct 30 16:54:03.977288 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:03.977267 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/957040df-7379-406f-8398-573f7ea3bde6-istio-envoy\") pod \"e2e-inf-b-480a60-openshift-default-fb894cdbc-4zzvx\" (UID: \"957040df-7379-406f-8398-573f7ea3bde6\") " pod="openshift-ingress/e2e-inf-b-480a60-openshift-default-fb894cdbc-4zzvx" Oct 30 16:54:03.977528 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:03.977509 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/957040df-7379-406f-8398-573f7ea3bde6-istio-podinfo\") pod \"e2e-inf-b-480a60-openshift-default-fb894cdbc-4zzvx\" (UID: \"957040df-7379-406f-8398-573f7ea3bde6\") " pod="openshift-ingress/e2e-inf-b-480a60-openshift-default-fb894cdbc-4zzvx" Oct 30 16:54:03.998990 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:03.998961 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/957040df-7379-406f-8398-573f7ea3bde6-istio-token\") pod \"e2e-inf-b-480a60-openshift-default-fb894cdbc-4zzvx\" (UID: \"957040df-7379-406f-8398-573f7ea3bde6\") " pod="openshift-ingress/e2e-inf-b-480a60-openshift-default-fb894cdbc-4zzvx" Oct 30 16:54:03.999229 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:03.999206 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfhq6\" (UniqueName: \"kubernetes.io/projected/957040df-7379-406f-8398-573f7ea3bde6-kube-api-access-gfhq6\") pod \"e2e-inf-b-480a60-openshift-default-fb894cdbc-4zzvx\" (UID: \"957040df-7379-406f-8398-573f7ea3bde6\") " pod="openshift-ingress/e2e-inf-b-480a60-openshift-default-fb894cdbc-4zzvx" Oct 30 16:54:04.091888 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:04.091806 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-480a60-openshift-default-fb894cdbc-4zzvx" Oct 30 16:54:04.477607 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:04.475854 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-480a60-openshift-default-fb894cdbc-4zzvx"] Oct 30 16:54:04.502684 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:04.502651 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:54:04.502763 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:04.502730 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:54:04.502763 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:04.502761 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:54:04.503468 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:04.503429 5052 patch_prober.go:29] interesting pod/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-c4z7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Oct 30 16:54:04.503520 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:04.503498 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-c4z7h" podUID="dd7dff09-720a-411c-9c69-97bc38ac2178" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Oct 30 16:54:05.281654 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:05.281619 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-480a60-openshift-default-fb894cdbc-4zzvx" event={"ID":"957040df-7379-406f-8398-573f7ea3bde6","Type":"ContainerStarted","Data":"05766a0cccc395d4f43886ad8d0fc6a646396ea11c0e582683be77aa7237fcb1"} Oct 30 16:54:05.281837 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:05.281674 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-480a60-openshift-default-fb894cdbc-4zzvx" event={"ID":"957040df-7379-406f-8398-573f7ea3bde6","Type":"ContainerStarted","Data":"b63be741b6dfc493e84f4ea4a7006d579ce9962da82d374172149723f89690ae"} Oct 30 16:54:05.356607 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:05.351462 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-480a60-openshift-default-fb894cdbc-4zzvx" podStartSLOduration=2.351442739 podStartE2EDuration="2.351442739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:54:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:54:05.347079189 +0000 UTC m=+2000.874414410" watchObservedRunningTime="2026-10-30 16:54:05.351442739 +0000 UTC m=+2000.878777959" Oct 30 16:54:05.503398 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:05.503366 5052 patch_prober.go:29] interesting pod/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-c4z7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Oct 30 16:54:05.503811 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:05.503415 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-c4z7h" podUID="dd7dff09-720a-411c-9c69-97bc38ac2178" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Oct 30 16:54:06.092650 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:06.092617 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-480a60-openshift-default-fb894cdbc-4zzvx" Oct 30 16:54:06.094341 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:06.094305 5052 patch_prober.go:29] interesting pod/e2e-inf-b-480a60-openshift-default-fb894cdbc-4zzvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Oct 30 16:54:06.094475 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:06.094362 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-480a60-openshift-default-fb894cdbc-4zzvx" podUID="957040df-7379-406f-8398-573f7ea3bde6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Oct 30 16:54:06.503417 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:06.503385 5052 patch_prober.go:29] interesting pod/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-c4z7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Oct 30 16:54:06.503824 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:06.503436 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-c4z7h" podUID="dd7dff09-720a-411c-9c69-97bc38ac2178" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Oct 30 16:54:07.093089 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:07.093051 5052 patch_prober.go:29] interesting pod/e2e-inf-b-480a60-openshift-default-fb894cdbc-4zzvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Oct 30 16:54:07.093274 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:07.093108 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-480a60-openshift-default-fb894cdbc-4zzvx" podUID="957040df-7379-406f-8398-573f7ea3bde6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Oct 30 16:54:07.504459 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:07.504415 5052 patch_prober.go:29] interesting pod/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-c4z7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Oct 30 16:54:07.504932 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:07.504475 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-c4z7h" podUID="dd7dff09-720a-411c-9c69-97bc38ac2178" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Oct 30 16:54:08.092759 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:08.092717 5052 patch_prober.go:29] interesting pod/e2e-inf-b-480a60-openshift-default-fb894cdbc-4zzvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Oct 30 16:54:08.092947 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:08.092779 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-480a60-openshift-default-fb894cdbc-4zzvx" podUID="957040df-7379-406f-8398-573f7ea3bde6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Oct 30 16:54:08.209311 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:08.209278 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-689899-84d4f48f6d-48std" Oct 30 16:54:08.503940 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:08.503896 5052 patch_prober.go:29] interesting pod/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-c4z7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Oct 30 16:54:08.504135 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:08.503987 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-c4z7h" podUID="dd7dff09-720a-411c-9c69-97bc38ac2178" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Oct 30 16:54:09.093600 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:09.093131 5052 patch_prober.go:29] interesting pod/e2e-inf-b-480a60-openshift-default-fb894cdbc-4zzvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Oct 30 16:54:09.093600 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:09.093178 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-480a60-openshift-default-fb894cdbc-4zzvx" podUID="957040df-7379-406f-8398-573f7ea3bde6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Oct 30 16:54:09.222984 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:09.222952 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-689899-7c7b96964c-ltxh4" Oct 30 16:54:09.344656 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:09.342908 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-c4z7h" Oct 30 16:54:09.508807 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:09.508777 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-c4z7h" Oct 30 16:54:09.509746 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:09.509724 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-c4z7h" Oct 30 16:54:10.092741 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:10.092711 5052 patch_prober.go:29] interesting pod/e2e-inf-b-480a60-openshift-default-fb894cdbc-4zzvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Oct 30 16:54:10.092934 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:10.092764 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-480a60-openshift-default-fb894cdbc-4zzvx" podUID="957040df-7379-406f-8398-573f7ea3bde6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Oct 30 16:54:11.092292 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:11.092249 5052 patch_prober.go:29] interesting pod/e2e-inf-b-480a60-openshift-default-fb894cdbc-4zzvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Oct 30 16:54:11.092846 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:11.092317 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-480a60-openshift-default-fb894cdbc-4zzvx" podUID="957040df-7379-406f-8398-573f7ea3bde6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Oct 30 16:54:12.097909 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:12.097823 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-480a60-openshift-default-fb894cdbc-4zzvx" Oct 30 16:54:12.098401 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:12.098180 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-480a60-openshift-default-fb894cdbc-4zzvx" Oct 30 16:54:12.098799 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:12.098778 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-480a60-openshift-default-fb894cdbc-4zzvx" Oct 30 16:54:18.842317 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:54:18.842277 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 16:54:34.202768 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:34.202733 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-f92caf-openshift-default-56d7b858bc-wg985"] Oct 30 16:54:34.208424 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:34.208399 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f92caf-openshift-default-56d7b858bc-wg985" Oct 30 16:54:34.238733 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:34.238700 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-f92caf-openshift-default-dockercfg-cxlmb\"" Oct 30 16:54:34.239073 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:34.239052 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f92caf-openshift-default-56d7b858bc-wg985"] Oct 30 16:54:34.556654 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:34.536616 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e47df7d5-b3d9-4ba4-91a7-7a530ab85616-workload-certs\") pod \"e2e-inf-b-f92caf-openshift-default-56d7b858bc-wg985\" (UID: \"e47df7d5-b3d9-4ba4-91a7-7a530ab85616\") " pod="openshift-ingress/e2e-inf-b-f92caf-openshift-default-56d7b858bc-wg985" Oct 30 16:54:34.556654 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:34.536752 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e47df7d5-b3d9-4ba4-91a7-7a530ab85616-istio-token\") pod \"e2e-inf-b-f92caf-openshift-default-56d7b858bc-wg985\" (UID: \"e47df7d5-b3d9-4ba4-91a7-7a530ab85616\") " pod="openshift-ingress/e2e-inf-b-f92caf-openshift-default-56d7b858bc-wg985" Oct 30 16:54:34.556654 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:34.536869 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e47df7d5-b3d9-4ba4-91a7-7a530ab85616-istio-podinfo\") pod \"e2e-inf-b-f92caf-openshift-default-56d7b858bc-wg985\" (UID: \"e47df7d5-b3d9-4ba4-91a7-7a530ab85616\") " pod="openshift-ingress/e2e-inf-b-f92caf-openshift-default-56d7b858bc-wg985" Oct 30 16:54:34.556654 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:34.536995 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e47df7d5-b3d9-4ba4-91a7-7a530ab85616-credential-socket\") pod \"e2e-inf-b-f92caf-openshift-default-56d7b858bc-wg985\" (UID: \"e47df7d5-b3d9-4ba4-91a7-7a530ab85616\") " pod="openshift-ingress/e2e-inf-b-f92caf-openshift-default-56d7b858bc-wg985" Oct 30 16:54:34.556654 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:34.537054 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zl7zm\" (UniqueName: \"kubernetes.io/projected/e47df7d5-b3d9-4ba4-91a7-7a530ab85616-kube-api-access-zl7zm\") pod \"e2e-inf-b-f92caf-openshift-default-56d7b858bc-wg985\" (UID: \"e47df7d5-b3d9-4ba4-91a7-7a530ab85616\") " pod="openshift-ingress/e2e-inf-b-f92caf-openshift-default-56d7b858bc-wg985" Oct 30 16:54:34.556654 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:34.537096 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e47df7d5-b3d9-4ba4-91a7-7a530ab85616-workload-socket\") pod \"e2e-inf-b-f92caf-openshift-default-56d7b858bc-wg985\" (UID: \"e47df7d5-b3d9-4ba4-91a7-7a530ab85616\") " pod="openshift-ingress/e2e-inf-b-f92caf-openshift-default-56d7b858bc-wg985" Oct 30 16:54:34.556654 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:34.537129 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e47df7d5-b3d9-4ba4-91a7-7a530ab85616-istio-data\") pod \"e2e-inf-b-f92caf-openshift-default-56d7b858bc-wg985\" (UID: \"e47df7d5-b3d9-4ba4-91a7-7a530ab85616\") " pod="openshift-ingress/e2e-inf-b-f92caf-openshift-default-56d7b858bc-wg985" Oct 30 16:54:34.556654 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:34.537220 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e47df7d5-b3d9-4ba4-91a7-7a530ab85616-istio-envoy\") pod \"e2e-inf-b-f92caf-openshift-default-56d7b858bc-wg985\" (UID: \"e47df7d5-b3d9-4ba4-91a7-7a530ab85616\") " pod="openshift-ingress/e2e-inf-b-f92caf-openshift-default-56d7b858bc-wg985" Oct 30 16:54:34.556654 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:34.537253 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e47df7d5-b3d9-4ba4-91a7-7a530ab85616-istiod-ca-cert\") pod \"e2e-inf-b-f92caf-openshift-default-56d7b858bc-wg985\" (UID: \"e47df7d5-b3d9-4ba4-91a7-7a530ab85616\") " pod="openshift-ingress/e2e-inf-b-f92caf-openshift-default-56d7b858bc-wg985" Oct 30 16:54:34.712374 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:34.712334 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e47df7d5-b3d9-4ba4-91a7-7a530ab85616-istio-envoy\") pod \"e2e-inf-b-f92caf-openshift-default-56d7b858bc-wg985\" (UID: \"e47df7d5-b3d9-4ba4-91a7-7a530ab85616\") " pod="openshift-ingress/e2e-inf-b-f92caf-openshift-default-56d7b858bc-wg985" Oct 30 16:54:34.713380 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:34.713346 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e47df7d5-b3d9-4ba4-91a7-7a530ab85616-istiod-ca-cert\") pod \"e2e-inf-b-f92caf-openshift-default-56d7b858bc-wg985\" (UID: \"e47df7d5-b3d9-4ba4-91a7-7a530ab85616\") " pod="openshift-ingress/e2e-inf-b-f92caf-openshift-default-56d7b858bc-wg985" Oct 30 16:54:34.713869 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:34.713837 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e47df7d5-b3d9-4ba4-91a7-7a530ab85616-workload-certs\") pod \"e2e-inf-b-f92caf-openshift-default-56d7b858bc-wg985\" (UID: \"e47df7d5-b3d9-4ba4-91a7-7a530ab85616\") " pod="openshift-ingress/e2e-inf-b-f92caf-openshift-default-56d7b858bc-wg985" Oct 30 16:54:34.714115 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:34.714097 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e47df7d5-b3d9-4ba4-91a7-7a530ab85616-istio-token\") pod \"e2e-inf-b-f92caf-openshift-default-56d7b858bc-wg985\" (UID: \"e47df7d5-b3d9-4ba4-91a7-7a530ab85616\") " pod="openshift-ingress/e2e-inf-b-f92caf-openshift-default-56d7b858bc-wg985" Oct 30 16:54:34.714325 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:34.714170 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e47df7d5-b3d9-4ba4-91a7-7a530ab85616-workload-certs\") pod \"e2e-inf-b-f92caf-openshift-default-56d7b858bc-wg985\" (UID: \"e47df7d5-b3d9-4ba4-91a7-7a530ab85616\") " pod="openshift-ingress/e2e-inf-b-f92caf-openshift-default-56d7b858bc-wg985" Oct 30 16:54:34.714603 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:34.714565 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e47df7d5-b3d9-4ba4-91a7-7a530ab85616-istio-podinfo\") pod \"e2e-inf-b-f92caf-openshift-default-56d7b858bc-wg985\" (UID: \"e47df7d5-b3d9-4ba4-91a7-7a530ab85616\") " pod="openshift-ingress/e2e-inf-b-f92caf-openshift-default-56d7b858bc-wg985" Oct 30 16:54:34.714745 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:34.714728 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e47df7d5-b3d9-4ba4-91a7-7a530ab85616-credential-socket\") pod \"e2e-inf-b-f92caf-openshift-default-56d7b858bc-wg985\" (UID: \"e47df7d5-b3d9-4ba4-91a7-7a530ab85616\") " pod="openshift-ingress/e2e-inf-b-f92caf-openshift-default-56d7b858bc-wg985" Oct 30 16:54:34.714813 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:34.714780 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zl7zm\" (UniqueName: \"kubernetes.io/projected/e47df7d5-b3d9-4ba4-91a7-7a530ab85616-kube-api-access-zl7zm\") pod \"e2e-inf-b-f92caf-openshift-default-56d7b858bc-wg985\" (UID: \"e47df7d5-b3d9-4ba4-91a7-7a530ab85616\") " pod="openshift-ingress/e2e-inf-b-f92caf-openshift-default-56d7b858bc-wg985" Oct 30 16:54:34.714862 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:34.714813 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e47df7d5-b3d9-4ba4-91a7-7a530ab85616-workload-socket\") pod \"e2e-inf-b-f92caf-openshift-default-56d7b858bc-wg985\" (UID: \"e47df7d5-b3d9-4ba4-91a7-7a530ab85616\") " pod="openshift-ingress/e2e-inf-b-f92caf-openshift-default-56d7b858bc-wg985" Oct 30 16:54:34.714862 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:34.714844 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e47df7d5-b3d9-4ba4-91a7-7a530ab85616-istio-data\") pod \"e2e-inf-b-f92caf-openshift-default-56d7b858bc-wg985\" (UID: \"e47df7d5-b3d9-4ba4-91a7-7a530ab85616\") " pod="openshift-ingress/e2e-inf-b-f92caf-openshift-default-56d7b858bc-wg985" Oct 30 16:54:34.715489 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:34.715331 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e47df7d5-b3d9-4ba4-91a7-7a530ab85616-istio-data\") pod \"e2e-inf-b-f92caf-openshift-default-56d7b858bc-wg985\" (UID: \"e47df7d5-b3d9-4ba4-91a7-7a530ab85616\") " pod="openshift-ingress/e2e-inf-b-f92caf-openshift-default-56d7b858bc-wg985" Oct 30 16:54:34.719870 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:34.718918 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e47df7d5-b3d9-4ba4-91a7-7a530ab85616-istio-podinfo\") pod \"e2e-inf-b-f92caf-openshift-default-56d7b858bc-wg985\" (UID: \"e47df7d5-b3d9-4ba4-91a7-7a530ab85616\") " pod="openshift-ingress/e2e-inf-b-f92caf-openshift-default-56d7b858bc-wg985" Oct 30 16:54:34.719870 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:34.718961 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e47df7d5-b3d9-4ba4-91a7-7a530ab85616-credential-socket\") pod \"e2e-inf-b-f92caf-openshift-default-56d7b858bc-wg985\" (UID: \"e47df7d5-b3d9-4ba4-91a7-7a530ab85616\") " pod="openshift-ingress/e2e-inf-b-f92caf-openshift-default-56d7b858bc-wg985" Oct 30 16:54:34.719870 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:34.719212 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e47df7d5-b3d9-4ba4-91a7-7a530ab85616-workload-socket\") pod \"e2e-inf-b-f92caf-openshift-default-56d7b858bc-wg985\" (UID: \"e47df7d5-b3d9-4ba4-91a7-7a530ab85616\") " pod="openshift-ingress/e2e-inf-b-f92caf-openshift-default-56d7b858bc-wg985" Oct 30 16:54:34.727941 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:34.723188 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e47df7d5-b3d9-4ba4-91a7-7a530ab85616-istiod-ca-cert\") pod \"e2e-inf-b-f92caf-openshift-default-56d7b858bc-wg985\" (UID: \"e47df7d5-b3d9-4ba4-91a7-7a530ab85616\") " pod="openshift-ingress/e2e-inf-b-f92caf-openshift-default-56d7b858bc-wg985" Oct 30 16:54:34.727941 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:34.724968 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e47df7d5-b3d9-4ba4-91a7-7a530ab85616-istio-envoy\") pod \"e2e-inf-b-f92caf-openshift-default-56d7b858bc-wg985\" (UID: \"e47df7d5-b3d9-4ba4-91a7-7a530ab85616\") " pod="openshift-ingress/e2e-inf-b-f92caf-openshift-default-56d7b858bc-wg985" Oct 30 16:54:34.739419 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:34.739390 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e47df7d5-b3d9-4ba4-91a7-7a530ab85616-istio-token\") pod \"e2e-inf-b-f92caf-openshift-default-56d7b858bc-wg985\" (UID: \"e47df7d5-b3d9-4ba4-91a7-7a530ab85616\") " pod="openshift-ingress/e2e-inf-b-f92caf-openshift-default-56d7b858bc-wg985" Oct 30 16:54:34.747086 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:34.747055 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zl7zm\" (UniqueName: \"kubernetes.io/projected/e47df7d5-b3d9-4ba4-91a7-7a530ab85616-kube-api-access-zl7zm\") pod \"e2e-inf-b-f92caf-openshift-default-56d7b858bc-wg985\" (UID: \"e47df7d5-b3d9-4ba4-91a7-7a530ab85616\") " pod="openshift-ingress/e2e-inf-b-f92caf-openshift-default-56d7b858bc-wg985" Oct 30 16:54:34.829011 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:34.828920 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f92caf-openshift-default-56d7b858bc-wg985" Oct 30 16:54:35.183310 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:35.183281 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f92caf-openshift-default-56d7b858bc-wg985"] Oct 30 16:54:35.184714 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:54:35.184677 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode47df7d5_b3d9_4ba4_91a7_7a530ab85616.slice/crio-9dedd91c60662c49eb431a67efb1f39ed0dac5ed7bf5fcabdb31df2472fa4a78 WatchSource:0}: Error finding container 9dedd91c60662c49eb431a67efb1f39ed0dac5ed7bf5fcabdb31df2472fa4a78: Status 404 returned error can't find the container with id 9dedd91c60662c49eb431a67efb1f39ed0dac5ed7bf5fcabdb31df2472fa4a78 Oct 30 16:54:35.187699 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:35.187670 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:54:35.187812 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:35.187732 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:54:35.187812 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:35.187764 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:54:35.716892 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:35.716852 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f92caf-openshift-default-56d7b858bc-wg985" event={"ID":"e47df7d5-b3d9-4ba4-91a7-7a530ab85616","Type":"ContainerStarted","Data":"f9f1ac74236c917e6bf027705dfb0f20b260a1dbef4d2363b76d7d39cd18f6e8"} Oct 30 16:54:35.717287 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:35.716918 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f92caf-openshift-default-56d7b858bc-wg985" event={"ID":"e47df7d5-b3d9-4ba4-91a7-7a530ab85616","Type":"ContainerStarted","Data":"9dedd91c60662c49eb431a67efb1f39ed0dac5ed7bf5fcabdb31df2472fa4a78"} Oct 30 16:54:35.782118 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:35.782069 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-f92caf-openshift-default-56d7b858bc-wg985" podStartSLOduration=1.782055852 podStartE2EDuration="1.782055852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:54:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:54:35.777820061 +0000 UTC m=+2031.305155282" watchObservedRunningTime="2026-10-30 16:54:35.782055852 +0000 UTC m=+2031.309391070" Oct 30 16:54:35.829045 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:35.829012 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-f92caf-openshift-default-56d7b858bc-wg985" Oct 30 16:54:36.841788 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:36.841755 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-f92caf-openshift-default-56d7b858bc-wg985" Oct 30 16:54:37.792457 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:37.792428 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-f92caf-openshift-default-56d7b858bc-wg985" Oct 30 16:54:37.793367 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:37.793347 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-f92caf-openshift-default-56d7b858bc-wg985" Oct 30 16:54:45.269742 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:54:45.269709 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-529bc7/test-model-529bc7-kserve-69c945bd47-d6dvw"] Oct 30 16:54:48.889904 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:54:48.889776 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 16:55:00.292526 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:00.292488 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-529bc7/test-model-529bc7-kserve-69c945bd47-d6dvw" podUID="cc5babf8-ffe2-4a65-bce7-b6b9b9cf174b" containerName="main" containerID="cri-o://3e8351e9c73edc47c1fc4b415cec79f713c1023a081acb7ab3e31dbcf5d29d32" gracePeriod=45 Oct 30 16:55:00.303230 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:00.303198 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-529bc7/test-model-529bc7-kserve-69c945bd47-d6dvw" podUID="cc5babf8-ffe2-4a65-bce7-b6b9b9cf174b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.145:8000/ready\": dial tcp 10.132.0.145:8000: connect: connection refused" Oct 30 16:55:00.614943 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:00.614918 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-529bc7/test-model-529bc7-kserve-69c945bd47-d6dvw" Oct 30 16:55:00.732202 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:00.732167 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc5babf8-ffe2-4a65-bce7-b6b9b9cf174b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc5babf8-ffe2-4a65-bce7-b6b9b9cf174b-tmp-dir\") pod \"cc5babf8-ffe2-4a65-bce7-b6b9b9cf174b\" (UID: \"cc5babf8-ffe2-4a65-bce7-b6b9b9cf174b\") " Oct 30 16:55:00.732394 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:00.732299 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc5babf8-ffe2-4a65-bce7-b6b9b9cf174b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc5babf8-ffe2-4a65-bce7-b6b9b9cf174b-dshm\") pod \"cc5babf8-ffe2-4a65-bce7-b6b9b9cf174b\" (UID: \"cc5babf8-ffe2-4a65-bce7-b6b9b9cf174b\") " Oct 30 16:55:00.732394 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:00.732361 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc5babf8-ffe2-4a65-bce7-b6b9b9cf174b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc5babf8-ffe2-4a65-bce7-b6b9b9cf174b-model-cache\") pod \"cc5babf8-ffe2-4a65-bce7-b6b9b9cf174b\" (UID: \"cc5babf8-ffe2-4a65-bce7-b6b9b9cf174b\") " Oct 30 16:55:00.732522 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:00.732450 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc5babf8-ffe2-4a65-bce7-b6b9b9cf174b-tmp-dir" pod "cc5babf8-ffe2-4a65-bce7-b6b9b9cf174b" (UID: "cc5babf8-ffe2-4a65-bce7-b6b9b9cf174b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:55:00.732522 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:00.732459 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc5babf8-ffe2-4a65-bce7-b6b9b9cf174b-kube-api-access-2x7nt\" (UniqueName: \"kubernetes.io/projected/cc5babf8-ffe2-4a65-bce7-b6b9b9cf174b-kube-api-access-2x7nt\") pod \"cc5babf8-ffe2-4a65-bce7-b6b9b9cf174b\" (UID: \"cc5babf8-ffe2-4a65-bce7-b6b9b9cf174b\") " Oct 30 16:55:00.732522 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:00.732511 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc5babf8-ffe2-4a65-bce7-b6b9b9cf174b-tls-certs\" (UniqueName: \"kubernetes.io/secret/cc5babf8-ffe2-4a65-bce7-b6b9b9cf174b-tls-certs\") pod \"cc5babf8-ffe2-4a65-bce7-b6b9b9cf174b\" (UID: \"cc5babf8-ffe2-4a65-bce7-b6b9b9cf174b\") " Oct 30 16:55:00.732744 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:00.732611 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc5babf8-ffe2-4a65-bce7-b6b9b9cf174b-home\" (UniqueName: \"kubernetes.io/empty-dir/cc5babf8-ffe2-4a65-bce7-b6b9b9cf174b-home\") pod \"cc5babf8-ffe2-4a65-bce7-b6b9b9cf174b\" (UID: \"cc5babf8-ffe2-4a65-bce7-b6b9b9cf174b\") " Oct 30 16:55:00.732744 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:00.732658 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc5babf8-ffe2-4a65-bce7-b6b9b9cf174b-model-cache" pod "cc5babf8-ffe2-4a65-bce7-b6b9b9cf174b" (UID: "cc5babf8-ffe2-4a65-bce7-b6b9b9cf174b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:55:00.733010 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:00.732987 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc5babf8-ffe2-4a65-bce7-b6b9b9cf174b-home" pod "cc5babf8-ffe2-4a65-bce7-b6b9b9cf174b" (UID: "cc5babf8-ffe2-4a65-bce7-b6b9b9cf174b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:55:00.733389 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:00.733366 5052 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc5babf8-ffe2-4a65-bce7-b6b9b9cf174b-tmp-dir\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:55:00.733478 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:00.733392 5052 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc5babf8-ffe2-4a65-bce7-b6b9b9cf174b-model-cache\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:55:00.733478 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:00.733404 5052 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc5babf8-ffe2-4a65-bce7-b6b9b9cf174b-home\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:55:00.734921 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:00.734892 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc5babf8-ffe2-4a65-bce7-b6b9b9cf174b-tls-certs" pod "cc5babf8-ffe2-4a65-bce7-b6b9b9cf174b" (UID: "cc5babf8-ffe2-4a65-bce7-b6b9b9cf174b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:55:00.735172 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:00.735146 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc5babf8-ffe2-4a65-bce7-b6b9b9cf174b-dshm" pod "cc5babf8-ffe2-4a65-bce7-b6b9b9cf174b" (UID: "cc5babf8-ffe2-4a65-bce7-b6b9b9cf174b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:55:00.735260 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:00.735227 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc5babf8-ffe2-4a65-bce7-b6b9b9cf174b-kube-api-access-2x7nt" pod "cc5babf8-ffe2-4a65-bce7-b6b9b9cf174b" (UID: "cc5babf8-ffe2-4a65-bce7-b6b9b9cf174b"). InnerVolumeSpecName "kube-api-access-2x7nt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:55:00.835509 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:00.835428 5052 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc5babf8-ffe2-4a65-bce7-b6b9b9cf174b-dshm\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:55:00.835509 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:00.835455 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2x7nt\" (UniqueName: \"kubernetes.io/projected/cc5babf8-ffe2-4a65-bce7-b6b9b9cf174b-kube-api-access-2x7nt\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:55:00.835509 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:00.835466 5052 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc5babf8-ffe2-4a65-bce7-b6b9b9cf174b-tls-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:55:01.081116 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:01.081083 5052 generic.go:356] "Generic (PLEG): container finished" podID="cc5babf8-ffe2-4a65-bce7-b6b9b9cf174b" containerID="3e8351e9c73edc47c1fc4b415cec79f713c1023a081acb7ab3e31dbcf5d29d32" exitCode=0 Oct 30 16:55:01.081320 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:01.081141 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-529bc7/test-model-529bc7-kserve-69c945bd47-d6dvw" Oct 30 16:55:01.081320 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:01.081169 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-529bc7/test-model-529bc7-kserve-69c945bd47-d6dvw" event={"ID":"cc5babf8-ffe2-4a65-bce7-b6b9b9cf174b","Type":"ContainerDied","Data":"3e8351e9c73edc47c1fc4b415cec79f713c1023a081acb7ab3e31dbcf5d29d32"} Oct 30 16:55:01.081320 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:01.081221 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-529bc7/test-model-529bc7-kserve-69c945bd47-d6dvw" event={"ID":"cc5babf8-ffe2-4a65-bce7-b6b9b9cf174b","Type":"ContainerDied","Data":"6f9ed158d720b562aad8a7796d63c61e9e5ce3bf6d286208f504475605d440bb"} Oct 30 16:55:01.081320 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:01.081248 5052 scope.go:122] "RemoveContainer" containerID="3e8351e9c73edc47c1fc4b415cec79f713c1023a081acb7ab3e31dbcf5d29d32" Oct 30 16:55:01.096410 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:01.096387 5052 scope.go:122] "RemoveContainer" containerID="3e8351e9c73edc47c1fc4b415cec79f713c1023a081acb7ab3e31dbcf5d29d32" Oct 30 16:55:01.096750 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:55:01.096730 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e8351e9c73edc47c1fc4b415cec79f713c1023a081acb7ab3e31dbcf5d29d32\": container with ID starting with 3e8351e9c73edc47c1fc4b415cec79f713c1023a081acb7ab3e31dbcf5d29d32 not found: ID does not exist" containerID="3e8351e9c73edc47c1fc4b415cec79f713c1023a081acb7ab3e31dbcf5d29d32" Oct 30 16:55:01.096805 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:01.096760 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e8351e9c73edc47c1fc4b415cec79f713c1023a081acb7ab3e31dbcf5d29d32"} err="failed to get container status \"3e8351e9c73edc47c1fc4b415cec79f713c1023a081acb7ab3e31dbcf5d29d32\": rpc error: code = NotFound desc = could not find container \"3e8351e9c73edc47c1fc4b415cec79f713c1023a081acb7ab3e31dbcf5d29d32\": container with ID starting with 3e8351e9c73edc47c1fc4b415cec79f713c1023a081acb7ab3e31dbcf5d29d32 not found: ID does not exist" Oct 30 16:55:01.138638 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:01.138613 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-529bc7/test-model-529bc7-kserve-69c945bd47-d6dvw"] Oct 30 16:55:01.151546 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:01.151519 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-529bc7/test-model-529bc7-kserve-69c945bd47-d6dvw"] Oct 30 16:55:02.600388 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:02.600356 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc5babf8-ffe2-4a65-bce7-b6b9b9cf174b" path="/var/lib/kubelet/pods/cc5babf8-ffe2-4a65-bce7-b6b9b9cf174b/volumes" Oct 30 16:55:05.260834 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:05.260803 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bvq2v"] Oct 30 16:55:05.261242 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:05.261010 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bvq2v" podUID="2cb92868-e7dd-4783-a4c1-2d2cd75b70fb" containerName="istio-proxy" containerID="cri-o://e8e58497167dbf6f3aadf3c42ec8cbdd05dc813bb74b2b262bc7d08d16c82d43" gracePeriod=30 Oct 30 16:55:05.276726 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:05.276695 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-l6dw5"] Oct 30 16:55:05.276936 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:05.276910 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-l6dw5" podUID="4b3c4341-826b-489d-81ae-14bf71548dd2" containerName="istio-proxy" containerID="cri-o://61a80927dc0ea83ef7657778c0d0804ae5e9df84cd7cd20f0694497902857cbd" gracePeriod=30 Oct 30 16:55:05.803610 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:05.799055 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cdwcj"] Oct 30 16:55:09.377171 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:09.377139 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-3ee682/test-model-3ee682-kserve-7c6cd66757-dm6lb"] Oct 30 16:55:10.580782 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:10.580753 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bvq2v" Oct 30 16:55:10.647794 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:10.647738 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-l6dw5" Oct 30 16:55:10.676302 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:10.676266 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cb92868-e7dd-4783-a4c1-2d2cd75b70fb-kube-api-access-gwdwh\" (UniqueName: \"kubernetes.io/projected/2cb92868-e7dd-4783-a4c1-2d2cd75b70fb-kube-api-access-gwdwh\") pod \"2cb92868-e7dd-4783-a4c1-2d2cd75b70fb\" (UID: \"2cb92868-e7dd-4783-a4c1-2d2cd75b70fb\") " Oct 30 16:55:10.676462 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:10.676373 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2cb92868-e7dd-4783-a4c1-2d2cd75b70fb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cb92868-e7dd-4783-a4c1-2d2cd75b70fb-istiod-ca-cert\") pod \"2cb92868-e7dd-4783-a4c1-2d2cd75b70fb\" (UID: \"2cb92868-e7dd-4783-a4c1-2d2cd75b70fb\") " Oct 30 16:55:10.676462 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:10.676427 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cb92868-e7dd-4783-a4c1-2d2cd75b70fb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cb92868-e7dd-4783-a4c1-2d2cd75b70fb-workload-certs\") pod \"2cb92868-e7dd-4783-a4c1-2d2cd75b70fb\" (UID: \"2cb92868-e7dd-4783-a4c1-2d2cd75b70fb\") " Oct 30 16:55:10.676668 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:10.676643 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cb92868-e7dd-4783-a4c1-2d2cd75b70fb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cb92868-e7dd-4783-a4c1-2d2cd75b70fb-istio-envoy\") pod \"2cb92868-e7dd-4783-a4c1-2d2cd75b70fb\" (UID: \"2cb92868-e7dd-4783-a4c1-2d2cd75b70fb\") " Oct 30 16:55:10.676815 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:10.676773 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cb92868-e7dd-4783-a4c1-2d2cd75b70fb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cb92868-e7dd-4783-a4c1-2d2cd75b70fb-istio-data\") pod \"2cb92868-e7dd-4783-a4c1-2d2cd75b70fb\" (UID: \"2cb92868-e7dd-4783-a4c1-2d2cd75b70fb\") " Oct 30 16:55:10.676889 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:10.676868 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2cb92868-e7dd-4783-a4c1-2d2cd75b70fb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cb92868-e7dd-4783-a4c1-2d2cd75b70fb-istio-podinfo\") pod \"2cb92868-e7dd-4783-a4c1-2d2cd75b70fb\" (UID: \"2cb92868-e7dd-4783-a4c1-2d2cd75b70fb\") " Oct 30 16:55:10.676947 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:10.676905 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cb92868-e7dd-4783-a4c1-2d2cd75b70fb-istio-token\" (UniqueName: \"kubernetes.io/projected/2cb92868-e7dd-4783-a4c1-2d2cd75b70fb-istio-token\") pod \"2cb92868-e7dd-4783-a4c1-2d2cd75b70fb\" (UID: \"2cb92868-e7dd-4783-a4c1-2d2cd75b70fb\") " Oct 30 16:55:10.677008 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:10.676954 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cb92868-e7dd-4783-a4c1-2d2cd75b70fb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cb92868-e7dd-4783-a4c1-2d2cd75b70fb-workload-socket\") pod \"2cb92868-e7dd-4783-a4c1-2d2cd75b70fb\" (UID: \"2cb92868-e7dd-4783-a4c1-2d2cd75b70fb\") " Oct 30 16:55:10.677008 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:10.677002 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cb92868-e7dd-4783-a4c1-2d2cd75b70fb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cb92868-e7dd-4783-a4c1-2d2cd75b70fb-credential-socket\") pod \"2cb92868-e7dd-4783-a4c1-2d2cd75b70fb\" (UID: \"2cb92868-e7dd-4783-a4c1-2d2cd75b70fb\") " Oct 30 16:55:10.678813 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:10.678788 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cb92868-e7dd-4783-a4c1-2d2cd75b70fb-istiod-ca-cert" pod "2cb92868-e7dd-4783-a4c1-2d2cd75b70fb" (UID: "2cb92868-e7dd-4783-a4c1-2d2cd75b70fb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:55:10.679006 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:10.678982 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cb92868-e7dd-4783-a4c1-2d2cd75b70fb-workload-certs" pod "2cb92868-e7dd-4783-a4c1-2d2cd75b70fb" (UID: "2cb92868-e7dd-4783-a4c1-2d2cd75b70fb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:55:10.679092 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:10.679002 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cb92868-e7dd-4783-a4c1-2d2cd75b70fb-istio-data" pod "2cb92868-e7dd-4783-a4c1-2d2cd75b70fb" (UID: "2cb92868-e7dd-4783-a4c1-2d2cd75b70fb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:55:10.679190 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:10.679150 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cb92868-e7dd-4783-a4c1-2d2cd75b70fb-kube-api-access-gwdwh" pod "2cb92868-e7dd-4783-a4c1-2d2cd75b70fb" (UID: "2cb92868-e7dd-4783-a4c1-2d2cd75b70fb"). InnerVolumeSpecName "kube-api-access-gwdwh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:55:10.679291 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:10.679269 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cb92868-e7dd-4783-a4c1-2d2cd75b70fb-workload-socket" pod "2cb92868-e7dd-4783-a4c1-2d2cd75b70fb" (UID: "2cb92868-e7dd-4783-a4c1-2d2cd75b70fb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:55:10.679531 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:10.679503 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cb92868-e7dd-4783-a4c1-2d2cd75b70fb-credential-socket" pod "2cb92868-e7dd-4783-a4c1-2d2cd75b70fb" (UID: "2cb92868-e7dd-4783-a4c1-2d2cd75b70fb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:55:10.680964 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:10.680939 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2cb92868-e7dd-4783-a4c1-2d2cd75b70fb-istio-podinfo" pod "2cb92868-e7dd-4783-a4c1-2d2cd75b70fb" (UID: "2cb92868-e7dd-4783-a4c1-2d2cd75b70fb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:55:10.681716 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:10.681697 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cb92868-e7dd-4783-a4c1-2d2cd75b70fb-istio-token" pod "2cb92868-e7dd-4783-a4c1-2d2cd75b70fb" (UID: "2cb92868-e7dd-4783-a4c1-2d2cd75b70fb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:55:10.681916 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:10.681889 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cb92868-e7dd-4783-a4c1-2d2cd75b70fb-istio-envoy" pod "2cb92868-e7dd-4783-a4c1-2d2cd75b70fb" (UID: "2cb92868-e7dd-4783-a4c1-2d2cd75b70fb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:55:10.779261 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:10.779179 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b3c4341-826b-489d-81ae-14bf71548dd2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b3c4341-826b-489d-81ae-14bf71548dd2-credential-socket\") pod \"4b3c4341-826b-489d-81ae-14bf71548dd2\" (UID: \"4b3c4341-826b-489d-81ae-14bf71548dd2\") " Oct 30 16:55:10.779412 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:10.779322 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b3c4341-826b-489d-81ae-14bf71548dd2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b3c4341-826b-489d-81ae-14bf71548dd2-istiod-ca-cert\") pod \"4b3c4341-826b-489d-81ae-14bf71548dd2\" (UID: \"4b3c4341-826b-489d-81ae-14bf71548dd2\") " Oct 30 16:55:10.779412 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:10.779386 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b3c4341-826b-489d-81ae-14bf71548dd2-istio-token\" (UniqueName: \"kubernetes.io/projected/4b3c4341-826b-489d-81ae-14bf71548dd2-istio-token\") pod \"4b3c4341-826b-489d-81ae-14bf71548dd2\" (UID: \"4b3c4341-826b-489d-81ae-14bf71548dd2\") " Oct 30 16:55:10.779568 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:10.779445 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b3c4341-826b-489d-81ae-14bf71548dd2-credential-socket" pod "4b3c4341-826b-489d-81ae-14bf71548dd2" (UID: "4b3c4341-826b-489d-81ae-14bf71548dd2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:55:10.779568 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:10.779482 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4b3c4341-826b-489d-81ae-14bf71548dd2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b3c4341-826b-489d-81ae-14bf71548dd2-istio-podinfo\") pod \"4b3c4341-826b-489d-81ae-14bf71548dd2\" (UID: \"4b3c4341-826b-489d-81ae-14bf71548dd2\") " Oct 30 16:55:10.779568 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:10.779515 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b3c4341-826b-489d-81ae-14bf71548dd2-kube-api-access-4fxpn\" (UniqueName: \"kubernetes.io/projected/4b3c4341-826b-489d-81ae-14bf71548dd2-kube-api-access-4fxpn\") pod \"4b3c4341-826b-489d-81ae-14bf71548dd2\" (UID: \"4b3c4341-826b-489d-81ae-14bf71548dd2\") " Oct 30 16:55:10.779568 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:10.779549 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b3c4341-826b-489d-81ae-14bf71548dd2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b3c4341-826b-489d-81ae-14bf71548dd2-istio-envoy\") pod \"4b3c4341-826b-489d-81ae-14bf71548dd2\" (UID: \"4b3c4341-826b-489d-81ae-14bf71548dd2\") " Oct 30 16:55:10.779856 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:10.779630 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b3c4341-826b-489d-81ae-14bf71548dd2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b3c4341-826b-489d-81ae-14bf71548dd2-istio-data\") pod \"4b3c4341-826b-489d-81ae-14bf71548dd2\" (UID: \"4b3c4341-826b-489d-81ae-14bf71548dd2\") " Oct 30 16:55:10.779856 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:10.779682 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b3c4341-826b-489d-81ae-14bf71548dd2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b3c4341-826b-489d-81ae-14bf71548dd2-workload-certs\") pod \"4b3c4341-826b-489d-81ae-14bf71548dd2\" (UID: \"4b3c4341-826b-489d-81ae-14bf71548dd2\") " Oct 30 16:55:10.779856 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:10.779688 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b3c4341-826b-489d-81ae-14bf71548dd2-istiod-ca-cert" pod "4b3c4341-826b-489d-81ae-14bf71548dd2" (UID: "4b3c4341-826b-489d-81ae-14bf71548dd2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:55:10.779856 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:10.779734 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b3c4341-826b-489d-81ae-14bf71548dd2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b3c4341-826b-489d-81ae-14bf71548dd2-workload-socket\") pod \"4b3c4341-826b-489d-81ae-14bf71548dd2\" (UID: \"4b3c4341-826b-489d-81ae-14bf71548dd2\") " Oct 30 16:55:10.780068 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:10.779998 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b3c4341-826b-489d-81ae-14bf71548dd2-istio-data" pod "4b3c4341-826b-489d-81ae-14bf71548dd2" (UID: "4b3c4341-826b-489d-81ae-14bf71548dd2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:55:10.780365 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:10.780147 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b3c4341-826b-489d-81ae-14bf71548dd2-workload-certs" pod "4b3c4341-826b-489d-81ae-14bf71548dd2" (UID: "4b3c4341-826b-489d-81ae-14bf71548dd2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:55:10.780365 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:10.780300 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b3c4341-826b-489d-81ae-14bf71548dd2-workload-socket" pod "4b3c4341-826b-489d-81ae-14bf71548dd2" (UID: "4b3c4341-826b-489d-81ae-14bf71548dd2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:55:10.780734 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:10.780709 5052 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b3c4341-826b-489d-81ae-14bf71548dd2-workload-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:55:10.780826 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:10.780743 5052 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cb92868-e7dd-4783-a4c1-2d2cd75b70fb-istio-envoy\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:55:10.780826 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:10.780758 5052 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b3c4341-826b-489d-81ae-14bf71548dd2-workload-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:55:10.780826 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:10.780773 5052 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cb92868-e7dd-4783-a4c1-2d2cd75b70fb-istio-data\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:55:10.780826 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:10.780788 5052 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cb92868-e7dd-4783-a4c1-2d2cd75b70fb-istio-podinfo\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:55:10.780826 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:10.780801 5052 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cb92868-e7dd-4783-a4c1-2d2cd75b70fb-istio-token\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:55:10.780826 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:10.780817 5052 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b3c4341-826b-489d-81ae-14bf71548dd2-credential-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:55:10.781172 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:10.780832 5052 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cb92868-e7dd-4783-a4c1-2d2cd75b70fb-workload-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:55:10.781172 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:10.780846 5052 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cb92868-e7dd-4783-a4c1-2d2cd75b70fb-credential-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:55:10.781172 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:10.780859 5052 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b3c4341-826b-489d-81ae-14bf71548dd2-istiod-ca-cert\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:55:10.781172 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:10.780874 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwdwh\" (UniqueName: \"kubernetes.io/projected/2cb92868-e7dd-4783-a4c1-2d2cd75b70fb-kube-api-access-gwdwh\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:55:10.781172 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:10.780887 5052 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cb92868-e7dd-4783-a4c1-2d2cd75b70fb-istiod-ca-cert\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:55:10.781172 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:10.780901 5052 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cb92868-e7dd-4783-a4c1-2d2cd75b70fb-workload-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:55:10.781172 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:10.780914 5052 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b3c4341-826b-489d-81ae-14bf71548dd2-istio-data\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:55:10.782087 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:10.782059 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b3c4341-826b-489d-81ae-14bf71548dd2-istio-token" pod "4b3c4341-826b-489d-81ae-14bf71548dd2" (UID: "4b3c4341-826b-489d-81ae-14bf71548dd2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:55:10.782190 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:10.782110 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4b3c4341-826b-489d-81ae-14bf71548dd2-istio-podinfo" pod "4b3c4341-826b-489d-81ae-14bf71548dd2" (UID: "4b3c4341-826b-489d-81ae-14bf71548dd2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:55:10.782260 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:10.782183 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b3c4341-826b-489d-81ae-14bf71548dd2-istio-envoy" pod "4b3c4341-826b-489d-81ae-14bf71548dd2" (UID: "4b3c4341-826b-489d-81ae-14bf71548dd2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:55:10.783346 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:10.783329 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b3c4341-826b-489d-81ae-14bf71548dd2-kube-api-access-4fxpn" pod "4b3c4341-826b-489d-81ae-14bf71548dd2" (UID: "4b3c4341-826b-489d-81ae-14bf71548dd2"). InnerVolumeSpecName "kube-api-access-4fxpn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:55:10.882813 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:10.882778 5052 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b3c4341-826b-489d-81ae-14bf71548dd2-istio-token\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:55:10.882813 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:10.882810 5052 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b3c4341-826b-489d-81ae-14bf71548dd2-istio-podinfo\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:55:10.882813 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:10.882820 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fxpn\" (UniqueName: \"kubernetes.io/projected/4b3c4341-826b-489d-81ae-14bf71548dd2-kube-api-access-4fxpn\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:55:10.883068 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:10.882830 5052 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b3c4341-826b-489d-81ae-14bf71548dd2-istio-envoy\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:55:11.242514 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:11.242480 5052 generic.go:356] "Generic (PLEG): container finished" podID="2cb92868-e7dd-4783-a4c1-2d2cd75b70fb" containerID="e8e58497167dbf6f3aadf3c42ec8cbdd05dc813bb74b2b262bc7d08d16c82d43" exitCode=0 Oct 30 16:55:11.242745 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:11.242538 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bvq2v" Oct 30 16:55:11.242745 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:11.242559 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bvq2v" event={"ID":"2cb92868-e7dd-4783-a4c1-2d2cd75b70fb","Type":"ContainerDied","Data":"e8e58497167dbf6f3aadf3c42ec8cbdd05dc813bb74b2b262bc7d08d16c82d43"} Oct 30 16:55:11.242745 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:11.242667 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bvq2v" event={"ID":"2cb92868-e7dd-4783-a4c1-2d2cd75b70fb","Type":"ContainerDied","Data":"11474e3cd12a86de143f1f711f41aea666bdbd87eb52736228384c1773984673"} Oct 30 16:55:11.242745 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:11.242703 5052 scope.go:122] "RemoveContainer" containerID="e8e58497167dbf6f3aadf3c42ec8cbdd05dc813bb74b2b262bc7d08d16c82d43" Oct 30 16:55:11.244300 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:11.244275 5052 generic.go:356] "Generic (PLEG): container finished" podID="4b3c4341-826b-489d-81ae-14bf71548dd2" containerID="61a80927dc0ea83ef7657778c0d0804ae5e9df84cd7cd20f0694497902857cbd" exitCode=0 Oct 30 16:55:11.244420 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:11.244325 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-l6dw5" event={"ID":"4b3c4341-826b-489d-81ae-14bf71548dd2","Type":"ContainerDied","Data":"61a80927dc0ea83ef7657778c0d0804ae5e9df84cd7cd20f0694497902857cbd"} Oct 30 16:55:11.244420 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:11.244338 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-l6dw5" Oct 30 16:55:11.244420 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:11.244364 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-l6dw5" event={"ID":"4b3c4341-826b-489d-81ae-14bf71548dd2","Type":"ContainerDied","Data":"6c135430463cbe1328bdae9a63dbcb496fbdb48e1472839813cd052a427ade35"} Oct 30 16:55:11.255692 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:11.255674 5052 scope.go:122] "RemoveContainer" containerID="e8e58497167dbf6f3aadf3c42ec8cbdd05dc813bb74b2b262bc7d08d16c82d43" Oct 30 16:55:11.255970 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:55:11.255946 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8e58497167dbf6f3aadf3c42ec8cbdd05dc813bb74b2b262bc7d08d16c82d43\": container with ID starting with e8e58497167dbf6f3aadf3c42ec8cbdd05dc813bb74b2b262bc7d08d16c82d43 not found: ID does not exist" containerID="e8e58497167dbf6f3aadf3c42ec8cbdd05dc813bb74b2b262bc7d08d16c82d43" Oct 30 16:55:11.256052 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:11.255975 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8e58497167dbf6f3aadf3c42ec8cbdd05dc813bb74b2b262bc7d08d16c82d43"} err="failed to get container status \"e8e58497167dbf6f3aadf3c42ec8cbdd05dc813bb74b2b262bc7d08d16c82d43\": rpc error: code = NotFound desc = could not find container \"e8e58497167dbf6f3aadf3c42ec8cbdd05dc813bb74b2b262bc7d08d16c82d43\": container with ID starting with e8e58497167dbf6f3aadf3c42ec8cbdd05dc813bb74b2b262bc7d08d16c82d43 not found: ID does not exist" Oct 30 16:55:11.256052 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:11.255991 5052 scope.go:122] "RemoveContainer" containerID="61a80927dc0ea83ef7657778c0d0804ae5e9df84cd7cd20f0694497902857cbd" Oct 30 16:55:11.267293 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:11.267269 5052 scope.go:122] "RemoveContainer" containerID="61a80927dc0ea83ef7657778c0d0804ae5e9df84cd7cd20f0694497902857cbd" Oct 30 16:55:11.267710 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:55:11.267686 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61a80927dc0ea83ef7657778c0d0804ae5e9df84cd7cd20f0694497902857cbd\": container with ID starting with 61a80927dc0ea83ef7657778c0d0804ae5e9df84cd7cd20f0694497902857cbd not found: ID does not exist" containerID="61a80927dc0ea83ef7657778c0d0804ae5e9df84cd7cd20f0694497902857cbd" Oct 30 16:55:11.267803 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:11.267718 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61a80927dc0ea83ef7657778c0d0804ae5e9df84cd7cd20f0694497902857cbd"} err="failed to get container status \"61a80927dc0ea83ef7657778c0d0804ae5e9df84cd7cd20f0694497902857cbd\": rpc error: code = NotFound desc = could not find container \"61a80927dc0ea83ef7657778c0d0804ae5e9df84cd7cd20f0694497902857cbd\": container with ID starting with 61a80927dc0ea83ef7657778c0d0804ae5e9df84cd7cd20f0694497902857cbd not found: ID does not exist" Oct 30 16:55:11.302608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:11.302570 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-l6dw5"] Oct 30 16:55:11.313652 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:11.313627 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-l6dw5"] Oct 30 16:55:11.346466 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:11.346408 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bvq2v"] Oct 30 16:55:11.351062 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:11.351039 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bvq2v"] Oct 30 16:55:12.599393 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:12.599351 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cb92868-e7dd-4783-a4c1-2d2cd75b70fb" path="/var/lib/kubelet/pods/2cb92868-e7dd-4783-a4c1-2d2cd75b70fb/volumes" Oct 30 16:55:12.600058 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:12.600037 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b3c4341-826b-489d-81ae-14bf71548dd2" path="/var/lib/kubelet/pods/4b3c4341-826b-489d-81ae-14bf71548dd2/volumes" Oct 30 16:55:18.285541 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:18.285508 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-wcwwf"] Oct 30 16:55:18.286369 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:18.286351 5052 cpu_manager.go:414] "containerMap: removing container" podUID="4b3c4341-826b-489d-81ae-14bf71548dd2" containerName="istio-proxy" Oct 30 16:55:18.286369 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:18.286368 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b3c4341-826b-489d-81ae-14bf71548dd2" containerName="istio-proxy" Oct 30 16:55:18.286495 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:18.286384 5052 cpu_manager.go:414] "containerMap: removing container" podUID="cc5babf8-ffe2-4a65-bce7-b6b9b9cf174b" containerName="main" Oct 30 16:55:18.286495 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:18.286389 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc5babf8-ffe2-4a65-bce7-b6b9b9cf174b" containerName="main" Oct 30 16:55:18.286495 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:18.286409 5052 cpu_manager.go:414] "containerMap: removing container" podUID="2cb92868-e7dd-4783-a4c1-2d2cd75b70fb" containerName="istio-proxy" Oct 30 16:55:18.286495 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:18.286414 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cb92868-e7dd-4783-a4c1-2d2cd75b70fb" containerName="istio-proxy" Oct 30 16:55:18.286658 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:18.286560 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b3c4341-826b-489d-81ae-14bf71548dd2" containerName="istio-proxy" Oct 30 16:55:18.286658 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:18.286570 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc5babf8-ffe2-4a65-bce7-b6b9b9cf174b" containerName="main" Oct 30 16:55:18.286658 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:18.286596 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cb92868-e7dd-4783-a4c1-2d2cd75b70fb" containerName="istio-proxy" Oct 30 16:55:18.292116 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:18.292097 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-wcwwf" Oct 30 16:55:18.304639 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:18.304617 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-p6cdk"] Oct 30 16:55:18.309278 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:18.309260 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-p6cdk" Oct 30 16:55:18.312979 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:18.312937 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-689899-openshift-default-dockercfg-x7vsc\"" Oct 30 16:55:18.319180 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:18.318985 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-wcwwf"] Oct 30 16:55:18.348639 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:18.341274 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-p6cdk"] Oct 30 16:55:18.424307 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:18.424275 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fff40615-0fac-46d8-ad06-9fe5942628dd-istio-token\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-p6cdk\" (UID: \"fff40615-0fac-46d8-ad06-9fe5942628dd\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-p6cdk" Oct 30 16:55:18.424478 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:18.424309 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a89cc50e-1a94-4b08-80af-963baa6f4927-credential-socket\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-wcwwf\" (UID: \"a89cc50e-1a94-4b08-80af-963baa6f4927\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-wcwwf" Oct 30 16:55:18.424478 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:18.424398 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fff40615-0fac-46d8-ad06-9fe5942628dd-istiod-ca-cert\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-p6cdk\" (UID: \"fff40615-0fac-46d8-ad06-9fe5942628dd\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-p6cdk" Oct 30 16:55:18.424478 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:18.424457 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a89cc50e-1a94-4b08-80af-963baa6f4927-workload-socket\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-wcwwf\" (UID: \"a89cc50e-1a94-4b08-80af-963baa6f4927\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-wcwwf" Oct 30 16:55:18.424618 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:18.424563 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a89cc50e-1a94-4b08-80af-963baa6f4927-istio-data\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-wcwwf\" (UID: \"a89cc50e-1a94-4b08-80af-963baa6f4927\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-wcwwf" Oct 30 16:55:18.424657 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:18.424626 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fff40615-0fac-46d8-ad06-9fe5942628dd-credential-socket\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-p6cdk\" (UID: \"fff40615-0fac-46d8-ad06-9fe5942628dd\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-p6cdk" Oct 30 16:55:18.424657 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:18.424645 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a89cc50e-1a94-4b08-80af-963baa6f4927-istio-envoy\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-wcwwf\" (UID: \"a89cc50e-1a94-4b08-80af-963baa6f4927\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-wcwwf" Oct 30 16:55:18.424723 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:18.424701 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fff40615-0fac-46d8-ad06-9fe5942628dd-istio-podinfo\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-p6cdk\" (UID: \"fff40615-0fac-46d8-ad06-9fe5942628dd\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-p6cdk" Oct 30 16:55:18.424763 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:18.424750 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fff40615-0fac-46d8-ad06-9fe5942628dd-istio-data\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-p6cdk\" (UID: \"fff40615-0fac-46d8-ad06-9fe5942628dd\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-p6cdk" Oct 30 16:55:18.424798 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:18.424769 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a89cc50e-1a94-4b08-80af-963baa6f4927-workload-certs\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-wcwwf\" (UID: \"a89cc50e-1a94-4b08-80af-963baa6f4927\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-wcwwf" Oct 30 16:55:18.424798 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:18.424790 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fff40615-0fac-46d8-ad06-9fe5942628dd-workload-socket\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-p6cdk\" (UID: \"fff40615-0fac-46d8-ad06-9fe5942628dd\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-p6cdk" Oct 30 16:55:18.424869 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:18.424807 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgkrx\" (UniqueName: \"kubernetes.io/projected/fff40615-0fac-46d8-ad06-9fe5942628dd-kube-api-access-cgkrx\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-p6cdk\" (UID: \"fff40615-0fac-46d8-ad06-9fe5942628dd\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-p6cdk" Oct 30 16:55:18.424869 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:18.424829 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fff40615-0fac-46d8-ad06-9fe5942628dd-workload-certs\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-p6cdk\" (UID: \"fff40615-0fac-46d8-ad06-9fe5942628dd\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-p6cdk" Oct 30 16:55:18.424958 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:18.424942 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fff40615-0fac-46d8-ad06-9fe5942628dd-istio-envoy\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-p6cdk\" (UID: \"fff40615-0fac-46d8-ad06-9fe5942628dd\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-p6cdk" Oct 30 16:55:18.425023 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:18.425011 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twcnd\" (UniqueName: \"kubernetes.io/projected/a89cc50e-1a94-4b08-80af-963baa6f4927-kube-api-access-twcnd\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-wcwwf\" (UID: \"a89cc50e-1a94-4b08-80af-963baa6f4927\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-wcwwf" Oct 30 16:55:18.425063 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:18.425036 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a89cc50e-1a94-4b08-80af-963baa6f4927-istiod-ca-cert\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-wcwwf\" (UID: \"a89cc50e-1a94-4b08-80af-963baa6f4927\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-wcwwf" Oct 30 16:55:18.425100 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:18.425063 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a89cc50e-1a94-4b08-80af-963baa6f4927-istio-podinfo\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-wcwwf\" (UID: \"a89cc50e-1a94-4b08-80af-963baa6f4927\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-wcwwf" Oct 30 16:55:18.425271 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:18.425252 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a89cc50e-1a94-4b08-80af-963baa6f4927-istio-token\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-wcwwf\" (UID: \"a89cc50e-1a94-4b08-80af-963baa6f4927\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-wcwwf" Oct 30 16:55:18.526273 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:18.526242 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a89cc50e-1a94-4b08-80af-963baa6f4927-istio-token\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-wcwwf\" (UID: \"a89cc50e-1a94-4b08-80af-963baa6f4927\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-wcwwf" Oct 30 16:55:18.526437 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:18.526286 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fff40615-0fac-46d8-ad06-9fe5942628dd-istio-token\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-p6cdk\" (UID: \"fff40615-0fac-46d8-ad06-9fe5942628dd\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-p6cdk" Oct 30 16:55:18.526437 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:18.526310 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a89cc50e-1a94-4b08-80af-963baa6f4927-credential-socket\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-wcwwf\" (UID: \"a89cc50e-1a94-4b08-80af-963baa6f4927\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-wcwwf" Oct 30 16:55:18.526437 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:18.526366 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fff40615-0fac-46d8-ad06-9fe5942628dd-istiod-ca-cert\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-p6cdk\" (UID: \"fff40615-0fac-46d8-ad06-9fe5942628dd\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-p6cdk" Oct 30 16:55:18.526437 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:18.526433 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a89cc50e-1a94-4b08-80af-963baa6f4927-workload-socket\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-wcwwf\" (UID: \"a89cc50e-1a94-4b08-80af-963baa6f4927\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-wcwwf" Oct 30 16:55:18.526736 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:18.526478 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a89cc50e-1a94-4b08-80af-963baa6f4927-istio-data\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-wcwwf\" (UID: \"a89cc50e-1a94-4b08-80af-963baa6f4927\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-wcwwf" Oct 30 16:55:18.526736 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:18.526516 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fff40615-0fac-46d8-ad06-9fe5942628dd-credential-socket\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-p6cdk\" (UID: \"fff40615-0fac-46d8-ad06-9fe5942628dd\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-p6cdk" Oct 30 16:55:18.526736 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:18.526543 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a89cc50e-1a94-4b08-80af-963baa6f4927-istio-envoy\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-wcwwf\" (UID: \"a89cc50e-1a94-4b08-80af-963baa6f4927\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-wcwwf" Oct 30 16:55:18.526736 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:18.526616 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fff40615-0fac-46d8-ad06-9fe5942628dd-istio-podinfo\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-p6cdk\" (UID: \"fff40615-0fac-46d8-ad06-9fe5942628dd\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-p6cdk" Oct 30 16:55:18.526736 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:18.526656 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fff40615-0fac-46d8-ad06-9fe5942628dd-istio-data\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-p6cdk\" (UID: \"fff40615-0fac-46d8-ad06-9fe5942628dd\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-p6cdk" Oct 30 16:55:18.526736 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:18.526679 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a89cc50e-1a94-4b08-80af-963baa6f4927-workload-certs\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-wcwwf\" (UID: \"a89cc50e-1a94-4b08-80af-963baa6f4927\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-wcwwf" Oct 30 16:55:18.526736 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:18.526720 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fff40615-0fac-46d8-ad06-9fe5942628dd-workload-socket\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-p6cdk\" (UID: \"fff40615-0fac-46d8-ad06-9fe5942628dd\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-p6cdk" Oct 30 16:55:18.527108 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:18.526745 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgkrx\" (UniqueName: \"kubernetes.io/projected/fff40615-0fac-46d8-ad06-9fe5942628dd-kube-api-access-cgkrx\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-p6cdk\" (UID: \"fff40615-0fac-46d8-ad06-9fe5942628dd\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-p6cdk" Oct 30 16:55:18.527108 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:18.526783 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fff40615-0fac-46d8-ad06-9fe5942628dd-workload-certs\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-p6cdk\" (UID: \"fff40615-0fac-46d8-ad06-9fe5942628dd\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-p6cdk" Oct 30 16:55:18.527108 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:18.526839 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fff40615-0fac-46d8-ad06-9fe5942628dd-istio-envoy\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-p6cdk\" (UID: \"fff40615-0fac-46d8-ad06-9fe5942628dd\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-p6cdk" Oct 30 16:55:18.527108 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:18.526859 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a89cc50e-1a94-4b08-80af-963baa6f4927-credential-socket\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-wcwwf\" (UID: \"a89cc50e-1a94-4b08-80af-963baa6f4927\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-wcwwf" Oct 30 16:55:18.527108 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:18.526902 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twcnd\" (UniqueName: \"kubernetes.io/projected/a89cc50e-1a94-4b08-80af-963baa6f4927-kube-api-access-twcnd\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-wcwwf\" (UID: \"a89cc50e-1a94-4b08-80af-963baa6f4927\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-wcwwf" Oct 30 16:55:18.527108 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:18.526938 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a89cc50e-1a94-4b08-80af-963baa6f4927-istiod-ca-cert\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-wcwwf\" (UID: \"a89cc50e-1a94-4b08-80af-963baa6f4927\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-wcwwf" Oct 30 16:55:18.527108 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:18.526937 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a89cc50e-1a94-4b08-80af-963baa6f4927-istio-data\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-wcwwf\" (UID: \"a89cc50e-1a94-4b08-80af-963baa6f4927\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-wcwwf" Oct 30 16:55:18.527108 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:18.526963 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fff40615-0fac-46d8-ad06-9fe5942628dd-credential-socket\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-p6cdk\" (UID: \"fff40615-0fac-46d8-ad06-9fe5942628dd\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-p6cdk" Oct 30 16:55:18.527108 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:18.526990 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a89cc50e-1a94-4b08-80af-963baa6f4927-istio-podinfo\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-wcwwf\" (UID: \"a89cc50e-1a94-4b08-80af-963baa6f4927\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-wcwwf" Oct 30 16:55:18.527600 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:18.527140 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fff40615-0fac-46d8-ad06-9fe5942628dd-workload-socket\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-p6cdk\" (UID: \"fff40615-0fac-46d8-ad06-9fe5942628dd\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-p6cdk" Oct 30 16:55:18.527600 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:18.527238 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fff40615-0fac-46d8-ad06-9fe5942628dd-workload-certs\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-p6cdk\" (UID: \"fff40615-0fac-46d8-ad06-9fe5942628dd\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-p6cdk" Oct 30 16:55:18.527600 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:18.527293 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fff40615-0fac-46d8-ad06-9fe5942628dd-istiod-ca-cert\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-p6cdk\" (UID: \"fff40615-0fac-46d8-ad06-9fe5942628dd\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-p6cdk" Oct 30 16:55:18.527600 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:18.527413 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a89cc50e-1a94-4b08-80af-963baa6f4927-workload-certs\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-wcwwf\" (UID: \"a89cc50e-1a94-4b08-80af-963baa6f4927\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-wcwwf" Oct 30 16:55:18.527600 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:18.527447 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fff40615-0fac-46d8-ad06-9fe5942628dd-istio-data\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-p6cdk\" (UID: \"fff40615-0fac-46d8-ad06-9fe5942628dd\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-p6cdk" Oct 30 16:55:18.527600 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:18.527553 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a89cc50e-1a94-4b08-80af-963baa6f4927-workload-socket\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-wcwwf\" (UID: \"a89cc50e-1a94-4b08-80af-963baa6f4927\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-wcwwf" Oct 30 16:55:18.527875 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:18.527622 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a89cc50e-1a94-4b08-80af-963baa6f4927-istiod-ca-cert\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-wcwwf\" (UID: \"a89cc50e-1a94-4b08-80af-963baa6f4927\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-wcwwf" Oct 30 16:55:18.529548 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:18.529521 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a89cc50e-1a94-4b08-80af-963baa6f4927-istio-envoy\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-wcwwf\" (UID: \"a89cc50e-1a94-4b08-80af-963baa6f4927\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-wcwwf" Oct 30 16:55:18.529655 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:18.529533 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fff40615-0fac-46d8-ad06-9fe5942628dd-istio-envoy\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-p6cdk\" (UID: \"fff40615-0fac-46d8-ad06-9fe5942628dd\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-p6cdk" Oct 30 16:55:18.529777 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:18.529760 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fff40615-0fac-46d8-ad06-9fe5942628dd-istio-podinfo\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-p6cdk\" (UID: \"fff40615-0fac-46d8-ad06-9fe5942628dd\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-p6cdk" Oct 30 16:55:18.529903 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:18.529883 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a89cc50e-1a94-4b08-80af-963baa6f4927-istio-podinfo\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-wcwwf\" (UID: \"a89cc50e-1a94-4b08-80af-963baa6f4927\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-wcwwf" Oct 30 16:55:18.557650 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:18.556673 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fff40615-0fac-46d8-ad06-9fe5942628dd-istio-token\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-p6cdk\" (UID: \"fff40615-0fac-46d8-ad06-9fe5942628dd\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-p6cdk" Oct 30 16:55:18.557650 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:18.556672 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a89cc50e-1a94-4b08-80af-963baa6f4927-istio-token\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-wcwwf\" (UID: \"a89cc50e-1a94-4b08-80af-963baa6f4927\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-wcwwf" Oct 30 16:55:18.557650 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:18.557090 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twcnd\" (UniqueName: \"kubernetes.io/projected/a89cc50e-1a94-4b08-80af-963baa6f4927-kube-api-access-twcnd\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-wcwwf\" (UID: \"a89cc50e-1a94-4b08-80af-963baa6f4927\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-wcwwf" Oct 30 16:55:18.558568 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:18.558545 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgkrx\" (UniqueName: \"kubernetes.io/projected/fff40615-0fac-46d8-ad06-9fe5942628dd-kube-api-access-cgkrx\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-p6cdk\" (UID: \"fff40615-0fac-46d8-ad06-9fe5942628dd\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-p6cdk" Oct 30 16:55:18.609602 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:18.609459 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-wcwwf" Oct 30 16:55:18.627865 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:18.627836 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-p6cdk" Oct 30 16:55:18.845758 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:55:18.845665 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 16:55:18.985572 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:55:18.985543 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfff40615_0fac_46d8_ad06_9fe5942628dd.slice/crio-e56efcbd87b681bfa4074029c78d75863ccf102be23813f063a199ff8b97c0f3 WatchSource:0}: Error finding container e56efcbd87b681bfa4074029c78d75863ccf102be23813f063a199ff8b97c0f3: Status 404 returned error can't find the container with id e56efcbd87b681bfa4074029c78d75863ccf102be23813f063a199ff8b97c0f3 Oct 30 16:55:18.986936 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:55:18.986913 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda89cc50e_1a94_4b08_80af_963baa6f4927.slice/crio-6402918c6cc2119b518f8f6bb5bb158b2f547d423b019a82afda1bd61f9b8321 WatchSource:0}: Error finding container 6402918c6cc2119b518f8f6bb5bb158b2f547d423b019a82afda1bd61f9b8321: Status 404 returned error can't find the container with id 6402918c6cc2119b518f8f6bb5bb158b2f547d423b019a82afda1bd61f9b8321 Oct 30 16:55:18.987095 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:18.987082 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-wcwwf"] Oct 30 16:55:18.989017 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:18.988988 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:55:18.989102 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:18.989050 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:55:18.989102 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:18.989086 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:55:18.990435 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:18.990400 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:55:18.990546 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:18.990477 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:55:18.990546 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:18.990518 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:55:18.993125 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:18.993100 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-p6cdk"] Oct 30 16:55:19.380431 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:19.380395 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-p6cdk" event={"ID":"fff40615-0fac-46d8-ad06-9fe5942628dd","Type":"ContainerStarted","Data":"3a14f6a0a7b3107db8ee0362834771ff79e779c6c05b2bdae99879b2482caeee"} Oct 30 16:55:19.381080 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:19.381054 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-p6cdk" event={"ID":"fff40615-0fac-46d8-ad06-9fe5942628dd","Type":"ContainerStarted","Data":"e56efcbd87b681bfa4074029c78d75863ccf102be23813f063a199ff8b97c0f3"} Oct 30 16:55:19.384354 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:19.384311 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-wcwwf" event={"ID":"a89cc50e-1a94-4b08-80af-963baa6f4927","Type":"ContainerStarted","Data":"8619a13fe20d3a5dec404c8e15d80778913a856f4f65a26dc3f359c1f4022c85"} Oct 30 16:55:19.384477 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:19.384372 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-wcwwf" event={"ID":"a89cc50e-1a94-4b08-80af-963baa6f4927","Type":"ContainerStarted","Data":"6402918c6cc2119b518f8f6bb5bb158b2f547d423b019a82afda1bd61f9b8321"} Oct 30 16:55:19.434712 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:19.434645 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-p6cdk" podStartSLOduration=1.434625454 podStartE2EDuration="1.434625454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:55:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:55:19.428888905 +0000 UTC m=+2074.956224125" watchObservedRunningTime="2026-10-30 16:55:19.434625454 +0000 UTC m=+2074.961960674" Oct 30 16:55:19.522604 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:19.504115 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-wcwwf" podStartSLOduration=1.504092663 podStartE2EDuration="1.504092663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:55:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:55:19.494254316 +0000 UTC m=+2075.021589530" watchObservedRunningTime="2026-10-30 16:55:19.504092663 +0000 UTC m=+2075.031427884" Oct 30 16:55:19.609575 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:19.609542 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-wcwwf" Oct 30 16:55:19.628891 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:19.628865 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-p6cdk" Oct 30 16:55:20.611217 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:20.611185 5052 patch_prober.go:29] interesting pod/e2e-inf-b-689899-openshift-default-66cbd45f99-wcwwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Oct 30 16:55:20.611656 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:20.611242 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-wcwwf" podUID="a89cc50e-1a94-4b08-80af-963baa6f4927" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Oct 30 16:55:20.675092 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:20.675044 5052 patch_prober.go:29] interesting pod/e2e-inf-b-689899-openshift-default-66cbd45f99-p6cdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Oct 30 16:55:20.675406 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:20.675352 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-p6cdk" podUID="fff40615-0fac-46d8-ad06-9fe5942628dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Oct 30 16:55:21.645605 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:21.644685 5052 patch_prober.go:29] interesting pod/e2e-inf-b-689899-openshift-default-66cbd45f99-p6cdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Oct 30 16:55:21.645605 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:21.644750 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-p6cdk" podUID="fff40615-0fac-46d8-ad06-9fe5942628dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Oct 30 16:55:21.645605 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:21.645180 5052 patch_prober.go:29] interesting pod/e2e-inf-b-689899-openshift-default-66cbd45f99-wcwwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Oct 30 16:55:21.645605 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:21.645217 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-wcwwf" podUID="a89cc50e-1a94-4b08-80af-963baa6f4927" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Oct 30 16:55:22.609929 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:22.609896 5052 patch_prober.go:29] interesting pod/e2e-inf-b-689899-openshift-default-66cbd45f99-wcwwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Oct 30 16:55:22.610103 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:22.609948 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-wcwwf" podUID="a89cc50e-1a94-4b08-80af-963baa6f4927" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Oct 30 16:55:22.638458 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:22.638425 5052 patch_prober.go:29] interesting pod/e2e-inf-b-689899-openshift-default-66cbd45f99-p6cdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Oct 30 16:55:22.638729 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:22.638701 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-p6cdk" podUID="fff40615-0fac-46d8-ad06-9fe5942628dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Oct 30 16:55:23.610658 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:23.610620 5052 patch_prober.go:29] interesting pod/e2e-inf-b-689899-openshift-default-66cbd45f99-wcwwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Oct 30 16:55:23.611073 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:23.610686 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-wcwwf" podUID="a89cc50e-1a94-4b08-80af-963baa6f4927" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Oct 30 16:55:23.645040 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:23.645001 5052 patch_prober.go:29] interesting pod/e2e-inf-b-689899-openshift-default-66cbd45f99-p6cdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Oct 30 16:55:23.645311 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:23.645284 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-p6cdk" podUID="fff40615-0fac-46d8-ad06-9fe5942628dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Oct 30 16:55:24.401924 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:24.401883 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-3ee682/test-model-3ee682-kserve-7c6cd66757-dm6lb" podUID="4a6bd706-03cb-4f01-a5ac-ffdd2757f279" containerName="main" containerID="cri-o://98faf4b68d5dbf67c77767a9d36ce4e6ecbd182b39788ca216a0aa59d8e362d4" gracePeriod=45 Oct 30 16:55:24.610098 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:24.610065 5052 patch_prober.go:29] interesting pod/e2e-inf-b-689899-openshift-default-66cbd45f99-wcwwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Oct 30 16:55:24.610252 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:24.610116 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-wcwwf" podUID="a89cc50e-1a94-4b08-80af-963baa6f4927" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Oct 30 16:55:24.692875 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:24.692662 5052 patch_prober.go:29] interesting pod/e2e-inf-b-689899-openshift-default-66cbd45f99-p6cdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Oct 30 16:55:24.692875 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:24.692717 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-p6cdk" podUID="fff40615-0fac-46d8-ad06-9fe5942628dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Oct 30 16:55:24.893664 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:24.893634 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-3ee682/test-model-3ee682-kserve-7c6cd66757-dm6lb" Oct 30 16:55:24.895906 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:24.895884 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a6bd706-03cb-4f01-a5ac-ffdd2757f279-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a6bd706-03cb-4f01-a5ac-ffdd2757f279-model-cache\") pod \"4a6bd706-03cb-4f01-a5ac-ffdd2757f279\" (UID: \"4a6bd706-03cb-4f01-a5ac-ffdd2757f279\") " Oct 30 16:55:24.895996 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:24.895949 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a6bd706-03cb-4f01-a5ac-ffdd2757f279-tls-certs\" (UniqueName: \"kubernetes.io/secret/4a6bd706-03cb-4f01-a5ac-ffdd2757f279-tls-certs\") pod \"4a6bd706-03cb-4f01-a5ac-ffdd2757f279\" (UID: \"4a6bd706-03cb-4f01-a5ac-ffdd2757f279\") " Oct 30 16:55:24.895996 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:24.895992 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a6bd706-03cb-4f01-a5ac-ffdd2757f279-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a6bd706-03cb-4f01-a5ac-ffdd2757f279-tmp-dir\") pod \"4a6bd706-03cb-4f01-a5ac-ffdd2757f279\" (UID: \"4a6bd706-03cb-4f01-a5ac-ffdd2757f279\") " Oct 30 16:55:24.896140 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:24.896027 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a6bd706-03cb-4f01-a5ac-ffdd2757f279-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a6bd706-03cb-4f01-a5ac-ffdd2757f279-dshm\") pod \"4a6bd706-03cb-4f01-a5ac-ffdd2757f279\" (UID: \"4a6bd706-03cb-4f01-a5ac-ffdd2757f279\") " Oct 30 16:55:24.896140 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:24.896107 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a6bd706-03cb-4f01-a5ac-ffdd2757f279-kube-api-access-mpggj\" (UniqueName: \"kubernetes.io/projected/4a6bd706-03cb-4f01-a5ac-ffdd2757f279-kube-api-access-mpggj\") pod \"4a6bd706-03cb-4f01-a5ac-ffdd2757f279\" (UID: \"4a6bd706-03cb-4f01-a5ac-ffdd2757f279\") " Oct 30 16:55:24.896207 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:24.896159 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a6bd706-03cb-4f01-a5ac-ffdd2757f279-home\" (UniqueName: \"kubernetes.io/empty-dir/4a6bd706-03cb-4f01-a5ac-ffdd2757f279-home\") pod \"4a6bd706-03cb-4f01-a5ac-ffdd2757f279\" (UID: \"4a6bd706-03cb-4f01-a5ac-ffdd2757f279\") " Oct 30 16:55:24.897002 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:24.896964 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a6bd706-03cb-4f01-a5ac-ffdd2757f279-tmp-dir" pod "4a6bd706-03cb-4f01-a5ac-ffdd2757f279" (UID: "4a6bd706-03cb-4f01-a5ac-ffdd2757f279"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:55:24.897068 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:24.897044 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a6bd706-03cb-4f01-a5ac-ffdd2757f279-model-cache" pod "4a6bd706-03cb-4f01-a5ac-ffdd2757f279" (UID: "4a6bd706-03cb-4f01-a5ac-ffdd2757f279"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:55:24.897299 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:24.897278 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a6bd706-03cb-4f01-a5ac-ffdd2757f279-home" pod "4a6bd706-03cb-4f01-a5ac-ffdd2757f279" (UID: "4a6bd706-03cb-4f01-a5ac-ffdd2757f279"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:55:24.899039 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:24.899011 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a6bd706-03cb-4f01-a5ac-ffdd2757f279-tls-certs" pod "4a6bd706-03cb-4f01-a5ac-ffdd2757f279" (UID: "4a6bd706-03cb-4f01-a5ac-ffdd2757f279"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:55:24.899149 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:24.899130 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a6bd706-03cb-4f01-a5ac-ffdd2757f279-kube-api-access-mpggj" pod "4a6bd706-03cb-4f01-a5ac-ffdd2757f279" (UID: "4a6bd706-03cb-4f01-a5ac-ffdd2757f279"). InnerVolumeSpecName "kube-api-access-mpggj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:55:24.901094 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:24.901073 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a6bd706-03cb-4f01-a5ac-ffdd2757f279-dshm" pod "4a6bd706-03cb-4f01-a5ac-ffdd2757f279" (UID: "4a6bd706-03cb-4f01-a5ac-ffdd2757f279"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:55:24.999674 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:24.998918 5052 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a6bd706-03cb-4f01-a5ac-ffdd2757f279-tls-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:55:24.999674 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:24.998950 5052 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a6bd706-03cb-4f01-a5ac-ffdd2757f279-tmp-dir\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:55:24.999674 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:24.998963 5052 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a6bd706-03cb-4f01-a5ac-ffdd2757f279-dshm\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:55:24.999674 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:24.998976 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpggj\" (UniqueName: \"kubernetes.io/projected/4a6bd706-03cb-4f01-a5ac-ffdd2757f279-kube-api-access-mpggj\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:55:24.999674 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:24.998989 5052 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a6bd706-03cb-4f01-a5ac-ffdd2757f279-home\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:55:24.999674 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:24.999003 5052 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a6bd706-03cb-4f01-a5ac-ffdd2757f279-model-cache\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:55:25.511764 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:25.511730 5052 generic.go:356] "Generic (PLEG): container finished" podID="4a6bd706-03cb-4f01-a5ac-ffdd2757f279" containerID="98faf4b68d5dbf67c77767a9d36ce4e6ecbd182b39788ca216a0aa59d8e362d4" exitCode=0 Oct 30 16:55:25.511982 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:25.511780 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-3ee682/test-model-3ee682-kserve-7c6cd66757-dm6lb" Oct 30 16:55:25.511982 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:25.511826 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-3ee682/test-model-3ee682-kserve-7c6cd66757-dm6lb" event={"ID":"4a6bd706-03cb-4f01-a5ac-ffdd2757f279","Type":"ContainerDied","Data":"98faf4b68d5dbf67c77767a9d36ce4e6ecbd182b39788ca216a0aa59d8e362d4"} Oct 30 16:55:25.511982 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:25.511869 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-3ee682/test-model-3ee682-kserve-7c6cd66757-dm6lb" event={"ID":"4a6bd706-03cb-4f01-a5ac-ffdd2757f279","Type":"ContainerDied","Data":"5d3d1e51969e41b34e2c2ff5dee68536325469f6bf2cfa24a367ba0b91f2105a"} Oct 30 16:55:25.511982 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:25.511894 5052 scope.go:122] "RemoveContainer" containerID="98faf4b68d5dbf67c77767a9d36ce4e6ecbd182b39788ca216a0aa59d8e362d4" Oct 30 16:55:25.524058 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:25.524041 5052 scope.go:122] "RemoveContainer" containerID="98faf4b68d5dbf67c77767a9d36ce4e6ecbd182b39788ca216a0aa59d8e362d4" Oct 30 16:55:25.524343 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:55:25.524322 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98faf4b68d5dbf67c77767a9d36ce4e6ecbd182b39788ca216a0aa59d8e362d4\": container with ID starting with 98faf4b68d5dbf67c77767a9d36ce4e6ecbd182b39788ca216a0aa59d8e362d4 not found: ID does not exist" containerID="98faf4b68d5dbf67c77767a9d36ce4e6ecbd182b39788ca216a0aa59d8e362d4" Oct 30 16:55:25.524437 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:25.524348 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98faf4b68d5dbf67c77767a9d36ce4e6ecbd182b39788ca216a0aa59d8e362d4"} err="failed to get container status \"98faf4b68d5dbf67c77767a9d36ce4e6ecbd182b39788ca216a0aa59d8e362d4\": rpc error: code = NotFound desc = could not find container \"98faf4b68d5dbf67c77767a9d36ce4e6ecbd182b39788ca216a0aa59d8e362d4\": container with ID starting with 98faf4b68d5dbf67c77767a9d36ce4e6ecbd182b39788ca216a0aa59d8e362d4 not found: ID does not exist" Oct 30 16:55:25.585254 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:25.585215 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-3ee682/test-model-3ee682-kserve-7c6cd66757-dm6lb"] Oct 30 16:55:25.596366 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:25.596343 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-3ee682/test-model-3ee682-kserve-7c6cd66757-dm6lb"] Oct 30 16:55:25.610168 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:25.610136 5052 patch_prober.go:29] interesting pod/e2e-inf-b-689899-openshift-default-66cbd45f99-wcwwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Oct 30 16:55:25.610292 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:25.610216 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-wcwwf" podUID="a89cc50e-1a94-4b08-80af-963baa6f4927" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Oct 30 16:55:25.649914 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:25.646030 5052 patch_prober.go:29] interesting pod/e2e-inf-b-689899-openshift-default-66cbd45f99-p6cdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Oct 30 16:55:25.649914 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:25.646092 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-p6cdk" podUID="fff40615-0fac-46d8-ad06-9fe5942628dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Oct 30 16:55:26.600165 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:26.600130 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a6bd706-03cb-4f01-a5ac-ffdd2757f279" path="/var/lib/kubelet/pods/4a6bd706-03cb-4f01-a5ac-ffdd2757f279/volumes" Oct 30 16:55:26.615433 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:26.615403 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-wcwwf" Oct 30 16:55:26.615782 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:26.615762 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-wcwwf" Oct 30 16:55:26.616354 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:26.616335 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-wcwwf" Oct 30 16:55:26.633700 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:26.633679 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-p6cdk" Oct 30 16:55:26.633952 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:26.633931 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-p6cdk" Oct 30 16:55:26.634555 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:26.634535 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-p6cdk" Oct 30 16:55:32.771742 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:32.771708 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-9bfcac-openshift-default-676454d6b5-wh8mx"] Oct 30 16:55:32.773732 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:32.773703 5052 cpu_manager.go:414] "containerMap: removing container" podUID="4a6bd706-03cb-4f01-a5ac-ffdd2757f279" containerName="main" Oct 30 16:55:32.773732 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:32.773731 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a6bd706-03cb-4f01-a5ac-ffdd2757f279" containerName="main" Oct 30 16:55:32.773930 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:32.773917 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a6bd706-03cb-4f01-a5ac-ffdd2757f279" containerName="main" Oct 30 16:55:32.779336 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:32.779314 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9bfcac-openshift-default-676454d6b5-wh8mx" Oct 30 16:55:32.798014 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:32.797982 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-9bfcac-openshift-default-676454d6b5-wh8mx"] Oct 30 16:55:32.805418 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:32.805385 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-9bfcac-openshift-default-dockercfg-7dd52\"" Oct 30 16:55:32.823088 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:32.823048 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/500d3721-818e-4ad7-b087-6dfd1c1d6368-workload-socket\") pod \"e2e-inf-b-9bfcac-openshift-default-676454d6b5-wh8mx\" (UID: \"500d3721-818e-4ad7-b087-6dfd1c1d6368\") " pod="openshift-ingress/e2e-inf-b-9bfcac-openshift-default-676454d6b5-wh8mx" Oct 30 16:55:32.823230 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:32.823179 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/500d3721-818e-4ad7-b087-6dfd1c1d6368-istio-podinfo\") pod \"e2e-inf-b-9bfcac-openshift-default-676454d6b5-wh8mx\" (UID: \"500d3721-818e-4ad7-b087-6dfd1c1d6368\") " pod="openshift-ingress/e2e-inf-b-9bfcac-openshift-default-676454d6b5-wh8mx" Oct 30 16:55:32.823303 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:32.823262 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/500d3721-818e-4ad7-b087-6dfd1c1d6368-workload-certs\") pod \"e2e-inf-b-9bfcac-openshift-default-676454d6b5-wh8mx\" (UID: \"500d3721-818e-4ad7-b087-6dfd1c1d6368\") " pod="openshift-ingress/e2e-inf-b-9bfcac-openshift-default-676454d6b5-wh8mx" Oct 30 16:55:32.823357 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:32.823306 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/500d3721-818e-4ad7-b087-6dfd1c1d6368-istio-envoy\") pod \"e2e-inf-b-9bfcac-openshift-default-676454d6b5-wh8mx\" (UID: \"500d3721-818e-4ad7-b087-6dfd1c1d6368\") " pod="openshift-ingress/e2e-inf-b-9bfcac-openshift-default-676454d6b5-wh8mx" Oct 30 16:55:32.823357 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:32.823344 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6hj4\" (UniqueName: \"kubernetes.io/projected/500d3721-818e-4ad7-b087-6dfd1c1d6368-kube-api-access-p6hj4\") pod \"e2e-inf-b-9bfcac-openshift-default-676454d6b5-wh8mx\" (UID: \"500d3721-818e-4ad7-b087-6dfd1c1d6368\") " pod="openshift-ingress/e2e-inf-b-9bfcac-openshift-default-676454d6b5-wh8mx" Oct 30 16:55:32.823476 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:32.823458 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/500d3721-818e-4ad7-b087-6dfd1c1d6368-credential-socket\") pod \"e2e-inf-b-9bfcac-openshift-default-676454d6b5-wh8mx\" (UID: \"500d3721-818e-4ad7-b087-6dfd1c1d6368\") " pod="openshift-ingress/e2e-inf-b-9bfcac-openshift-default-676454d6b5-wh8mx" Oct 30 16:55:32.823572 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:32.823553 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/500d3721-818e-4ad7-b087-6dfd1c1d6368-istio-token\") pod \"e2e-inf-b-9bfcac-openshift-default-676454d6b5-wh8mx\" (UID: \"500d3721-818e-4ad7-b087-6dfd1c1d6368\") " pod="openshift-ingress/e2e-inf-b-9bfcac-openshift-default-676454d6b5-wh8mx" Oct 30 16:55:32.823710 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:32.823689 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/500d3721-818e-4ad7-b087-6dfd1c1d6368-istio-data\") pod \"e2e-inf-b-9bfcac-openshift-default-676454d6b5-wh8mx\" (UID: \"500d3721-818e-4ad7-b087-6dfd1c1d6368\") " pod="openshift-ingress/e2e-inf-b-9bfcac-openshift-default-676454d6b5-wh8mx" Oct 30 16:55:32.823760 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:32.823743 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/500d3721-818e-4ad7-b087-6dfd1c1d6368-istiod-ca-cert\") pod \"e2e-inf-b-9bfcac-openshift-default-676454d6b5-wh8mx\" (UID: \"500d3721-818e-4ad7-b087-6dfd1c1d6368\") " pod="openshift-ingress/e2e-inf-b-9bfcac-openshift-default-676454d6b5-wh8mx" Oct 30 16:55:32.925512 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:32.925477 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/500d3721-818e-4ad7-b087-6dfd1c1d6368-istio-data\") pod \"e2e-inf-b-9bfcac-openshift-default-676454d6b5-wh8mx\" (UID: \"500d3721-818e-4ad7-b087-6dfd1c1d6368\") " pod="openshift-ingress/e2e-inf-b-9bfcac-openshift-default-676454d6b5-wh8mx" Oct 30 16:55:32.925727 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:32.925527 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/500d3721-818e-4ad7-b087-6dfd1c1d6368-istiod-ca-cert\") pod \"e2e-inf-b-9bfcac-openshift-default-676454d6b5-wh8mx\" (UID: \"500d3721-818e-4ad7-b087-6dfd1c1d6368\") " pod="openshift-ingress/e2e-inf-b-9bfcac-openshift-default-676454d6b5-wh8mx" Oct 30 16:55:32.925727 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:32.925640 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/500d3721-818e-4ad7-b087-6dfd1c1d6368-workload-socket\") pod \"e2e-inf-b-9bfcac-openshift-default-676454d6b5-wh8mx\" (UID: \"500d3721-818e-4ad7-b087-6dfd1c1d6368\") " pod="openshift-ingress/e2e-inf-b-9bfcac-openshift-default-676454d6b5-wh8mx" Oct 30 16:55:32.925727 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:32.925698 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/500d3721-818e-4ad7-b087-6dfd1c1d6368-istio-podinfo\") pod \"e2e-inf-b-9bfcac-openshift-default-676454d6b5-wh8mx\" (UID: \"500d3721-818e-4ad7-b087-6dfd1c1d6368\") " pod="openshift-ingress/e2e-inf-b-9bfcac-openshift-default-676454d6b5-wh8mx" Oct 30 16:55:32.925727 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:32.925720 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/500d3721-818e-4ad7-b087-6dfd1c1d6368-workload-certs\") pod \"e2e-inf-b-9bfcac-openshift-default-676454d6b5-wh8mx\" (UID: \"500d3721-818e-4ad7-b087-6dfd1c1d6368\") " pod="openshift-ingress/e2e-inf-b-9bfcac-openshift-default-676454d6b5-wh8mx" Oct 30 16:55:32.925989 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:32.925758 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/500d3721-818e-4ad7-b087-6dfd1c1d6368-istio-envoy\") pod \"e2e-inf-b-9bfcac-openshift-default-676454d6b5-wh8mx\" (UID: \"500d3721-818e-4ad7-b087-6dfd1c1d6368\") " pod="openshift-ingress/e2e-inf-b-9bfcac-openshift-default-676454d6b5-wh8mx" Oct 30 16:55:32.925989 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:32.925942 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/500d3721-818e-4ad7-b087-6dfd1c1d6368-istio-data\") pod \"e2e-inf-b-9bfcac-openshift-default-676454d6b5-wh8mx\" (UID: \"500d3721-818e-4ad7-b087-6dfd1c1d6368\") " pod="openshift-ingress/e2e-inf-b-9bfcac-openshift-default-676454d6b5-wh8mx" Oct 30 16:55:32.926109 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:32.925997 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6hj4\" (UniqueName: \"kubernetes.io/projected/500d3721-818e-4ad7-b087-6dfd1c1d6368-kube-api-access-p6hj4\") pod \"e2e-inf-b-9bfcac-openshift-default-676454d6b5-wh8mx\" (UID: \"500d3721-818e-4ad7-b087-6dfd1c1d6368\") " pod="openshift-ingress/e2e-inf-b-9bfcac-openshift-default-676454d6b5-wh8mx" Oct 30 16:55:32.926109 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:32.926088 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/500d3721-818e-4ad7-b087-6dfd1c1d6368-credential-socket\") pod \"e2e-inf-b-9bfcac-openshift-default-676454d6b5-wh8mx\" (UID: \"500d3721-818e-4ad7-b087-6dfd1c1d6368\") " pod="openshift-ingress/e2e-inf-b-9bfcac-openshift-default-676454d6b5-wh8mx" Oct 30 16:55:32.926238 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:32.926183 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/500d3721-818e-4ad7-b087-6dfd1c1d6368-workload-certs\") pod \"e2e-inf-b-9bfcac-openshift-default-676454d6b5-wh8mx\" (UID: \"500d3721-818e-4ad7-b087-6dfd1c1d6368\") " pod="openshift-ingress/e2e-inf-b-9bfcac-openshift-default-676454d6b5-wh8mx" Oct 30 16:55:32.926447 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:32.926228 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/500d3721-818e-4ad7-b087-6dfd1c1d6368-istio-token\") pod \"e2e-inf-b-9bfcac-openshift-default-676454d6b5-wh8mx\" (UID: \"500d3721-818e-4ad7-b087-6dfd1c1d6368\") " pod="openshift-ingress/e2e-inf-b-9bfcac-openshift-default-676454d6b5-wh8mx" Oct 30 16:55:32.926447 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:32.926415 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/500d3721-818e-4ad7-b087-6dfd1c1d6368-workload-socket\") pod \"e2e-inf-b-9bfcac-openshift-default-676454d6b5-wh8mx\" (UID: \"500d3721-818e-4ad7-b087-6dfd1c1d6368\") " pod="openshift-ingress/e2e-inf-b-9bfcac-openshift-default-676454d6b5-wh8mx" Oct 30 16:55:32.926643 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:32.926441 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/500d3721-818e-4ad7-b087-6dfd1c1d6368-istiod-ca-cert\") pod \"e2e-inf-b-9bfcac-openshift-default-676454d6b5-wh8mx\" (UID: \"500d3721-818e-4ad7-b087-6dfd1c1d6368\") " pod="openshift-ingress/e2e-inf-b-9bfcac-openshift-default-676454d6b5-wh8mx" Oct 30 16:55:32.926643 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:32.926474 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/500d3721-818e-4ad7-b087-6dfd1c1d6368-credential-socket\") pod \"e2e-inf-b-9bfcac-openshift-default-676454d6b5-wh8mx\" (UID: \"500d3721-818e-4ad7-b087-6dfd1c1d6368\") " pod="openshift-ingress/e2e-inf-b-9bfcac-openshift-default-676454d6b5-wh8mx" Oct 30 16:55:32.928169 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:32.928139 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/500d3721-818e-4ad7-b087-6dfd1c1d6368-istio-envoy\") pod \"e2e-inf-b-9bfcac-openshift-default-676454d6b5-wh8mx\" (UID: \"500d3721-818e-4ad7-b087-6dfd1c1d6368\") " pod="openshift-ingress/e2e-inf-b-9bfcac-openshift-default-676454d6b5-wh8mx" Oct 30 16:55:32.928489 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:32.928473 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/500d3721-818e-4ad7-b087-6dfd1c1d6368-istio-podinfo\") pod \"e2e-inf-b-9bfcac-openshift-default-676454d6b5-wh8mx\" (UID: \"500d3721-818e-4ad7-b087-6dfd1c1d6368\") " pod="openshift-ingress/e2e-inf-b-9bfcac-openshift-default-676454d6b5-wh8mx" Oct 30 16:55:32.950344 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:32.950321 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/500d3721-818e-4ad7-b087-6dfd1c1d6368-istio-token\") pod \"e2e-inf-b-9bfcac-openshift-default-676454d6b5-wh8mx\" (UID: \"500d3721-818e-4ad7-b087-6dfd1c1d6368\") " pod="openshift-ingress/e2e-inf-b-9bfcac-openshift-default-676454d6b5-wh8mx" Oct 30 16:55:32.950710 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:32.950691 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6hj4\" (UniqueName: \"kubernetes.io/projected/500d3721-818e-4ad7-b087-6dfd1c1d6368-kube-api-access-p6hj4\") pod \"e2e-inf-b-9bfcac-openshift-default-676454d6b5-wh8mx\" (UID: \"500d3721-818e-4ad7-b087-6dfd1c1d6368\") " pod="openshift-ingress/e2e-inf-b-9bfcac-openshift-default-676454d6b5-wh8mx" Oct 30 16:55:33.100173 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:33.100097 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9bfcac-openshift-default-676454d6b5-wh8mx" Oct 30 16:55:33.541232 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:33.541200 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-9bfcac-openshift-default-676454d6b5-wh8mx"] Oct 30 16:55:33.541705 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:55:33.541674 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod500d3721_818e_4ad7_b087_6dfd1c1d6368.slice/crio-6cbba401ecfe585bbcd13ab20b150469c626ae20c3c0d6b8090b507e2c05200b WatchSource:0}: Error finding container 6cbba401ecfe585bbcd13ab20b150469c626ae20c3c0d6b8090b507e2c05200b: Status 404 returned error can't find the container with id 6cbba401ecfe585bbcd13ab20b150469c626ae20c3c0d6b8090b507e2c05200b Oct 30 16:55:33.545265 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:33.545229 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:55:33.545378 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:33.545301 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:55:33.545378 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:33.545337 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:55:33.833700 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:33.830868 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9bfcac-openshift-default-676454d6b5-wh8mx" event={"ID":"500d3721-818e-4ad7-b087-6dfd1c1d6368","Type":"ContainerStarted","Data":"099c1797818bf395a22a04383d5563beb9fff9169289d825621339785aa8ceec"} Oct 30 16:55:33.833700 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:33.830935 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9bfcac-openshift-default-676454d6b5-wh8mx" event={"ID":"500d3721-818e-4ad7-b087-6dfd1c1d6368","Type":"ContainerStarted","Data":"6cbba401ecfe585bbcd13ab20b150469c626ae20c3c0d6b8090b507e2c05200b"} Oct 30 16:55:33.890870 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:33.890797 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-9bfcac-openshift-default-676454d6b5-wh8mx" podStartSLOduration=1.89077902 podStartE2EDuration="1.89077902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:55:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:55:33.880642415 +0000 UTC m=+2089.407977636" watchObservedRunningTime="2026-10-30 16:55:33.89077902 +0000 UTC m=+2089.418114240" Oct 30 16:55:34.100453 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:34.100373 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-9bfcac-openshift-default-676454d6b5-wh8mx" Oct 30 16:55:34.102421 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:34.102394 5052 patch_prober.go:29] interesting pod/e2e-inf-b-9bfcac-openshift-default-676454d6b5-wh8mx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Oct 30 16:55:34.102533 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:34.102451 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9bfcac-openshift-default-676454d6b5-wh8mx" podUID="500d3721-818e-4ad7-b087-6dfd1c1d6368" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Oct 30 16:55:35.108657 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:35.100995 5052 patch_prober.go:29] interesting pod/e2e-inf-b-9bfcac-openshift-default-676454d6b5-wh8mx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Oct 30 16:55:35.108657 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:35.101061 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9bfcac-openshift-default-676454d6b5-wh8mx" podUID="500d3721-818e-4ad7-b087-6dfd1c1d6368" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Oct 30 16:55:36.100698 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:36.100659 5052 patch_prober.go:29] interesting pod/e2e-inf-b-9bfcac-openshift-default-676454d6b5-wh8mx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Oct 30 16:55:36.100891 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:36.100717 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9bfcac-openshift-default-676454d6b5-wh8mx" podUID="500d3721-818e-4ad7-b087-6dfd1c1d6368" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Oct 30 16:55:37.101337 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:37.101304 5052 patch_prober.go:29] interesting pod/e2e-inf-b-9bfcac-openshift-default-676454d6b5-wh8mx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Oct 30 16:55:37.101826 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:37.101364 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9bfcac-openshift-default-676454d6b5-wh8mx" podUID="500d3721-818e-4ad7-b087-6dfd1c1d6368" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Oct 30 16:55:38.110325 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:38.110286 5052 patch_prober.go:29] interesting pod/e2e-inf-b-9bfcac-openshift-default-676454d6b5-wh8mx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Oct 30 16:55:38.110852 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:38.110348 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9bfcac-openshift-default-676454d6b5-wh8mx" podUID="500d3721-818e-4ad7-b087-6dfd1c1d6368" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Oct 30 16:55:38.557296 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:38.557246 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-a59502/test-model-a59502-kserve-798b8c7859-5kptw"] Oct 30 16:55:38.569795 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:38.569772 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-a59502/test-model-a59502-kserve-798b8c7859-5kptw"] Oct 30 16:55:38.569971 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:38.569925 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a59502/test-model-a59502-kserve-798b8c7859-5kptw" Oct 30 16:55:38.598969 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:38.598942 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-a59502\"/\"default-dockercfg-jgmz6\"" Oct 30 16:55:38.599781 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:38.599755 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-a59502\"/\"kube-root-ca.crt\"" Oct 30 16:55:38.599781 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:38.599760 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-a59502\"/\"openshift-service-ca.crt\"" Oct 30 16:55:38.599928 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:38.599888 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-a59502\"/\"test-model-a59502-kserve-self-signed-certs\"" Oct 30 16:55:38.707246 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:38.707212 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2915da3-9830-4751-b58f-837063130fa1-tls-certs\") pod \"test-model-a59502-kserve-798b8c7859-5kptw\" (UID: \"c2915da3-9830-4751-b58f-837063130fa1\") " pod="ai-tenant-e2e-inf-a-a59502/test-model-a59502-kserve-798b8c7859-5kptw" Oct 30 16:55:38.707626 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:38.707556 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2915da3-9830-4751-b58f-837063130fa1-model-cache\") pod \"test-model-a59502-kserve-798b8c7859-5kptw\" (UID: \"c2915da3-9830-4751-b58f-837063130fa1\") " pod="ai-tenant-e2e-inf-a-a59502/test-model-a59502-kserve-798b8c7859-5kptw" Oct 30 16:55:38.707696 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:38.707677 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2915da3-9830-4751-b58f-837063130fa1-home\") pod \"test-model-a59502-kserve-798b8c7859-5kptw\" (UID: \"c2915da3-9830-4751-b58f-837063130fa1\") " pod="ai-tenant-e2e-inf-a-a59502/test-model-a59502-kserve-798b8c7859-5kptw" Oct 30 16:55:38.707735 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:38.707725 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2915da3-9830-4751-b58f-837063130fa1-tmp-dir\") pod \"test-model-a59502-kserve-798b8c7859-5kptw\" (UID: \"c2915da3-9830-4751-b58f-837063130fa1\") " pod="ai-tenant-e2e-inf-a-a59502/test-model-a59502-kserve-798b8c7859-5kptw" Oct 30 16:55:38.707795 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:38.707780 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2915da3-9830-4751-b58f-837063130fa1-dshm\") pod \"test-model-a59502-kserve-798b8c7859-5kptw\" (UID: \"c2915da3-9830-4751-b58f-837063130fa1\") " pod="ai-tenant-e2e-inf-a-a59502/test-model-a59502-kserve-798b8c7859-5kptw" Oct 30 16:55:38.707862 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:38.707848 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsnfg\" (UniqueName: \"kubernetes.io/projected/c2915da3-9830-4751-b58f-837063130fa1-kube-api-access-gsnfg\") pod \"test-model-a59502-kserve-798b8c7859-5kptw\" (UID: \"c2915da3-9830-4751-b58f-837063130fa1\") " pod="ai-tenant-e2e-inf-a-a59502/test-model-a59502-kserve-798b8c7859-5kptw" Oct 30 16:55:38.818659 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:38.818105 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2915da3-9830-4751-b58f-837063130fa1-model-cache\") pod \"test-model-a59502-kserve-798b8c7859-5kptw\" (UID: \"c2915da3-9830-4751-b58f-837063130fa1\") " pod="ai-tenant-e2e-inf-a-a59502/test-model-a59502-kserve-798b8c7859-5kptw" Oct 30 16:55:38.818659 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:38.818197 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2915da3-9830-4751-b58f-837063130fa1-home\") pod \"test-model-a59502-kserve-798b8c7859-5kptw\" (UID: \"c2915da3-9830-4751-b58f-837063130fa1\") " pod="ai-tenant-e2e-inf-a-a59502/test-model-a59502-kserve-798b8c7859-5kptw" Oct 30 16:55:38.818659 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:38.818255 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2915da3-9830-4751-b58f-837063130fa1-tmp-dir\") pod \"test-model-a59502-kserve-798b8c7859-5kptw\" (UID: \"c2915da3-9830-4751-b58f-837063130fa1\") " pod="ai-tenant-e2e-inf-a-a59502/test-model-a59502-kserve-798b8c7859-5kptw" Oct 30 16:55:38.818659 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:38.818289 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2915da3-9830-4751-b58f-837063130fa1-dshm\") pod \"test-model-a59502-kserve-798b8c7859-5kptw\" (UID: \"c2915da3-9830-4751-b58f-837063130fa1\") " pod="ai-tenant-e2e-inf-a-a59502/test-model-a59502-kserve-798b8c7859-5kptw" Oct 30 16:55:38.818659 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:38.818404 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsnfg\" (UniqueName: \"kubernetes.io/projected/c2915da3-9830-4751-b58f-837063130fa1-kube-api-access-gsnfg\") pod \"test-model-a59502-kserve-798b8c7859-5kptw\" (UID: \"c2915da3-9830-4751-b58f-837063130fa1\") " pod="ai-tenant-e2e-inf-a-a59502/test-model-a59502-kserve-798b8c7859-5kptw" Oct 30 16:55:38.818659 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:38.818507 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2915da3-9830-4751-b58f-837063130fa1-tls-certs\") pod \"test-model-a59502-kserve-798b8c7859-5kptw\" (UID: \"c2915da3-9830-4751-b58f-837063130fa1\") " pod="ai-tenant-e2e-inf-a-a59502/test-model-a59502-kserve-798b8c7859-5kptw" Oct 30 16:55:38.822611 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:38.819818 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2915da3-9830-4751-b58f-837063130fa1-tmp-dir\") pod \"test-model-a59502-kserve-798b8c7859-5kptw\" (UID: \"c2915da3-9830-4751-b58f-837063130fa1\") " pod="ai-tenant-e2e-inf-a-a59502/test-model-a59502-kserve-798b8c7859-5kptw" Oct 30 16:55:38.822611 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:38.820082 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2915da3-9830-4751-b58f-837063130fa1-model-cache\") pod \"test-model-a59502-kserve-798b8c7859-5kptw\" (UID: \"c2915da3-9830-4751-b58f-837063130fa1\") " pod="ai-tenant-e2e-inf-a-a59502/test-model-a59502-kserve-798b8c7859-5kptw" Oct 30 16:55:38.822611 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:38.820321 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2915da3-9830-4751-b58f-837063130fa1-home\") pod \"test-model-a59502-kserve-798b8c7859-5kptw\" (UID: \"c2915da3-9830-4751-b58f-837063130fa1\") " pod="ai-tenant-e2e-inf-a-a59502/test-model-a59502-kserve-798b8c7859-5kptw" Oct 30 16:55:38.834316 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:38.834282 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2915da3-9830-4751-b58f-837063130fa1-tls-certs\") pod \"test-model-a59502-kserve-798b8c7859-5kptw\" (UID: \"c2915da3-9830-4751-b58f-837063130fa1\") " pod="ai-tenant-e2e-inf-a-a59502/test-model-a59502-kserve-798b8c7859-5kptw" Oct 30 16:55:38.897603 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:38.892468 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2915da3-9830-4751-b58f-837063130fa1-dshm\") pod \"test-model-a59502-kserve-798b8c7859-5kptw\" (UID: \"c2915da3-9830-4751-b58f-837063130fa1\") " pod="ai-tenant-e2e-inf-a-a59502/test-model-a59502-kserve-798b8c7859-5kptw" Oct 30 16:55:38.927769 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:38.927480 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsnfg\" (UniqueName: \"kubernetes.io/projected/c2915da3-9830-4751-b58f-837063130fa1-kube-api-access-gsnfg\") pod \"test-model-a59502-kserve-798b8c7859-5kptw\" (UID: \"c2915da3-9830-4751-b58f-837063130fa1\") " pod="ai-tenant-e2e-inf-a-a59502/test-model-a59502-kserve-798b8c7859-5kptw" Oct 30 16:55:39.101173 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:39.101094 5052 patch_prober.go:29] interesting pod/e2e-inf-b-9bfcac-openshift-default-676454d6b5-wh8mx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Oct 30 16:55:39.101173 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:39.101153 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9bfcac-openshift-default-676454d6b5-wh8mx" podUID="500d3721-818e-4ad7-b087-6dfd1c1d6368" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Oct 30 16:55:39.193617 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:39.192973 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a59502/test-model-a59502-kserve-798b8c7859-5kptw" Oct 30 16:55:39.574462 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:39.574429 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-a59502/test-model-a59502-kserve-798b8c7859-5kptw"] Oct 30 16:55:39.574792 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:55:39.574770 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc2915da3_9830_4751_b58f_837063130fa1.slice/crio-a6518d714ff909bc376b93e79bba17751e040fa195f45b4c0598b1b080a59284 WatchSource:0}: Error finding container a6518d714ff909bc376b93e79bba17751e040fa195f45b4c0598b1b080a59284: Status 404 returned error can't find the container with id a6518d714ff909bc376b93e79bba17751e040fa195f45b4c0598b1b080a59284 Oct 30 16:55:39.975328 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:39.975291 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a59502/test-model-a59502-kserve-798b8c7859-5kptw" event={"ID":"c2915da3-9830-4751-b58f-837063130fa1","Type":"ContainerStarted","Data":"f371d71aca03863c6c26020f9da6660406f29ea1434a5941db18646238321ccd"} Oct 30 16:55:39.975542 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:39.975346 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a59502/test-model-a59502-kserve-798b8c7859-5kptw" event={"ID":"c2915da3-9830-4751-b58f-837063130fa1","Type":"ContainerStarted","Data":"a6518d714ff909bc376b93e79bba17751e040fa195f45b4c0598b1b080a59284"} Oct 30 16:55:40.026738 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:40.026681 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-a59502/test-model-a59502-kserve-798b8c7859-5kptw" podStartSLOduration=2.026663843 podStartE2EDuration="2.026663843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:55:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:55:40.023179801 +0000 UTC m=+2095.550515015" watchObservedRunningTime="2026-10-30 16:55:40.026663843 +0000 UTC m=+2095.553999063" Oct 30 16:55:40.100878 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:40.100841 5052 patch_prober.go:29] interesting pod/e2e-inf-b-9bfcac-openshift-default-676454d6b5-wh8mx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Oct 30 16:55:40.101052 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:40.100895 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9bfcac-openshift-default-676454d6b5-wh8mx" podUID="500d3721-818e-4ad7-b087-6dfd1c1d6368" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Oct 30 16:55:40.210606 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:40.204542 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-a59502/test-model-a59502-kserve-798b8c7859-5kptw" Oct 30 16:55:41.106362 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:41.106324 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-9bfcac-openshift-default-676454d6b5-wh8mx" Oct 30 16:55:41.106624 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:41.106605 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-9bfcac-openshift-default-676454d6b5-wh8mx" Oct 30 16:55:41.107367 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:41.107351 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-9bfcac-openshift-default-676454d6b5-wh8mx" Oct 30 16:55:45.470056 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:45.469982 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4wrf8_4f0f46f4-5cda-4d67-b6f3-47df06eb844c/console-operator/2.log" Oct 30 16:55:45.471068 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:45.471045 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4wrf8_4f0f46f4-5cda-4d67-b6f3-47df06eb844c/console-operator/2.log" Oct 30 16:55:46.149131 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:46.149098 5052 scope.go:122] "RemoveContainer" containerID="4112f8da7f8b4120b5371d0eb443db95af671acdd97f308e429d186d604463c8" Oct 30 16:55:48.403056 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:48.403023 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-8w9ml"] Oct 30 16:55:48.410330 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:48.410302 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-8w9ml" Oct 30 16:55:48.432615 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:48.432554 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-8w9ml"] Oct 30 16:55:48.554754 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:48.554717 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ceae9daf-1d9c-4510-bb01-b001d2fc9151-workload-certs\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-8w9ml\" (UID: \"ceae9daf-1d9c-4510-bb01-b001d2fc9151\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-8w9ml" Oct 30 16:55:48.554923 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:48.554777 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ceae9daf-1d9c-4510-bb01-b001d2fc9151-istio-data\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-8w9ml\" (UID: \"ceae9daf-1d9c-4510-bb01-b001d2fc9151\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-8w9ml" Oct 30 16:55:48.554923 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:48.554907 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceae9daf-1d9c-4510-bb01-b001d2fc9151-credential-socket\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-8w9ml\" (UID: \"ceae9daf-1d9c-4510-bb01-b001d2fc9151\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-8w9ml" Oct 30 16:55:48.555004 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:48.554941 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5lxl\" (UniqueName: \"kubernetes.io/projected/ceae9daf-1d9c-4510-bb01-b001d2fc9151-kube-api-access-f5lxl\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-8w9ml\" (UID: \"ceae9daf-1d9c-4510-bb01-b001d2fc9151\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-8w9ml" Oct 30 16:55:48.555092 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:48.555076 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceae9daf-1d9c-4510-bb01-b001d2fc9151-workload-socket\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-8w9ml\" (UID: \"ceae9daf-1d9c-4510-bb01-b001d2fc9151\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-8w9ml" Oct 30 16:55:48.555129 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:48.555117 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ceae9daf-1d9c-4510-bb01-b001d2fc9151-istio-podinfo\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-8w9ml\" (UID: \"ceae9daf-1d9c-4510-bb01-b001d2fc9151\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-8w9ml" Oct 30 16:55:48.555165 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:48.555147 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ceae9daf-1d9c-4510-bb01-b001d2fc9151-istio-envoy\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-8w9ml\" (UID: \"ceae9daf-1d9c-4510-bb01-b001d2fc9151\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-8w9ml" Oct 30 16:55:48.555288 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:48.555274 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ceae9daf-1d9c-4510-bb01-b001d2fc9151-istiod-ca-cert\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-8w9ml\" (UID: \"ceae9daf-1d9c-4510-bb01-b001d2fc9151\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-8w9ml" Oct 30 16:55:48.555363 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:48.555346 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ceae9daf-1d9c-4510-bb01-b001d2fc9151-istio-token\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-8w9ml\" (UID: \"ceae9daf-1d9c-4510-bb01-b001d2fc9151\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-8w9ml" Oct 30 16:55:48.657074 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:48.656983 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ceae9daf-1d9c-4510-bb01-b001d2fc9151-istio-data\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-8w9ml\" (UID: \"ceae9daf-1d9c-4510-bb01-b001d2fc9151\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-8w9ml" Oct 30 16:55:48.657233 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:48.657095 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceae9daf-1d9c-4510-bb01-b001d2fc9151-credential-socket\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-8w9ml\" (UID: \"ceae9daf-1d9c-4510-bb01-b001d2fc9151\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-8w9ml" Oct 30 16:55:48.657233 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:48.657131 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5lxl\" (UniqueName: \"kubernetes.io/projected/ceae9daf-1d9c-4510-bb01-b001d2fc9151-kube-api-access-f5lxl\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-8w9ml\" (UID: \"ceae9daf-1d9c-4510-bb01-b001d2fc9151\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-8w9ml" Oct 30 16:55:48.657355 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:48.657252 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceae9daf-1d9c-4510-bb01-b001d2fc9151-workload-socket\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-8w9ml\" (UID: \"ceae9daf-1d9c-4510-bb01-b001d2fc9151\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-8w9ml" Oct 30 16:55:48.657355 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:48.657305 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ceae9daf-1d9c-4510-bb01-b001d2fc9151-istio-podinfo\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-8w9ml\" (UID: \"ceae9daf-1d9c-4510-bb01-b001d2fc9151\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-8w9ml" Oct 30 16:55:48.657355 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:48.657330 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ceae9daf-1d9c-4510-bb01-b001d2fc9151-istio-envoy\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-8w9ml\" (UID: \"ceae9daf-1d9c-4510-bb01-b001d2fc9151\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-8w9ml" Oct 30 16:55:48.657520 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:48.657444 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ceae9daf-1d9c-4510-bb01-b001d2fc9151-istio-data\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-8w9ml\" (UID: \"ceae9daf-1d9c-4510-bb01-b001d2fc9151\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-8w9ml" Oct 30 16:55:48.657520 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:48.657460 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceae9daf-1d9c-4510-bb01-b001d2fc9151-credential-socket\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-8w9ml\" (UID: \"ceae9daf-1d9c-4510-bb01-b001d2fc9151\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-8w9ml" Oct 30 16:55:48.657520 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:48.657477 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ceae9daf-1d9c-4510-bb01-b001d2fc9151-istiod-ca-cert\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-8w9ml\" (UID: \"ceae9daf-1d9c-4510-bb01-b001d2fc9151\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-8w9ml" Oct 30 16:55:48.657711 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:48.657543 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ceae9daf-1d9c-4510-bb01-b001d2fc9151-istio-token\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-8w9ml\" (UID: \"ceae9daf-1d9c-4510-bb01-b001d2fc9151\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-8w9ml" Oct 30 16:55:48.657711 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:48.657631 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ceae9daf-1d9c-4510-bb01-b001d2fc9151-workload-certs\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-8w9ml\" (UID: \"ceae9daf-1d9c-4510-bb01-b001d2fc9151\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-8w9ml" Oct 30 16:55:48.657827 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:48.657809 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceae9daf-1d9c-4510-bb01-b001d2fc9151-workload-socket\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-8w9ml\" (UID: \"ceae9daf-1d9c-4510-bb01-b001d2fc9151\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-8w9ml" Oct 30 16:55:48.658041 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:48.658015 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ceae9daf-1d9c-4510-bb01-b001d2fc9151-workload-certs\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-8w9ml\" (UID: \"ceae9daf-1d9c-4510-bb01-b001d2fc9151\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-8w9ml" Oct 30 16:55:48.658315 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:48.658293 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ceae9daf-1d9c-4510-bb01-b001d2fc9151-istiod-ca-cert\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-8w9ml\" (UID: \"ceae9daf-1d9c-4510-bb01-b001d2fc9151\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-8w9ml" Oct 30 16:55:48.659706 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:48.659682 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ceae9daf-1d9c-4510-bb01-b001d2fc9151-istio-envoy\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-8w9ml\" (UID: \"ceae9daf-1d9c-4510-bb01-b001d2fc9151\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-8w9ml" Oct 30 16:55:48.660349 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:48.660332 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ceae9daf-1d9c-4510-bb01-b001d2fc9151-istio-podinfo\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-8w9ml\" (UID: \"ceae9daf-1d9c-4510-bb01-b001d2fc9151\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-8w9ml" Oct 30 16:55:48.681539 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:48.681513 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5lxl\" (UniqueName: \"kubernetes.io/projected/ceae9daf-1d9c-4510-bb01-b001d2fc9151-kube-api-access-f5lxl\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-8w9ml\" (UID: \"ceae9daf-1d9c-4510-bb01-b001d2fc9151\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-8w9ml" Oct 30 16:55:48.682483 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:48.682462 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ceae9daf-1d9c-4510-bb01-b001d2fc9151-istio-token\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-8w9ml\" (UID: \"ceae9daf-1d9c-4510-bb01-b001d2fc9151\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-8w9ml" Oct 30 16:55:48.724370 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:48.724339 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-529bc7-6bb85cdff-prc7w"] Oct 30 16:55:48.724569 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:48.724530 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-529bc7-6bb85cdff-prc7w" podUID="fdece98a-1500-46d3-96cb-ab50d9dd2068" containerName="maas-api" containerID="cri-o://60134c0f5fee1a88aea5f502ba106601381f01c6242ce1541cddc3247be3e0be" gracePeriod=30 Oct 30 16:55:48.738202 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:48.737942 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-8w9ml" Oct 30 16:55:48.869829 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:55:48.869271 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 16:55:49.223603 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:49.208643 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-a59502/test-model-a59502-kserve-798b8c7859-5kptw" Oct 30 16:55:49.378863 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:49.368877 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:55:49.378863 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:49.369032 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:55:49.378863 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:49.369109 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:55:49.468604 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:49.463510 5052 generic.go:356] "Generic (PLEG): container finished" podID="fdece98a-1500-46d3-96cb-ab50d9dd2068" containerID="60134c0f5fee1a88aea5f502ba106601381f01c6242ce1541cddc3247be3e0be" exitCode=0 Oct 30 16:55:49.468604 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:49.463706 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-529bc7-6bb85cdff-prc7w" event={"ID":"fdece98a-1500-46d3-96cb-ab50d9dd2068","Type":"ContainerDied","Data":"60134c0f5fee1a88aea5f502ba106601381f01c6242ce1541cddc3247be3e0be"} Oct 30 16:55:49.493303 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:49.486549 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-a59502/test-model-a59502-kserve-798b8c7859-5kptw" Oct 30 16:55:49.493303 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:49.491723 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-8w9ml"] Oct 30 16:55:49.516672 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:49.516648 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-529bc7-6bb85cdff-prc7w" Oct 30 16:55:49.538412 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:49.538381 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fdece98a-1500-46d3-96cb-ab50d9dd2068-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fdece98a-1500-46d3-96cb-ab50d9dd2068-maas-api-tls\") pod \"fdece98a-1500-46d3-96cb-ab50d9dd2068\" (UID: \"fdece98a-1500-46d3-96cb-ab50d9dd2068\") " Oct 30 16:55:49.538571 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:49.538514 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fdece98a-1500-46d3-96cb-ab50d9dd2068-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdece98a-1500-46d3-96cb-ab50d9dd2068-metrics-certs\") pod \"fdece98a-1500-46d3-96cb-ab50d9dd2068\" (UID: \"fdece98a-1500-46d3-96cb-ab50d9dd2068\") " Oct 30 16:55:49.538681 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:49.538642 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdece98a-1500-46d3-96cb-ab50d9dd2068-kube-api-access-ncwjk\" (UniqueName: \"kubernetes.io/projected/fdece98a-1500-46d3-96cb-ab50d9dd2068-kube-api-access-ncwjk\") pod \"fdece98a-1500-46d3-96cb-ab50d9dd2068\" (UID: \"fdece98a-1500-46d3-96cb-ab50d9dd2068\") " Oct 30 16:55:49.538881 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:49.538855 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdece98a-1500-46d3-96cb-ab50d9dd2068-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdece98a-1500-46d3-96cb-ab50d9dd2068-openshift-service-ca\") pod \"fdece98a-1500-46d3-96cb-ab50d9dd2068\" (UID: \"fdece98a-1500-46d3-96cb-ab50d9dd2068\") " Oct 30 16:55:49.539204 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:49.539182 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdece98a-1500-46d3-96cb-ab50d9dd2068-openshift-service-ca" pod "fdece98a-1500-46d3-96cb-ab50d9dd2068" (UID: "fdece98a-1500-46d3-96cb-ab50d9dd2068"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:55:49.539636 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:49.539617 5052 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdece98a-1500-46d3-96cb-ab50d9dd2068-openshift-service-ca\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:55:49.541049 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:49.540998 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdece98a-1500-46d3-96cb-ab50d9dd2068-maas-api-tls" pod "fdece98a-1500-46d3-96cb-ab50d9dd2068" (UID: "fdece98a-1500-46d3-96cb-ab50d9dd2068"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:55:49.541838 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:49.541807 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdece98a-1500-46d3-96cb-ab50d9dd2068-metrics-certs" pod "fdece98a-1500-46d3-96cb-ab50d9dd2068" (UID: "fdece98a-1500-46d3-96cb-ab50d9dd2068"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:55:49.542692 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:49.542665 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdece98a-1500-46d3-96cb-ab50d9dd2068-kube-api-access-ncwjk" pod "fdece98a-1500-46d3-96cb-ab50d9dd2068" (UID: "fdece98a-1500-46d3-96cb-ab50d9dd2068"). InnerVolumeSpecName "kube-api-access-ncwjk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:55:49.642217 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:49.642038 5052 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fdece98a-1500-46d3-96cb-ab50d9dd2068-maas-api-tls\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:55:49.642217 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:49.642072 5052 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdece98a-1500-46d3-96cb-ab50d9dd2068-metrics-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:55:49.642217 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:49.642086 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncwjk\" (UniqueName: \"kubernetes.io/projected/fdece98a-1500-46d3-96cb-ab50d9dd2068-kube-api-access-ncwjk\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:55:50.217957 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:50.217929 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-a59502/test-model-a59502-kserve-798b8c7859-5kptw" Oct 30 16:55:50.473023 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:50.472920 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-8w9ml" event={"ID":"ceae9daf-1d9c-4510-bb01-b001d2fc9151","Type":"ContainerStarted","Data":"74784cc27ad61fd80786a90a94d39f17556f1e7993f452deb76ff87492c70380"} Oct 30 16:55:50.473023 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:50.472988 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-8w9ml" event={"ID":"ceae9daf-1d9c-4510-bb01-b001d2fc9151","Type":"ContainerStarted","Data":"3fa6e719c2eacc351d1fd8edad06dbe14dd933767d314dca2cf2146f71655854"} Oct 30 16:55:50.474447 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:50.474427 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-529bc7-6bb85cdff-prc7w" Oct 30 16:55:50.474562 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:50.474474 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-529bc7-6bb85cdff-prc7w" event={"ID":"fdece98a-1500-46d3-96cb-ab50d9dd2068","Type":"ContainerDied","Data":"7e35e50bc6ce9913ba8be02f44e6970ec70f9bffc4f205d8f5406c99e327da53"} Oct 30 16:55:50.474562 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:50.474522 5052 scope.go:122] "RemoveContainer" containerID="60134c0f5fee1a88aea5f502ba106601381f01c6242ce1541cddc3247be3e0be" Oct 30 16:55:50.521861 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:50.521804 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-8w9ml" podStartSLOduration=2.521786416 podStartE2EDuration="2.521786416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:55:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:55:50.519646979 +0000 UTC m=+2106.046982195" watchObservedRunningTime="2026-10-30 16:55:50.521786416 +0000 UTC m=+2106.049121650" Oct 30 16:55:50.573979 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:50.573945 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-529bc7-6bb85cdff-prc7w"] Oct 30 16:55:50.578137 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:50.578110 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-529bc7-6bb85cdff-prc7w"] Oct 30 16:55:50.599803 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:50.599775 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdece98a-1500-46d3-96cb-ab50d9dd2068" path="/var/lib/kubelet/pods/fdece98a-1500-46d3-96cb-ab50d9dd2068/volumes" Oct 30 16:55:50.745896 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:50.739026 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-8w9ml" Oct 30 16:55:50.746235 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:50.746205 5052 patch_prober.go:29] interesting pod/e2e-inf-b-689899-openshift-default-66cbd45f99-8w9ml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Oct 30 16:55:50.746420 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:50.746393 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-8w9ml" podUID="ceae9daf-1d9c-4510-bb01-b001d2fc9151" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Oct 30 16:55:51.745384 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:51.744929 5052 patch_prober.go:29] interesting pod/e2e-inf-b-689899-openshift-default-66cbd45f99-8w9ml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Oct 30 16:55:51.745384 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:51.744995 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-8w9ml" podUID="ceae9daf-1d9c-4510-bb01-b001d2fc9151" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Oct 30 16:55:52.749383 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:52.748932 5052 patch_prober.go:29] interesting pod/e2e-inf-b-689899-openshift-default-66cbd45f99-8w9ml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Oct 30 16:55:52.749383 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:52.749008 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-8w9ml" podUID="ceae9daf-1d9c-4510-bb01-b001d2fc9151" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Oct 30 16:55:53.749972 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:53.749933 5052 patch_prober.go:29] interesting pod/e2e-inf-b-689899-openshift-default-66cbd45f99-8w9ml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Oct 30 16:55:53.750879 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:53.750844 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-8w9ml" podUID="ceae9daf-1d9c-4510-bb01-b001d2fc9151" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Oct 30 16:55:54.742389 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:54.741928 5052 patch_prober.go:29] interesting pod/e2e-inf-b-689899-openshift-default-66cbd45f99-8w9ml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Oct 30 16:55:54.742389 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:54.741987 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-8w9ml" podUID="ceae9daf-1d9c-4510-bb01-b001d2fc9151" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Oct 30 16:55:55.741626 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:55.741574 5052 patch_prober.go:29] interesting pod/e2e-inf-b-689899-openshift-default-66cbd45f99-8w9ml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Oct 30 16:55:55.742228 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:55.742180 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-8w9ml" podUID="ceae9daf-1d9c-4510-bb01-b001d2fc9151" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Oct 30 16:55:55.939319 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:55.939191 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-9c1e26/test-model-9c1e26-kserve-7cc944cb5f-xjljc"] Oct 30 16:55:55.940748 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:55.940721 5052 cpu_manager.go:414] "containerMap: removing container" podUID="fdece98a-1500-46d3-96cb-ab50d9dd2068" containerName="maas-api" Oct 30 16:55:55.940912 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:55.940897 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdece98a-1500-46d3-96cb-ab50d9dd2068" containerName="maas-api" Oct 30 16:55:55.941236 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:55.941223 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdece98a-1500-46d3-96cb-ab50d9dd2068" containerName="maas-api" Oct 30 16:55:55.946075 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:55.946056 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9c1e26/test-model-9c1e26-kserve-7cc944cb5f-xjljc" Oct 30 16:55:55.964759 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:55.964732 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-9c1e26\"/\"openshift-service-ca.crt\"" Oct 30 16:55:55.964941 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:55.964765 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-9c1e26\"/\"kube-root-ca.crt\"" Oct 30 16:55:55.967085 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:55.967062 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-9c1e26/test-model-9c1e26-kserve-7cc944cb5f-xjljc"] Oct 30 16:55:55.978877 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:55.978850 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-9c1e26\"/\"test-model-9c1e26-kserve-self-signed-certs\"" Oct 30 16:55:55.979060 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:55.978850 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-9c1e26\"/\"default-dockercfg-9prrh\"" Oct 30 16:55:56.113230 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:56.113145 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17327d7e-8d87-4461-be7f-1ed4c1d17701-home\") pod \"test-model-9c1e26-kserve-7cc944cb5f-xjljc\" (UID: \"17327d7e-8d87-4461-be7f-1ed4c1d17701\") " pod="ai-tenant-e2e-inf-b-9c1e26/test-model-9c1e26-kserve-7cc944cb5f-xjljc" Oct 30 16:55:56.113414 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:56.113334 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17327d7e-8d87-4461-be7f-1ed4c1d17701-dshm\") pod \"test-model-9c1e26-kserve-7cc944cb5f-xjljc\" (UID: \"17327d7e-8d87-4461-be7f-1ed4c1d17701\") " pod="ai-tenant-e2e-inf-b-9c1e26/test-model-9c1e26-kserve-7cc944cb5f-xjljc" Oct 30 16:55:56.113414 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:56.113366 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17327d7e-8d87-4461-be7f-1ed4c1d17701-tmp-dir\") pod \"test-model-9c1e26-kserve-7cc944cb5f-xjljc\" (UID: \"17327d7e-8d87-4461-be7f-1ed4c1d17701\") " pod="ai-tenant-e2e-inf-b-9c1e26/test-model-9c1e26-kserve-7cc944cb5f-xjljc" Oct 30 16:55:56.113536 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:56.113455 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17327d7e-8d87-4461-be7f-1ed4c1d17701-tls-certs\") pod \"test-model-9c1e26-kserve-7cc944cb5f-xjljc\" (UID: \"17327d7e-8d87-4461-be7f-1ed4c1d17701\") " pod="ai-tenant-e2e-inf-b-9c1e26/test-model-9c1e26-kserve-7cc944cb5f-xjljc" Oct 30 16:55:56.113536 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:56.113488 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzrk4\" (UniqueName: \"kubernetes.io/projected/17327d7e-8d87-4461-be7f-1ed4c1d17701-kube-api-access-zzrk4\") pod \"test-model-9c1e26-kserve-7cc944cb5f-xjljc\" (UID: \"17327d7e-8d87-4461-be7f-1ed4c1d17701\") " pod="ai-tenant-e2e-inf-b-9c1e26/test-model-9c1e26-kserve-7cc944cb5f-xjljc" Oct 30 16:55:56.113722 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:56.113559 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17327d7e-8d87-4461-be7f-1ed4c1d17701-model-cache\") pod \"test-model-9c1e26-kserve-7cc944cb5f-xjljc\" (UID: \"17327d7e-8d87-4461-be7f-1ed4c1d17701\") " pod="ai-tenant-e2e-inf-b-9c1e26/test-model-9c1e26-kserve-7cc944cb5f-xjljc" Oct 30 16:55:56.221604 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:56.220856 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17327d7e-8d87-4461-be7f-1ed4c1d17701-dshm\") pod \"test-model-9c1e26-kserve-7cc944cb5f-xjljc\" (UID: \"17327d7e-8d87-4461-be7f-1ed4c1d17701\") " pod="ai-tenant-e2e-inf-b-9c1e26/test-model-9c1e26-kserve-7cc944cb5f-xjljc" Oct 30 16:55:56.221604 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:56.220921 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17327d7e-8d87-4461-be7f-1ed4c1d17701-tmp-dir\") pod \"test-model-9c1e26-kserve-7cc944cb5f-xjljc\" (UID: \"17327d7e-8d87-4461-be7f-1ed4c1d17701\") " pod="ai-tenant-e2e-inf-b-9c1e26/test-model-9c1e26-kserve-7cc944cb5f-xjljc" Oct 30 16:55:56.221604 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:56.220963 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17327d7e-8d87-4461-be7f-1ed4c1d17701-tls-certs\") pod \"test-model-9c1e26-kserve-7cc944cb5f-xjljc\" (UID: \"17327d7e-8d87-4461-be7f-1ed4c1d17701\") " pod="ai-tenant-e2e-inf-b-9c1e26/test-model-9c1e26-kserve-7cc944cb5f-xjljc" Oct 30 16:55:56.221604 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:56.221001 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzrk4\" (UniqueName: \"kubernetes.io/projected/17327d7e-8d87-4461-be7f-1ed4c1d17701-kube-api-access-zzrk4\") pod \"test-model-9c1e26-kserve-7cc944cb5f-xjljc\" (UID: \"17327d7e-8d87-4461-be7f-1ed4c1d17701\") " pod="ai-tenant-e2e-inf-b-9c1e26/test-model-9c1e26-kserve-7cc944cb5f-xjljc" Oct 30 16:55:56.221604 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:56.221065 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17327d7e-8d87-4461-be7f-1ed4c1d17701-model-cache\") pod \"test-model-9c1e26-kserve-7cc944cb5f-xjljc\" (UID: \"17327d7e-8d87-4461-be7f-1ed4c1d17701\") " pod="ai-tenant-e2e-inf-b-9c1e26/test-model-9c1e26-kserve-7cc944cb5f-xjljc" Oct 30 16:55:56.221604 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:56.221276 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17327d7e-8d87-4461-be7f-1ed4c1d17701-home\") pod \"test-model-9c1e26-kserve-7cc944cb5f-xjljc\" (UID: \"17327d7e-8d87-4461-be7f-1ed4c1d17701\") " pod="ai-tenant-e2e-inf-b-9c1e26/test-model-9c1e26-kserve-7cc944cb5f-xjljc" Oct 30 16:55:56.222077 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:56.222027 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17327d7e-8d87-4461-be7f-1ed4c1d17701-model-cache\") pod \"test-model-9c1e26-kserve-7cc944cb5f-xjljc\" (UID: \"17327d7e-8d87-4461-be7f-1ed4c1d17701\") " pod="ai-tenant-e2e-inf-b-9c1e26/test-model-9c1e26-kserve-7cc944cb5f-xjljc" Oct 30 16:55:56.225603 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:56.222212 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17327d7e-8d87-4461-be7f-1ed4c1d17701-home\") pod \"test-model-9c1e26-kserve-7cc944cb5f-xjljc\" (UID: \"17327d7e-8d87-4461-be7f-1ed4c1d17701\") " pod="ai-tenant-e2e-inf-b-9c1e26/test-model-9c1e26-kserve-7cc944cb5f-xjljc" Oct 30 16:55:56.225603 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:56.223722 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17327d7e-8d87-4461-be7f-1ed4c1d17701-tmp-dir\") pod \"test-model-9c1e26-kserve-7cc944cb5f-xjljc\" (UID: \"17327d7e-8d87-4461-be7f-1ed4c1d17701\") " pod="ai-tenant-e2e-inf-b-9c1e26/test-model-9c1e26-kserve-7cc944cb5f-xjljc" Oct 30 16:55:56.246605 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:56.226749 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17327d7e-8d87-4461-be7f-1ed4c1d17701-tls-certs\") pod \"test-model-9c1e26-kserve-7cc944cb5f-xjljc\" (UID: \"17327d7e-8d87-4461-be7f-1ed4c1d17701\") " pod="ai-tenant-e2e-inf-b-9c1e26/test-model-9c1e26-kserve-7cc944cb5f-xjljc" Oct 30 16:55:56.264608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:56.259144 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17327d7e-8d87-4461-be7f-1ed4c1d17701-dshm\") pod \"test-model-9c1e26-kserve-7cc944cb5f-xjljc\" (UID: \"17327d7e-8d87-4461-be7f-1ed4c1d17701\") " pod="ai-tenant-e2e-inf-b-9c1e26/test-model-9c1e26-kserve-7cc944cb5f-xjljc" Oct 30 16:55:56.264608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:56.260435 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzrk4\" (UniqueName: \"kubernetes.io/projected/17327d7e-8d87-4461-be7f-1ed4c1d17701-kube-api-access-zzrk4\") pod \"test-model-9c1e26-kserve-7cc944cb5f-xjljc\" (UID: \"17327d7e-8d87-4461-be7f-1ed4c1d17701\") " pod="ai-tenant-e2e-inf-b-9c1e26/test-model-9c1e26-kserve-7cc944cb5f-xjljc" Oct 30 16:55:56.288888 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:56.288854 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9c1e26/test-model-9c1e26-kserve-7cc944cb5f-xjljc" Oct 30 16:55:56.698123 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:56.698090 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-9c1e26/test-model-9c1e26-kserve-7cc944cb5f-xjljc"] Oct 30 16:55:56.698721 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:55:56.698692 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod17327d7e_8d87_4461_be7f_1ed4c1d17701.slice/crio-e9e2fe5b128b5646ac69b708b860798e2095ec11d1653a461d743fc3605e6d7d WatchSource:0}: Error finding container e9e2fe5b128b5646ac69b708b860798e2095ec11d1653a461d743fc3605e6d7d: Status 404 returned error can't find the container with id e9e2fe5b128b5646ac69b708b860798e2095ec11d1653a461d743fc3605e6d7d Oct 30 16:55:56.744395 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:56.744367 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-8w9ml" Oct 30 16:55:56.744973 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:56.744923 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-8w9ml" Oct 30 16:55:56.746061 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:56.746037 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-8w9ml" Oct 30 16:55:57.585867 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:57.585829 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9c1e26/test-model-9c1e26-kserve-7cc944cb5f-xjljc" event={"ID":"17327d7e-8d87-4461-be7f-1ed4c1d17701","Type":"ContainerStarted","Data":"3aea17113c5222245dffb11430a8b3d00ed45d767d3571ef83b44f9ce2933d4b"} Oct 30 16:55:57.586294 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:57.586266 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9c1e26/test-model-9c1e26-kserve-7cc944cb5f-xjljc" event={"ID":"17327d7e-8d87-4461-be7f-1ed4c1d17701","Type":"ContainerStarted","Data":"e9e2fe5b128b5646ac69b708b860798e2095ec11d1653a461d743fc3605e6d7d"} Oct 30 16:55:57.679517 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:57.676690 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-9c1e26/test-model-9c1e26-kserve-7cc944cb5f-xjljc" podStartSLOduration=2.676667816 podStartE2EDuration="2.676667816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:55:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:55:57.649820734 +0000 UTC m=+2113.177155955" watchObservedRunningTime="2026-10-30 16:55:57.676667816 +0000 UTC m=+2113.204003037" Oct 30 16:55:58.289853 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:58.289814 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-9c1e26/test-model-9c1e26-kserve-7cc944cb5f-xjljc" Oct 30 16:55:58.379910 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:58.379866 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-17f60f/test-model-17f60f-kserve-6d99c5d4f5-n2vss"] Oct 30 16:55:58.385388 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:58.385365 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-17f60f/test-model-17f60f-kserve-6d99c5d4f5-n2vss" Oct 30 16:55:58.406251 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:58.406218 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-17f60f/test-model-17f60f-kserve-6d99c5d4f5-n2vss"] Oct 30 16:55:58.412283 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:58.412259 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-17f60f\"/\"test-model-17f60f-kserve-self-signed-certs\"" Oct 30 16:55:58.412283 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:58.412259 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-17f60f\"/\"openshift-service-ca.crt\"" Oct 30 16:55:58.414495 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:58.414474 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-17f60f\"/\"default-dockercfg-8jfgt\"" Oct 30 16:55:58.414689 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:58.414602 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-17f60f\"/\"kube-root-ca.crt\"" Oct 30 16:55:58.426277 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:58.426246 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06b94bcd-8c5a-4143-a936-34d62f4189b0-model-cache\") pod \"test-model-17f60f-kserve-6d99c5d4f5-n2vss\" (UID: \"06b94bcd-8c5a-4143-a936-34d62f4189b0\") " pod="ai-tenant-e2e-inf-a-17f60f/test-model-17f60f-kserve-6d99c5d4f5-n2vss" Oct 30 16:55:58.426463 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:58.426440 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06b94bcd-8c5a-4143-a936-34d62f4189b0-tls-certs\") pod \"test-model-17f60f-kserve-6d99c5d4f5-n2vss\" (UID: \"06b94bcd-8c5a-4143-a936-34d62f4189b0\") " pod="ai-tenant-e2e-inf-a-17f60f/test-model-17f60f-kserve-6d99c5d4f5-n2vss" Oct 30 16:55:58.426663 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:58.426642 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06b94bcd-8c5a-4143-a936-34d62f4189b0-tmp-dir\") pod \"test-model-17f60f-kserve-6d99c5d4f5-n2vss\" (UID: \"06b94bcd-8c5a-4143-a936-34d62f4189b0\") " pod="ai-tenant-e2e-inf-a-17f60f/test-model-17f60f-kserve-6d99c5d4f5-n2vss" Oct 30 16:55:58.426851 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:58.426825 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99569\" (UniqueName: \"kubernetes.io/projected/06b94bcd-8c5a-4143-a936-34d62f4189b0-kube-api-access-99569\") pod \"test-model-17f60f-kserve-6d99c5d4f5-n2vss\" (UID: \"06b94bcd-8c5a-4143-a936-34d62f4189b0\") " pod="ai-tenant-e2e-inf-a-17f60f/test-model-17f60f-kserve-6d99c5d4f5-n2vss" Oct 30 16:55:58.426911 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:58.426893 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06b94bcd-8c5a-4143-a936-34d62f4189b0-home\") pod \"test-model-17f60f-kserve-6d99c5d4f5-n2vss\" (UID: \"06b94bcd-8c5a-4143-a936-34d62f4189b0\") " pod="ai-tenant-e2e-inf-a-17f60f/test-model-17f60f-kserve-6d99c5d4f5-n2vss" Oct 30 16:55:58.426989 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:58.426966 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06b94bcd-8c5a-4143-a936-34d62f4189b0-dshm\") pod \"test-model-17f60f-kserve-6d99c5d4f5-n2vss\" (UID: \"06b94bcd-8c5a-4143-a936-34d62f4189b0\") " pod="ai-tenant-e2e-inf-a-17f60f/test-model-17f60f-kserve-6d99c5d4f5-n2vss" Oct 30 16:55:58.528243 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:58.528204 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99569\" (UniqueName: \"kubernetes.io/projected/06b94bcd-8c5a-4143-a936-34d62f4189b0-kube-api-access-99569\") pod \"test-model-17f60f-kserve-6d99c5d4f5-n2vss\" (UID: \"06b94bcd-8c5a-4143-a936-34d62f4189b0\") " pod="ai-tenant-e2e-inf-a-17f60f/test-model-17f60f-kserve-6d99c5d4f5-n2vss" Oct 30 16:55:58.528412 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:58.528262 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06b94bcd-8c5a-4143-a936-34d62f4189b0-home\") pod \"test-model-17f60f-kserve-6d99c5d4f5-n2vss\" (UID: \"06b94bcd-8c5a-4143-a936-34d62f4189b0\") " pod="ai-tenant-e2e-inf-a-17f60f/test-model-17f60f-kserve-6d99c5d4f5-n2vss" Oct 30 16:55:58.528412 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:58.528381 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06b94bcd-8c5a-4143-a936-34d62f4189b0-dshm\") pod \"test-model-17f60f-kserve-6d99c5d4f5-n2vss\" (UID: \"06b94bcd-8c5a-4143-a936-34d62f4189b0\") " pod="ai-tenant-e2e-inf-a-17f60f/test-model-17f60f-kserve-6d99c5d4f5-n2vss" Oct 30 16:55:58.528495 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:58.528480 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06b94bcd-8c5a-4143-a936-34d62f4189b0-model-cache\") pod \"test-model-17f60f-kserve-6d99c5d4f5-n2vss\" (UID: \"06b94bcd-8c5a-4143-a936-34d62f4189b0\") " pod="ai-tenant-e2e-inf-a-17f60f/test-model-17f60f-kserve-6d99c5d4f5-n2vss" Oct 30 16:55:58.528614 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:58.528570 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06b94bcd-8c5a-4143-a936-34d62f4189b0-tls-certs\") pod \"test-model-17f60f-kserve-6d99c5d4f5-n2vss\" (UID: \"06b94bcd-8c5a-4143-a936-34d62f4189b0\") " pod="ai-tenant-e2e-inf-a-17f60f/test-model-17f60f-kserve-6d99c5d4f5-n2vss" Oct 30 16:55:58.528710 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:58.528694 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06b94bcd-8c5a-4143-a936-34d62f4189b0-tmp-dir\") pod \"test-model-17f60f-kserve-6d99c5d4f5-n2vss\" (UID: \"06b94bcd-8c5a-4143-a936-34d62f4189b0\") " pod="ai-tenant-e2e-inf-a-17f60f/test-model-17f60f-kserve-6d99c5d4f5-n2vss" Oct 30 16:55:58.528870 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:58.528842 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06b94bcd-8c5a-4143-a936-34d62f4189b0-home\") pod \"test-model-17f60f-kserve-6d99c5d4f5-n2vss\" (UID: \"06b94bcd-8c5a-4143-a936-34d62f4189b0\") " pod="ai-tenant-e2e-inf-a-17f60f/test-model-17f60f-kserve-6d99c5d4f5-n2vss" Oct 30 16:55:58.528966 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:58.528944 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06b94bcd-8c5a-4143-a936-34d62f4189b0-model-cache\") pod \"test-model-17f60f-kserve-6d99c5d4f5-n2vss\" (UID: \"06b94bcd-8c5a-4143-a936-34d62f4189b0\") " pod="ai-tenant-e2e-inf-a-17f60f/test-model-17f60f-kserve-6d99c5d4f5-n2vss" Oct 30 16:55:58.529110 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:58.529092 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06b94bcd-8c5a-4143-a936-34d62f4189b0-tmp-dir\") pod \"test-model-17f60f-kserve-6d99c5d4f5-n2vss\" (UID: \"06b94bcd-8c5a-4143-a936-34d62f4189b0\") " pod="ai-tenant-e2e-inf-a-17f60f/test-model-17f60f-kserve-6d99c5d4f5-n2vss" Oct 30 16:55:58.530844 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:58.530820 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06b94bcd-8c5a-4143-a936-34d62f4189b0-dshm\") pod \"test-model-17f60f-kserve-6d99c5d4f5-n2vss\" (UID: \"06b94bcd-8c5a-4143-a936-34d62f4189b0\") " pod="ai-tenant-e2e-inf-a-17f60f/test-model-17f60f-kserve-6d99c5d4f5-n2vss" Oct 30 16:55:58.531157 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:58.531137 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06b94bcd-8c5a-4143-a936-34d62f4189b0-tls-certs\") pod \"test-model-17f60f-kserve-6d99c5d4f5-n2vss\" (UID: \"06b94bcd-8c5a-4143-a936-34d62f4189b0\") " pod="ai-tenant-e2e-inf-a-17f60f/test-model-17f60f-kserve-6d99c5d4f5-n2vss" Oct 30 16:55:58.552642 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:58.552570 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99569\" (UniqueName: \"kubernetes.io/projected/06b94bcd-8c5a-4143-a936-34d62f4189b0-kube-api-access-99569\") pod \"test-model-17f60f-kserve-6d99c5d4f5-n2vss\" (UID: \"06b94bcd-8c5a-4143-a936-34d62f4189b0\") " pod="ai-tenant-e2e-inf-a-17f60f/test-model-17f60f-kserve-6d99c5d4f5-n2vss" Oct 30 16:55:58.724176 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:58.724139 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-17f60f/test-model-17f60f-kserve-6d99c5d4f5-n2vss" Oct 30 16:55:59.329209 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:59.329174 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-17f60f/test-model-17f60f-kserve-6d99c5d4f5-n2vss"] Oct 30 16:55:59.698158 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:59.698117 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-17f60f/test-model-17f60f-kserve-6d99c5d4f5-n2vss" event={"ID":"06b94bcd-8c5a-4143-a936-34d62f4189b0","Type":"ContainerStarted","Data":"a95ab4dfe0d08568ad11183f96f79b971e509ca877f188bee22411ab5c27614f"} Oct 30 16:55:59.698350 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:59.698170 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-17f60f/test-model-17f60f-kserve-6d99c5d4f5-n2vss" event={"ID":"06b94bcd-8c5a-4143-a936-34d62f4189b0","Type":"ContainerStarted","Data":"3d9745094c61d86ccfe25185b494025ae1bf9686cdb14f759829b92e80ad36e6"} Oct 30 16:55:59.730286 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:59.730257 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-17f60f/test-model-17f60f-kserve-6d99c5d4f5-n2vss" Oct 30 16:55:59.745061 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:55:59.745011 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-17f60f/test-model-17f60f-kserve-6d99c5d4f5-n2vss" podStartSLOduration=1.744998454 podStartE2EDuration="1.744998454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:55:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:55:59.74334908 +0000 UTC m=+2115.270684299" watchObservedRunningTime="2026-10-30 16:55:59.744998454 +0000 UTC m=+2115.272333673" Oct 30 16:56:06.290023 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:06.289981 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-9c1e26/test-model-9c1e26-kserve-7cc944cb5f-xjljc" Oct 30 16:56:06.304036 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:06.304006 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-9c1e26/test-model-9c1e26-kserve-7cc944cb5f-xjljc" Oct 30 16:56:06.613535 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:06.613458 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-ff1189/test-model-ff1189-kserve-84bc7d6c57-5f475"] Oct 30 16:56:06.916257 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:06.916228 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-9c1e26/test-model-9c1e26-kserve-7cc944cb5f-xjljc" Oct 30 16:56:08.730081 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:08.730053 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-17f60f/test-model-17f60f-kserve-6d99c5d4f5-n2vss" Oct 30 16:56:08.743211 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:08.743169 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-17f60f/test-model-17f60f-kserve-6d99c5d4f5-n2vss" Oct 30 16:56:08.928455 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:08.928421 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-17f60f/test-model-17f60f-kserve-6d99c5d4f5-n2vss" Oct 30 16:56:11.443776 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:11.443737 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-689899/test-model-689899-kserve-5f8945cf6f-925q2"] Oct 30 16:56:11.454053 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:11.454024 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-689899/test-model-689899-kserve-5f8945cf6f-925q2" Oct 30 16:56:11.471675 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:11.471645 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-689899/test-model-689899-kserve-5f8945cf6f-925q2"] Oct 30 16:56:11.472876 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:11.472852 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-689899\"/\"kube-root-ca.crt\"" Oct 30 16:56:11.473947 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:11.473913 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-689899\"/\"openshift-service-ca.crt\"" Oct 30 16:56:11.483564 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:11.483545 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-689899\"/\"test-model-689899-kserve-self-signed-certs\"" Oct 30 16:56:11.508007 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:11.507970 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbc3abfb-2bcf-4133-9a49-65b29f06f12a-tmp-dir\") pod \"test-model-689899-kserve-5f8945cf6f-925q2\" (UID: \"cbc3abfb-2bcf-4133-9a49-65b29f06f12a\") " pod="ai-tenant-e2e-inf-b-689899/test-model-689899-kserve-5f8945cf6f-925q2" Oct 30 16:56:11.508314 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:11.508289 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbc3abfb-2bcf-4133-9a49-65b29f06f12a-model-cache\") pod \"test-model-689899-kserve-5f8945cf6f-925q2\" (UID: \"cbc3abfb-2bcf-4133-9a49-65b29f06f12a\") " pod="ai-tenant-e2e-inf-b-689899/test-model-689899-kserve-5f8945cf6f-925q2" Oct 30 16:56:11.508412 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:11.508395 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzgv2\" (UniqueName: \"kubernetes.io/projected/cbc3abfb-2bcf-4133-9a49-65b29f06f12a-kube-api-access-gzgv2\") pod \"test-model-689899-kserve-5f8945cf6f-925q2\" (UID: \"cbc3abfb-2bcf-4133-9a49-65b29f06f12a\") " pod="ai-tenant-e2e-inf-b-689899/test-model-689899-kserve-5f8945cf6f-925q2" Oct 30 16:56:11.508481 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:11.508442 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbc3abfb-2bcf-4133-9a49-65b29f06f12a-tls-certs\") pod \"test-model-689899-kserve-5f8945cf6f-925q2\" (UID: \"cbc3abfb-2bcf-4133-9a49-65b29f06f12a\") " pod="ai-tenant-e2e-inf-b-689899/test-model-689899-kserve-5f8945cf6f-925q2" Oct 30 16:56:11.508546 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:11.508524 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbc3abfb-2bcf-4133-9a49-65b29f06f12a-dshm\") pod \"test-model-689899-kserve-5f8945cf6f-925q2\" (UID: \"cbc3abfb-2bcf-4133-9a49-65b29f06f12a\") " pod="ai-tenant-e2e-inf-b-689899/test-model-689899-kserve-5f8945cf6f-925q2" Oct 30 16:56:11.508626 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:11.508562 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cbc3abfb-2bcf-4133-9a49-65b29f06f12a-home\") pod \"test-model-689899-kserve-5f8945cf6f-925q2\" (UID: \"cbc3abfb-2bcf-4133-9a49-65b29f06f12a\") " pod="ai-tenant-e2e-inf-b-689899/test-model-689899-kserve-5f8945cf6f-925q2" Oct 30 16:56:11.544662 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:11.544635 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-689899\"/\"default-dockercfg-bpmhj\"" Oct 30 16:56:11.609970 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:11.609937 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbc3abfb-2bcf-4133-9a49-65b29f06f12a-tls-certs\") pod \"test-model-689899-kserve-5f8945cf6f-925q2\" (UID: \"cbc3abfb-2bcf-4133-9a49-65b29f06f12a\") " pod="ai-tenant-e2e-inf-b-689899/test-model-689899-kserve-5f8945cf6f-925q2" Oct 30 16:56:11.610122 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:11.610011 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbc3abfb-2bcf-4133-9a49-65b29f06f12a-dshm\") pod \"test-model-689899-kserve-5f8945cf6f-925q2\" (UID: \"cbc3abfb-2bcf-4133-9a49-65b29f06f12a\") " pod="ai-tenant-e2e-inf-b-689899/test-model-689899-kserve-5f8945cf6f-925q2" Oct 30 16:56:11.610122 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:11.610048 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cbc3abfb-2bcf-4133-9a49-65b29f06f12a-home\") pod \"test-model-689899-kserve-5f8945cf6f-925q2\" (UID: \"cbc3abfb-2bcf-4133-9a49-65b29f06f12a\") " pod="ai-tenant-e2e-inf-b-689899/test-model-689899-kserve-5f8945cf6f-925q2" Oct 30 16:56:11.610122 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:11.610106 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbc3abfb-2bcf-4133-9a49-65b29f06f12a-tmp-dir\") pod \"test-model-689899-kserve-5f8945cf6f-925q2\" (UID: \"cbc3abfb-2bcf-4133-9a49-65b29f06f12a\") " pod="ai-tenant-e2e-inf-b-689899/test-model-689899-kserve-5f8945cf6f-925q2" Oct 30 16:56:11.610370 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:11.610350 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbc3abfb-2bcf-4133-9a49-65b29f06f12a-model-cache\") pod \"test-model-689899-kserve-5f8945cf6f-925q2\" (UID: \"cbc3abfb-2bcf-4133-9a49-65b29f06f12a\") " pod="ai-tenant-e2e-inf-b-689899/test-model-689899-kserve-5f8945cf6f-925q2" Oct 30 16:56:11.610467 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:11.610444 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzgv2\" (UniqueName: \"kubernetes.io/projected/cbc3abfb-2bcf-4133-9a49-65b29f06f12a-kube-api-access-gzgv2\") pod \"test-model-689899-kserve-5f8945cf6f-925q2\" (UID: \"cbc3abfb-2bcf-4133-9a49-65b29f06f12a\") " pod="ai-tenant-e2e-inf-b-689899/test-model-689899-kserve-5f8945cf6f-925q2" Oct 30 16:56:11.610645 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:11.610568 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cbc3abfb-2bcf-4133-9a49-65b29f06f12a-home\") pod \"test-model-689899-kserve-5f8945cf6f-925q2\" (UID: \"cbc3abfb-2bcf-4133-9a49-65b29f06f12a\") " pod="ai-tenant-e2e-inf-b-689899/test-model-689899-kserve-5f8945cf6f-925q2" Oct 30 16:56:11.610645 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:11.610628 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbc3abfb-2bcf-4133-9a49-65b29f06f12a-tmp-dir\") pod \"test-model-689899-kserve-5f8945cf6f-925q2\" (UID: \"cbc3abfb-2bcf-4133-9a49-65b29f06f12a\") " pod="ai-tenant-e2e-inf-b-689899/test-model-689899-kserve-5f8945cf6f-925q2" Oct 30 16:56:11.610816 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:11.610714 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbc3abfb-2bcf-4133-9a49-65b29f06f12a-model-cache\") pod \"test-model-689899-kserve-5f8945cf6f-925q2\" (UID: \"cbc3abfb-2bcf-4133-9a49-65b29f06f12a\") " pod="ai-tenant-e2e-inf-b-689899/test-model-689899-kserve-5f8945cf6f-925q2" Oct 30 16:56:11.612351 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:11.612329 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbc3abfb-2bcf-4133-9a49-65b29f06f12a-dshm\") pod \"test-model-689899-kserve-5f8945cf6f-925q2\" (UID: \"cbc3abfb-2bcf-4133-9a49-65b29f06f12a\") " pod="ai-tenant-e2e-inf-b-689899/test-model-689899-kserve-5f8945cf6f-925q2" Oct 30 16:56:11.614668 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:11.614646 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbc3abfb-2bcf-4133-9a49-65b29f06f12a-tls-certs\") pod \"test-model-689899-kserve-5f8945cf6f-925q2\" (UID: \"cbc3abfb-2bcf-4133-9a49-65b29f06f12a\") " pod="ai-tenant-e2e-inf-b-689899/test-model-689899-kserve-5f8945cf6f-925q2" Oct 30 16:56:11.634894 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:11.634867 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzgv2\" (UniqueName: \"kubernetes.io/projected/cbc3abfb-2bcf-4133-9a49-65b29f06f12a-kube-api-access-gzgv2\") pod \"test-model-689899-kserve-5f8945cf6f-925q2\" (UID: \"cbc3abfb-2bcf-4133-9a49-65b29f06f12a\") " pod="ai-tenant-e2e-inf-b-689899/test-model-689899-kserve-5f8945cf6f-925q2" Oct 30 16:56:11.789150 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:11.789063 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-689899/test-model-689899-kserve-5f8945cf6f-925q2" Oct 30 16:56:12.214181 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:56:12.214137 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcbc3abfb_2bcf_4133_9a49_65b29f06f12a.slice/crio-644c5299ef2c48cab18c5d998f498f5a5426e7455875d9fcc28c7474ea079183 WatchSource:0}: Error finding container 644c5299ef2c48cab18c5d998f498f5a5426e7455875d9fcc28c7474ea079183: Status 404 returned error can't find the container with id 644c5299ef2c48cab18c5d998f498f5a5426e7455875d9fcc28c7474ea079183 Oct 30 16:56:12.216119 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:12.216098 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-689899/test-model-689899-kserve-5f8945cf6f-925q2"] Oct 30 16:56:12.959848 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:12.959807 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-689899/test-model-689899-kserve-5f8945cf6f-925q2" event={"ID":"cbc3abfb-2bcf-4133-9a49-65b29f06f12a","Type":"ContainerStarted","Data":"d14238fc4fd13d9b07f20e44075b2921d50f02e6db0c76c7c95d7796177f42be"} Oct 30 16:56:12.960335 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:12.959859 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-689899/test-model-689899-kserve-5f8945cf6f-925q2" event={"ID":"cbc3abfb-2bcf-4133-9a49-65b29f06f12a","Type":"ContainerStarted","Data":"644c5299ef2c48cab18c5d998f498f5a5426e7455875d9fcc28c7474ea079183"} Oct 30 16:56:13.006034 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:13.005987 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-689899/test-model-689899-kserve-5f8945cf6f-925q2" podStartSLOduration=2.005971849 podStartE2EDuration="2.005971849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:56:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:56:13.002824431 +0000 UTC m=+2128.530159653" watchObservedRunningTime="2026-10-30 16:56:13.005971849 +0000 UTC m=+2128.533307103" Oct 30 16:56:13.789851 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:13.789817 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-689899/test-model-689899-kserve-5f8945cf6f-925q2" Oct 30 16:56:18.847889 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:56:18.847848 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 16:56:21.642483 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:21.642431 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-ff1189/test-model-ff1189-kserve-84bc7d6c57-5f475" podUID="a4e168d8-c482-4fdf-9d3c-aa41fd252c90" containerName="main" containerID="cri-o://139cd2aa8b19eba534fd6f80e60264580164404115226f981d0b2ecf3b351b8b" gracePeriod=45 Oct 30 16:56:21.791219 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:21.790770 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-689899/test-model-689899-kserve-5f8945cf6f-925q2" Oct 30 16:56:21.806715 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:21.806653 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-689899/test-model-689899-kserve-5f8945cf6f-925q2" Oct 30 16:56:22.064318 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:22.063985 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ff1189/test-model-ff1189-kserve-84bc7d6c57-5f475" Oct 30 16:56:22.069632 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:22.069562 5052 generic.go:356] "Generic (PLEG): container finished" podID="a4e168d8-c482-4fdf-9d3c-aa41fd252c90" containerID="139cd2aa8b19eba534fd6f80e60264580164404115226f981d0b2ecf3b351b8b" exitCode=0 Oct 30 16:56:22.069632 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:22.069613 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ff1189/test-model-ff1189-kserve-84bc7d6c57-5f475" event={"ID":"a4e168d8-c482-4fdf-9d3c-aa41fd252c90","Type":"ContainerDied","Data":"139cd2aa8b19eba534fd6f80e60264580164404115226f981d0b2ecf3b351b8b"} Oct 30 16:56:22.069833 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:22.069636 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ff1189/test-model-ff1189-kserve-84bc7d6c57-5f475" Oct 30 16:56:22.069833 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:22.069671 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ff1189/test-model-ff1189-kserve-84bc7d6c57-5f475" event={"ID":"a4e168d8-c482-4fdf-9d3c-aa41fd252c90","Type":"ContainerDied","Data":"2811e531d15763581a8f3e6ea61540fddd5640082e8bc34fa814c4b45d07dfcf"} Oct 30 16:56:22.069833 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:22.069696 5052 scope.go:122] "RemoveContainer" containerID="139cd2aa8b19eba534fd6f80e60264580164404115226f981d0b2ecf3b351b8b" Oct 30 16:56:22.082386 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:22.082354 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-689899/test-model-689899-kserve-5f8945cf6f-925q2" Oct 30 16:56:22.084086 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:22.084070 5052 scope.go:122] "RemoveContainer" containerID="139cd2aa8b19eba534fd6f80e60264580164404115226f981d0b2ecf3b351b8b" Oct 30 16:56:22.084371 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:22.084345 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4e168d8-c482-4fdf-9d3c-aa41fd252c90-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4e168d8-c482-4fdf-9d3c-aa41fd252c90-dshm\") pod \"a4e168d8-c482-4fdf-9d3c-aa41fd252c90\" (UID: \"a4e168d8-c482-4fdf-9d3c-aa41fd252c90\") " Oct 30 16:56:22.084497 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:56:22.084409 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"139cd2aa8b19eba534fd6f80e60264580164404115226f981d0b2ecf3b351b8b\": container with ID starting with 139cd2aa8b19eba534fd6f80e60264580164404115226f981d0b2ecf3b351b8b not found: ID does not exist" containerID="139cd2aa8b19eba534fd6f80e60264580164404115226f981d0b2ecf3b351b8b" Oct 30 16:56:22.084497 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:22.084440 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"139cd2aa8b19eba534fd6f80e60264580164404115226f981d0b2ecf3b351b8b"} err="failed to get container status \"139cd2aa8b19eba534fd6f80e60264580164404115226f981d0b2ecf3b351b8b\": rpc error: code = NotFound desc = could not find container \"139cd2aa8b19eba534fd6f80e60264580164404115226f981d0b2ecf3b351b8b\": container with ID starting with 139cd2aa8b19eba534fd6f80e60264580164404115226f981d0b2ecf3b351b8b not found: ID does not exist" Oct 30 16:56:22.084497 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:22.084446 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4e168d8-c482-4fdf-9d3c-aa41fd252c90-home\" (UniqueName: \"kubernetes.io/empty-dir/a4e168d8-c482-4fdf-9d3c-aa41fd252c90-home\") pod \"a4e168d8-c482-4fdf-9d3c-aa41fd252c90\" (UID: \"a4e168d8-c482-4fdf-9d3c-aa41fd252c90\") " Oct 30 16:56:22.084724 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:22.084700 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4e168d8-c482-4fdf-9d3c-aa41fd252c90-home" pod "a4e168d8-c482-4fdf-9d3c-aa41fd252c90" (UID: "a4e168d8-c482-4fdf-9d3c-aa41fd252c90"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:56:22.085056 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:22.085028 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4e168d8-c482-4fdf-9d3c-aa41fd252c90-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4e168d8-c482-4fdf-9d3c-aa41fd252c90-model-cache\") pod \"a4e168d8-c482-4fdf-9d3c-aa41fd252c90\" (UID: \"a4e168d8-c482-4fdf-9d3c-aa41fd252c90\") " Oct 30 16:56:22.085272 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:22.085099 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4e168d8-c482-4fdf-9d3c-aa41fd252c90-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4e168d8-c482-4fdf-9d3c-aa41fd252c90-tmp-dir\") pod \"a4e168d8-c482-4fdf-9d3c-aa41fd252c90\" (UID: \"a4e168d8-c482-4fdf-9d3c-aa41fd252c90\") " Oct 30 16:56:22.085272 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:22.085155 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4e168d8-c482-4fdf-9d3c-aa41fd252c90-tls-certs\" (UniqueName: \"kubernetes.io/secret/a4e168d8-c482-4fdf-9d3c-aa41fd252c90-tls-certs\") pod \"a4e168d8-c482-4fdf-9d3c-aa41fd252c90\" (UID: \"a4e168d8-c482-4fdf-9d3c-aa41fd252c90\") " Oct 30 16:56:22.085272 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:22.085188 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4e168d8-c482-4fdf-9d3c-aa41fd252c90-kube-api-access-hbnc6\" (UniqueName: \"kubernetes.io/projected/a4e168d8-c482-4fdf-9d3c-aa41fd252c90-kube-api-access-hbnc6\") pod \"a4e168d8-c482-4fdf-9d3c-aa41fd252c90\" (UID: \"a4e168d8-c482-4fdf-9d3c-aa41fd252c90\") " Oct 30 16:56:22.085883 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:22.085296 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4e168d8-c482-4fdf-9d3c-aa41fd252c90-model-cache" pod "a4e168d8-c482-4fdf-9d3c-aa41fd252c90" (UID: "a4e168d8-c482-4fdf-9d3c-aa41fd252c90"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:56:22.085883 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:22.085485 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4e168d8-c482-4fdf-9d3c-aa41fd252c90-tmp-dir" pod "a4e168d8-c482-4fdf-9d3c-aa41fd252c90" (UID: "a4e168d8-c482-4fdf-9d3c-aa41fd252c90"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:56:22.086011 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:22.085937 5052 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4e168d8-c482-4fdf-9d3c-aa41fd252c90-model-cache\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:56:22.086011 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:22.085959 5052 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4e168d8-c482-4fdf-9d3c-aa41fd252c90-tmp-dir\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:56:22.086011 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:22.085971 5052 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4e168d8-c482-4fdf-9d3c-aa41fd252c90-home\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:56:22.087510 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:22.087468 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4e168d8-c482-4fdf-9d3c-aa41fd252c90-dshm" pod "a4e168d8-c482-4fdf-9d3c-aa41fd252c90" (UID: "a4e168d8-c482-4fdf-9d3c-aa41fd252c90"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:56:22.088001 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:22.087963 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4e168d8-c482-4fdf-9d3c-aa41fd252c90-kube-api-access-hbnc6" pod "a4e168d8-c482-4fdf-9d3c-aa41fd252c90" (UID: "a4e168d8-c482-4fdf-9d3c-aa41fd252c90"). InnerVolumeSpecName "kube-api-access-hbnc6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:56:22.088205 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:22.088180 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4e168d8-c482-4fdf-9d3c-aa41fd252c90-tls-certs" pod "a4e168d8-c482-4fdf-9d3c-aa41fd252c90" (UID: "a4e168d8-c482-4fdf-9d3c-aa41fd252c90"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:56:22.188520 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:22.188481 5052 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4e168d8-c482-4fdf-9d3c-aa41fd252c90-tls-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:56:22.188520 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:22.188524 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbnc6\" (UniqueName: \"kubernetes.io/projected/a4e168d8-c482-4fdf-9d3c-aa41fd252c90-kube-api-access-hbnc6\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:56:22.188795 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:22.188541 5052 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4e168d8-c482-4fdf-9d3c-aa41fd252c90-dshm\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:56:22.421967 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:22.421931 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-ff1189/test-model-ff1189-kserve-84bc7d6c57-5f475"] Oct 30 16:56:22.426473 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:22.426444 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-ff1189/test-model-ff1189-kserve-84bc7d6c57-5f475"] Oct 30 16:56:22.603020 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:22.602988 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4e168d8-c482-4fdf-9d3c-aa41fd252c90" path="/var/lib/kubelet/pods/a4e168d8-c482-4fdf-9d3c-aa41fd252c90/volumes" Oct 30 16:56:28.133495 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:28.133456 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cdwcj"] Oct 30 16:56:29.273022 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:29.272983 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ff1189-759ff754b7-gl6mj"] Oct 30 16:56:32.175408 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:32.175328 5052 generic.go:356] "Generic (PLEG): container finished" podID="e47df7d5-b3d9-4ba4-91a7-7a530ab85616" containerID="f9f1ac74236c917e6bf027705dfb0f20b260a1dbef4d2363b76d7d39cd18f6e8" exitCode=0 Oct 30 16:56:32.175844 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:32.175408 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f92caf-openshift-default-56d7b858bc-wg985" event={"ID":"e47df7d5-b3d9-4ba4-91a7-7a530ab85616","Type":"ContainerDied","Data":"f9f1ac74236c917e6bf027705dfb0f20b260a1dbef4d2363b76d7d39cd18f6e8"} Oct 30 16:56:32.176031 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:32.176012 5052 scope.go:122] "RemoveContainer" containerID="f9f1ac74236c917e6bf027705dfb0f20b260a1dbef4d2363b76d7d39cd18f6e8" Oct 30 16:56:32.178212 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:32.178186 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:56:32.178285 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:32.178258 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:56:32.178324 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:32.178305 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:56:33.188436 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:33.188398 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f92caf-openshift-default-56d7b858bc-wg985" event={"ID":"e47df7d5-b3d9-4ba4-91a7-7a530ab85616","Type":"ContainerStarted","Data":"1a9aac3d8750fbb0e7e87936730cc26ebfd9d548bb5164bcf6bc21ec551e462f"} Oct 30 16:56:33.841609 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:33.841224 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-f92caf-openshift-default-56d7b858bc-wg985" Oct 30 16:56:33.856036 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:33.855999 5052 patch_prober.go:29] interesting pod/e2e-inf-b-f92caf-openshift-default-56d7b858bc-wg985 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 30 16:56:33.856200 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:33.856063 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f92caf-openshift-default-56d7b858bc-wg985" podUID="e47df7d5-b3d9-4ba4-91a7-7a530ab85616" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 30 16:56:34.282618 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:34.275106 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ff1189-759ff754b7-gl6mj" podUID="5cbbc24b-bf64-455e-9dc2-c34487840dfb" containerName="payload-pre-processing" containerID="cri-o://321c963f3ddeb9e3048bc683bc1bd9dde745780526526d048f87684b21d0146f" gracePeriod=25 Oct 30 16:56:34.696681 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:34.696646 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ff1189-759ff754b7-gl6mj" podUID="5cbbc24b-bf64-455e-9dc2-c34487840dfb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 16:56:34.849004 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:34.848964 5052 patch_prober.go:29] interesting pod/e2e-inf-b-f92caf-openshift-default-56d7b858bc-wg985 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 30 16:56:34.849294 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:34.849266 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f92caf-openshift-default-56d7b858bc-wg985" podUID="e47df7d5-b3d9-4ba4-91a7-7a530ab85616" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 30 16:56:35.839606 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:35.838517 5052 patch_prober.go:29] interesting pod/e2e-inf-b-f92caf-openshift-default-56d7b858bc-wg985 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 30 16:56:35.839606 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:35.838605 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f92caf-openshift-default-56d7b858bc-wg985" podUID="e47df7d5-b3d9-4ba4-91a7-7a530ab85616" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 30 16:56:36.847602 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:36.833893 5052 patch_prober.go:29] interesting pod/e2e-inf-b-f92caf-openshift-default-56d7b858bc-wg985 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 30 16:56:36.847602 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:36.833965 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f92caf-openshift-default-56d7b858bc-wg985" podUID="e47df7d5-b3d9-4ba4-91a7-7a530ab85616" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 30 16:56:37.796655 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:37.794211 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-f92caf-openshift-default-56d7b858bc-wg985" Oct 30 16:56:37.833661 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:37.832952 5052 patch_prober.go:29] interesting pod/e2e-inf-b-f92caf-openshift-default-56d7b858bc-wg985 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 30 16:56:37.833661 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:37.833027 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f92caf-openshift-default-56d7b858bc-wg985" podUID="e47df7d5-b3d9-4ba4-91a7-7a530ab85616" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 30 16:56:38.905389 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:38.905337 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-f92caf-openshift-default-56d7b858bc-wg985" Oct 30 16:56:38.908334 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:38.908304 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-f92caf-openshift-default-56d7b858bc-wg985" Oct 30 16:56:44.695802 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:44.695769 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ff1189-759ff754b7-gl6mj" podUID="5cbbc24b-bf64-455e-9dc2-c34487840dfb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 16:56:45.753807 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:45.753771 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-ff1189-openshift-default-7d9f9f5c65-jjdbw"] Oct 30 16:56:45.754322 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:45.754017 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-ff1189-openshift-default-7d9f9f5c65-jjdbw" podUID="ba6262c9-b3ea-4c61-bb84-4f60954d02a0" containerName="istio-proxy" containerID="cri-o://4e8706eab0d61e2df4b675f8a4d18d470b42ce6fa1fcb61ad1ba4ba12d340e54" gracePeriod=30 Oct 30 16:56:47.427185 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:47.427154 5052 patch_prober.go:29] interesting pod/e2e-inf-a-ff1189-openshift-default-7d9f9f5c65-jjdbw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 16:56:47.427626 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:47.427210 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-ff1189-openshift-default-7d9f9f5c65-jjdbw" podUID="ba6262c9-b3ea-4c61-bb84-4f60954d02a0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 16:56:48.922576 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:56:48.922536 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 16:56:51.060251 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:51.060227 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ff1189-openshift-default-7d9f9f5c65-jjdbw" Oct 30 16:56:51.180545 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:51.180498 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba6262c9-b3ea-4c61-bb84-4f60954d02a0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba6262c9-b3ea-4c61-bb84-4f60954d02a0-istiod-ca-cert\") pod \"ba6262c9-b3ea-4c61-bb84-4f60954d02a0\" (UID: \"ba6262c9-b3ea-4c61-bb84-4f60954d02a0\") " Oct 30 16:56:51.180763 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:51.180563 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba6262c9-b3ea-4c61-bb84-4f60954d02a0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba6262c9-b3ea-4c61-bb84-4f60954d02a0-credential-socket\") pod \"ba6262c9-b3ea-4c61-bb84-4f60954d02a0\" (UID: \"ba6262c9-b3ea-4c61-bb84-4f60954d02a0\") " Oct 30 16:56:51.180847 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:51.180775 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba6262c9-b3ea-4c61-bb84-4f60954d02a0-kube-api-access-cq9nr\" (UniqueName: \"kubernetes.io/projected/ba6262c9-b3ea-4c61-bb84-4f60954d02a0-kube-api-access-cq9nr\") pod \"ba6262c9-b3ea-4c61-bb84-4f60954d02a0\" (UID: \"ba6262c9-b3ea-4c61-bb84-4f60954d02a0\") " Oct 30 16:56:51.180847 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:51.180813 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba6262c9-b3ea-4c61-bb84-4f60954d02a0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba6262c9-b3ea-4c61-bb84-4f60954d02a0-istio-data\") pod \"ba6262c9-b3ea-4c61-bb84-4f60954d02a0\" (UID: \"ba6262c9-b3ea-4c61-bb84-4f60954d02a0\") " Oct 30 16:56:51.180971 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:51.180872 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba6262c9-b3ea-4c61-bb84-4f60954d02a0-credential-socket" pod "ba6262c9-b3ea-4c61-bb84-4f60954d02a0" (UID: "ba6262c9-b3ea-4c61-bb84-4f60954d02a0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:56:51.180971 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:51.180906 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba6262c9-b3ea-4c61-bb84-4f60954d02a0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba6262c9-b3ea-4c61-bb84-4f60954d02a0-istio-envoy\") pod \"ba6262c9-b3ea-4c61-bb84-4f60954d02a0\" (UID: \"ba6262c9-b3ea-4c61-bb84-4f60954d02a0\") " Oct 30 16:56:51.180971 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:51.180943 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ba6262c9-b3ea-4c61-bb84-4f60954d02a0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba6262c9-b3ea-4c61-bb84-4f60954d02a0-istio-podinfo\") pod \"ba6262c9-b3ea-4c61-bb84-4f60954d02a0\" (UID: \"ba6262c9-b3ea-4c61-bb84-4f60954d02a0\") " Oct 30 16:56:51.181140 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:51.180979 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba6262c9-b3ea-4c61-bb84-4f60954d02a0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba6262c9-b3ea-4c61-bb84-4f60954d02a0-workload-certs\") pod \"ba6262c9-b3ea-4c61-bb84-4f60954d02a0\" (UID: \"ba6262c9-b3ea-4c61-bb84-4f60954d02a0\") " Oct 30 16:56:51.181140 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:51.180991 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba6262c9-b3ea-4c61-bb84-4f60954d02a0-istiod-ca-cert" pod "ba6262c9-b3ea-4c61-bb84-4f60954d02a0" (UID: "ba6262c9-b3ea-4c61-bb84-4f60954d02a0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:56:51.181140 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:51.181011 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba6262c9-b3ea-4c61-bb84-4f60954d02a0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba6262c9-b3ea-4c61-bb84-4f60954d02a0-workload-socket\") pod \"ba6262c9-b3ea-4c61-bb84-4f60954d02a0\" (UID: \"ba6262c9-b3ea-4c61-bb84-4f60954d02a0\") " Oct 30 16:56:51.181140 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:51.181086 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba6262c9-b3ea-4c61-bb84-4f60954d02a0-istio-token\" (UniqueName: \"kubernetes.io/projected/ba6262c9-b3ea-4c61-bb84-4f60954d02a0-istio-token\") pod \"ba6262c9-b3ea-4c61-bb84-4f60954d02a0\" (UID: \"ba6262c9-b3ea-4c61-bb84-4f60954d02a0\") " Oct 30 16:56:51.181366 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:51.181321 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba6262c9-b3ea-4c61-bb84-4f60954d02a0-workload-certs" pod "ba6262c9-b3ea-4c61-bb84-4f60954d02a0" (UID: "ba6262c9-b3ea-4c61-bb84-4f60954d02a0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:56:51.181491 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:51.181462 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba6262c9-b3ea-4c61-bb84-4f60954d02a0-workload-socket" pod "ba6262c9-b3ea-4c61-bb84-4f60954d02a0" (UID: "ba6262c9-b3ea-4c61-bb84-4f60954d02a0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:56:51.181838 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:51.181801 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba6262c9-b3ea-4c61-bb84-4f60954d02a0-istio-data" pod "ba6262c9-b3ea-4c61-bb84-4f60954d02a0" (UID: "ba6262c9-b3ea-4c61-bb84-4f60954d02a0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:56:51.182569 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:51.182540 5052 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba6262c9-b3ea-4c61-bb84-4f60954d02a0-istio-data\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:56:51.182715 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:51.182572 5052 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba6262c9-b3ea-4c61-bb84-4f60954d02a0-workload-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:56:51.182715 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:51.182604 5052 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba6262c9-b3ea-4c61-bb84-4f60954d02a0-workload-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:56:51.182715 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:51.182618 5052 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba6262c9-b3ea-4c61-bb84-4f60954d02a0-istiod-ca-cert\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:56:51.182715 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:51.182631 5052 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba6262c9-b3ea-4c61-bb84-4f60954d02a0-credential-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:56:51.183523 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:51.183491 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba6262c9-b3ea-4c61-bb84-4f60954d02a0-istio-envoy" pod "ba6262c9-b3ea-4c61-bb84-4f60954d02a0" (UID: "ba6262c9-b3ea-4c61-bb84-4f60954d02a0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:56:51.183916 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:51.183874 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba6262c9-b3ea-4c61-bb84-4f60954d02a0-kube-api-access-cq9nr" pod "ba6262c9-b3ea-4c61-bb84-4f60954d02a0" (UID: "ba6262c9-b3ea-4c61-bb84-4f60954d02a0"). InnerVolumeSpecName "kube-api-access-cq9nr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:56:51.184012 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:51.183973 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ba6262c9-b3ea-4c61-bb84-4f60954d02a0-istio-podinfo" pod "ba6262c9-b3ea-4c61-bb84-4f60954d02a0" (UID: "ba6262c9-b3ea-4c61-bb84-4f60954d02a0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:56:51.184948 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:51.184923 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba6262c9-b3ea-4c61-bb84-4f60954d02a0-istio-token" pod "ba6262c9-b3ea-4c61-bb84-4f60954d02a0" (UID: "ba6262c9-b3ea-4c61-bb84-4f60954d02a0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:56:51.284051 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:51.283990 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cq9nr\" (UniqueName: \"kubernetes.io/projected/ba6262c9-b3ea-4c61-bb84-4f60954d02a0-kube-api-access-cq9nr\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:56:51.284051 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:51.284019 5052 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba6262c9-b3ea-4c61-bb84-4f60954d02a0-istio-envoy\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:56:51.284051 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:51.284028 5052 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba6262c9-b3ea-4c61-bb84-4f60954d02a0-istio-podinfo\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:56:51.284051 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:51.284037 5052 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba6262c9-b3ea-4c61-bb84-4f60954d02a0-istio-token\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:56:51.412954 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:51.412919 5052 generic.go:356] "Generic (PLEG): container finished" podID="ba6262c9-b3ea-4c61-bb84-4f60954d02a0" containerID="4e8706eab0d61e2df4b675f8a4d18d470b42ce6fa1fcb61ad1ba4ba12d340e54" exitCode=0 Oct 30 16:56:51.413113 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:51.412967 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ff1189-openshift-default-7d9f9f5c65-jjdbw" Oct 30 16:56:51.413113 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:51.412970 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ff1189-openshift-default-7d9f9f5c65-jjdbw" event={"ID":"ba6262c9-b3ea-4c61-bb84-4f60954d02a0","Type":"ContainerDied","Data":"4e8706eab0d61e2df4b675f8a4d18d470b42ce6fa1fcb61ad1ba4ba12d340e54"} Oct 30 16:56:51.413113 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:51.413026 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ff1189-openshift-default-7d9f9f5c65-jjdbw" event={"ID":"ba6262c9-b3ea-4c61-bb84-4f60954d02a0","Type":"ContainerDied","Data":"21622c1501bf04f45dfb293e19e3c8ceb1d47685a3a3d9768d2ea38d70de5447"} Oct 30 16:56:51.413113 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:51.413053 5052 scope.go:122] "RemoveContainer" containerID="4e8706eab0d61e2df4b675f8a4d18d470b42ce6fa1fcb61ad1ba4ba12d340e54" Oct 30 16:56:51.425906 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:51.425887 5052 scope.go:122] "RemoveContainer" containerID="32ccfb376ada51e1c34cb823a8a1cc25e29e84c2523661656fa4b3b14e66ec53" Oct 30 16:56:51.436518 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:51.436497 5052 scope.go:122] "RemoveContainer" containerID="4e8706eab0d61e2df4b675f8a4d18d470b42ce6fa1fcb61ad1ba4ba12d340e54" Oct 30 16:56:51.436801 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:56:51.436780 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e8706eab0d61e2df4b675f8a4d18d470b42ce6fa1fcb61ad1ba4ba12d340e54\": container with ID starting with 4e8706eab0d61e2df4b675f8a4d18d470b42ce6fa1fcb61ad1ba4ba12d340e54 not found: ID does not exist" containerID="4e8706eab0d61e2df4b675f8a4d18d470b42ce6fa1fcb61ad1ba4ba12d340e54" Oct 30 16:56:51.436872 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:51.436809 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e8706eab0d61e2df4b675f8a4d18d470b42ce6fa1fcb61ad1ba4ba12d340e54"} err="failed to get container status \"4e8706eab0d61e2df4b675f8a4d18d470b42ce6fa1fcb61ad1ba4ba12d340e54\": rpc error: code = NotFound desc = could not find container \"4e8706eab0d61e2df4b675f8a4d18d470b42ce6fa1fcb61ad1ba4ba12d340e54\": container with ID starting with 4e8706eab0d61e2df4b675f8a4d18d470b42ce6fa1fcb61ad1ba4ba12d340e54 not found: ID does not exist" Oct 30 16:56:51.436872 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:51.436825 5052 scope.go:122] "RemoveContainer" containerID="32ccfb376ada51e1c34cb823a8a1cc25e29e84c2523661656fa4b3b14e66ec53" Oct 30 16:56:51.437105 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:56:51.437086 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32ccfb376ada51e1c34cb823a8a1cc25e29e84c2523661656fa4b3b14e66ec53\": container with ID starting with 32ccfb376ada51e1c34cb823a8a1cc25e29e84c2523661656fa4b3b14e66ec53 not found: ID does not exist" containerID="32ccfb376ada51e1c34cb823a8a1cc25e29e84c2523661656fa4b3b14e66ec53" Oct 30 16:56:51.437157 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:51.437112 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32ccfb376ada51e1c34cb823a8a1cc25e29e84c2523661656fa4b3b14e66ec53"} err="failed to get container status \"32ccfb376ada51e1c34cb823a8a1cc25e29e84c2523661656fa4b3b14e66ec53\": rpc error: code = NotFound desc = could not find container \"32ccfb376ada51e1c34cb823a8a1cc25e29e84c2523661656fa4b3b14e66ec53\": container with ID starting with 32ccfb376ada51e1c34cb823a8a1cc25e29e84c2523661656fa4b3b14e66ec53 not found: ID does not exist" Oct 30 16:56:51.481041 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:51.481012 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-ff1189-openshift-default-7d9f9f5c65-jjdbw"] Oct 30 16:56:51.493728 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:51.493705 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-ff1189-openshift-default-7d9f9f5c65-jjdbw"] Oct 30 16:56:52.600134 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:52.600098 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba6262c9-b3ea-4c61-bb84-4f60954d02a0" path="/var/lib/kubelet/pods/ba6262c9-b3ea-4c61-bb84-4f60954d02a0/volumes" Oct 30 16:56:52.718511 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:52.718479 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-55x7p"] Oct 30 16:56:52.718725 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:52.718684 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-55x7p" podUID="52ba3acd-4de6-4ae2-8a20-49bf7f966ab0" containerName="istio-proxy" containerID="cri-o://f6e0da38fce207d9afc7ba78924333b92ac36829e21634dccc8f63a00a430af3" gracePeriod=30 Oct 30 16:56:53.358483 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:53.358450 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-3ee682-6ddc5f6db7-wlnkt"] Oct 30 16:56:53.358748 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:53.358707 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3ee682-6ddc5f6db7-wlnkt" podUID="6d254a10-0b28-40b3-8240-e278e39c2f9b" containerName="maas-api" containerID="cri-o://ab703084114e4f6ff8dc76f954a96eed87c72473273e649bad82249a528f3c2e" gracePeriod=30 Oct 30 16:56:54.091606 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:54.090857 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3ee682-6ddc5f6db7-wlnkt" Oct 30 16:56:54.102604 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:54.101432 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d254a10-0b28-40b3-8240-e278e39c2f9b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d254a10-0b28-40b3-8240-e278e39c2f9b-metrics-certs\") pod \"6d254a10-0b28-40b3-8240-e278e39c2f9b\" (UID: \"6d254a10-0b28-40b3-8240-e278e39c2f9b\") " Oct 30 16:56:54.102604 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:54.101525 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d254a10-0b28-40b3-8240-e278e39c2f9b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d254a10-0b28-40b3-8240-e278e39c2f9b-openshift-service-ca\") pod \"6d254a10-0b28-40b3-8240-e278e39c2f9b\" (UID: \"6d254a10-0b28-40b3-8240-e278e39c2f9b\") " Oct 30 16:56:54.102604 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:54.102048 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d254a10-0b28-40b3-8240-e278e39c2f9b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d254a10-0b28-40b3-8240-e278e39c2f9b-maas-api-tls\") pod \"6d254a10-0b28-40b3-8240-e278e39c2f9b\" (UID: \"6d254a10-0b28-40b3-8240-e278e39c2f9b\") " Oct 30 16:56:54.102604 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:54.102104 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d254a10-0b28-40b3-8240-e278e39c2f9b-kube-api-access-vf9z4\" (UniqueName: \"kubernetes.io/projected/6d254a10-0b28-40b3-8240-e278e39c2f9b-kube-api-access-vf9z4\") pod \"6d254a10-0b28-40b3-8240-e278e39c2f9b\" (UID: \"6d254a10-0b28-40b3-8240-e278e39c2f9b\") " Oct 30 16:56:54.103413 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:54.103385 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d254a10-0b28-40b3-8240-e278e39c2f9b-openshift-service-ca" pod "6d254a10-0b28-40b3-8240-e278e39c2f9b" (UID: "6d254a10-0b28-40b3-8240-e278e39c2f9b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:56:54.107769 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:54.107742 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d254a10-0b28-40b3-8240-e278e39c2f9b-maas-api-tls" pod "6d254a10-0b28-40b3-8240-e278e39c2f9b" (UID: "6d254a10-0b28-40b3-8240-e278e39c2f9b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:56:54.108390 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:54.108367 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d254a10-0b28-40b3-8240-e278e39c2f9b-kube-api-access-vf9z4" pod "6d254a10-0b28-40b3-8240-e278e39c2f9b" (UID: "6d254a10-0b28-40b3-8240-e278e39c2f9b"). InnerVolumeSpecName "kube-api-access-vf9z4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:56:54.113312 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:54.113285 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d254a10-0b28-40b3-8240-e278e39c2f9b-metrics-certs" pod "6d254a10-0b28-40b3-8240-e278e39c2f9b" (UID: "6d254a10-0b28-40b3-8240-e278e39c2f9b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:56:54.205346 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:54.205305 5052 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d254a10-0b28-40b3-8240-e278e39c2f9b-metrics-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:56:54.205346 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:54.205344 5052 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d254a10-0b28-40b3-8240-e278e39c2f9b-openshift-service-ca\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:56:54.205653 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:54.205360 5052 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d254a10-0b28-40b3-8240-e278e39c2f9b-maas-api-tls\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:56:54.205653 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:54.205374 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vf9z4\" (UniqueName: \"kubernetes.io/projected/6d254a10-0b28-40b3-8240-e278e39c2f9b-kube-api-access-vf9z4\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:56:54.447030 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:54.446987 5052 generic.go:356] "Generic (PLEG): container finished" podID="5cbbc24b-bf64-455e-9dc2-c34487840dfb" containerID="321c963f3ddeb9e3048bc683bc1bd9dde745780526526d048f87684b21d0146f" exitCode=0 Oct 30 16:56:54.447235 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:54.447058 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ff1189-759ff754b7-gl6mj" event={"ID":"5cbbc24b-bf64-455e-9dc2-c34487840dfb","Type":"ContainerDied","Data":"321c963f3ddeb9e3048bc683bc1bd9dde745780526526d048f87684b21d0146f"} Oct 30 16:56:54.449022 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:54.448992 5052 generic.go:356] "Generic (PLEG): container finished" podID="6d254a10-0b28-40b3-8240-e278e39c2f9b" containerID="ab703084114e4f6ff8dc76f954a96eed87c72473273e649bad82249a528f3c2e" exitCode=0 Oct 30 16:56:54.449206 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:54.449036 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3ee682-6ddc5f6db7-wlnkt" Oct 30 16:56:54.449206 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:54.449073 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3ee682-6ddc5f6db7-wlnkt" event={"ID":"6d254a10-0b28-40b3-8240-e278e39c2f9b","Type":"ContainerDied","Data":"ab703084114e4f6ff8dc76f954a96eed87c72473273e649bad82249a528f3c2e"} Oct 30 16:56:54.449206 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:54.449138 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3ee682-6ddc5f6db7-wlnkt" event={"ID":"6d254a10-0b28-40b3-8240-e278e39c2f9b","Type":"ContainerDied","Data":"e2ccd3c3a48c9460aed0444aa83d905b4b1846f0cb47b7e84224d638f762e424"} Oct 30 16:56:54.449206 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:54.449172 5052 scope.go:122] "RemoveContainer" containerID="ab703084114e4f6ff8dc76f954a96eed87c72473273e649bad82249a528f3c2e" Oct 30 16:56:54.466274 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:54.466248 5052 scope.go:122] "RemoveContainer" containerID="ab703084114e4f6ff8dc76f954a96eed87c72473273e649bad82249a528f3c2e" Oct 30 16:56:54.466556 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:56:54.466533 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab703084114e4f6ff8dc76f954a96eed87c72473273e649bad82249a528f3c2e\": container with ID starting with ab703084114e4f6ff8dc76f954a96eed87c72473273e649bad82249a528f3c2e not found: ID does not exist" containerID="ab703084114e4f6ff8dc76f954a96eed87c72473273e649bad82249a528f3c2e" Oct 30 16:56:54.466684 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:54.466563 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab703084114e4f6ff8dc76f954a96eed87c72473273e649bad82249a528f3c2e"} err="failed to get container status \"ab703084114e4f6ff8dc76f954a96eed87c72473273e649bad82249a528f3c2e\": rpc error: code = NotFound desc = could not find container \"ab703084114e4f6ff8dc76f954a96eed87c72473273e649bad82249a528f3c2e\": container with ID starting with ab703084114e4f6ff8dc76f954a96eed87c72473273e649bad82249a528f3c2e not found: ID does not exist" Oct 30 16:56:54.613216 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:54.611687 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-3ee682-6ddc5f6db7-wlnkt"] Oct 30 16:56:54.614822 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:54.614790 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-3ee682-6ddc5f6db7-wlnkt"] Oct 30 16:56:54.623847 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:54.623826 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-ea77d8/test-model-ea77d8-kserve-7f798c774c-tbxxl"] Oct 30 16:56:54.625514 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:54.625496 5052 cpu_manager.go:414] "containerMap: removing container" podUID="6d254a10-0b28-40b3-8240-e278e39c2f9b" containerName="maas-api" Oct 30 16:56:54.625637 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:54.625627 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d254a10-0b28-40b3-8240-e278e39c2f9b" containerName="maas-api" Oct 30 16:56:54.625720 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:54.625712 5052 cpu_manager.go:414] "containerMap: removing container" podUID="ba6262c9-b3ea-4c61-bb84-4f60954d02a0" containerName="istio-proxy" Oct 30 16:56:54.625778 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:54.625771 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba6262c9-b3ea-4c61-bb84-4f60954d02a0" containerName="istio-proxy" Oct 30 16:56:54.625836 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:54.625830 5052 cpu_manager.go:414] "containerMap: removing container" podUID="ba6262c9-b3ea-4c61-bb84-4f60954d02a0" containerName="istio-proxy" Oct 30 16:56:54.625927 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:54.625920 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba6262c9-b3ea-4c61-bb84-4f60954d02a0" containerName="istio-proxy" Oct 30 16:56:54.625980 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:54.625974 5052 cpu_manager.go:414] "containerMap: removing container" podUID="a4e168d8-c482-4fdf-9d3c-aa41fd252c90" containerName="main" Oct 30 16:56:54.626028 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:54.626020 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4e168d8-c482-4fdf-9d3c-aa41fd252c90" containerName="main" Oct 30 16:56:54.626294 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:54.626281 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d254a10-0b28-40b3-8240-e278e39c2f9b" containerName="maas-api" Oct 30 16:56:54.626375 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:54.626368 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba6262c9-b3ea-4c61-bb84-4f60954d02a0" containerName="istio-proxy" Oct 30 16:56:54.626431 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:54.626425 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba6262c9-b3ea-4c61-bb84-4f60954d02a0" containerName="istio-proxy" Oct 30 16:56:54.626480 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:54.626474 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4e168d8-c482-4fdf-9d3c-aa41fd252c90" containerName="main" Oct 30 16:56:54.632567 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:54.632551 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ea77d8/test-model-ea77d8-kserve-7f798c774c-tbxxl" Oct 30 16:56:54.645021 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:54.644995 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67ba1b17-147c-4b27-b2bd-9660840a9778-home\") pod \"test-model-ea77d8-kserve-7f798c774c-tbxxl\" (UID: \"67ba1b17-147c-4b27-b2bd-9660840a9778\") " pod="ai-tenant-e2e-inf-a-ea77d8/test-model-ea77d8-kserve-7f798c774c-tbxxl" Oct 30 16:56:54.645474 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:54.645456 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-ea77d8/test-model-ea77d8-kserve-7f798c774c-tbxxl"] Oct 30 16:56:54.645616 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:54.645468 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsb74\" (UniqueName: \"kubernetes.io/projected/67ba1b17-147c-4b27-b2bd-9660840a9778-kube-api-access-vsb74\") pod \"test-model-ea77d8-kserve-7f798c774c-tbxxl\" (UID: \"67ba1b17-147c-4b27-b2bd-9660840a9778\") " pod="ai-tenant-e2e-inf-a-ea77d8/test-model-ea77d8-kserve-7f798c774c-tbxxl" Oct 30 16:56:54.645772 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:54.645755 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67ba1b17-147c-4b27-b2bd-9660840a9778-model-cache\") pod \"test-model-ea77d8-kserve-7f798c774c-tbxxl\" (UID: \"67ba1b17-147c-4b27-b2bd-9660840a9778\") " pod="ai-tenant-e2e-inf-a-ea77d8/test-model-ea77d8-kserve-7f798c774c-tbxxl" Oct 30 16:56:54.645909 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:54.645892 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67ba1b17-147c-4b27-b2bd-9660840a9778-tls-certs\") pod \"test-model-ea77d8-kserve-7f798c774c-tbxxl\" (UID: \"67ba1b17-147c-4b27-b2bd-9660840a9778\") " pod="ai-tenant-e2e-inf-a-ea77d8/test-model-ea77d8-kserve-7f798c774c-tbxxl" Oct 30 16:56:54.646483 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:54.646462 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67ba1b17-147c-4b27-b2bd-9660840a9778-dshm\") pod \"test-model-ea77d8-kserve-7f798c774c-tbxxl\" (UID: \"67ba1b17-147c-4b27-b2bd-9660840a9778\") " pod="ai-tenant-e2e-inf-a-ea77d8/test-model-ea77d8-kserve-7f798c774c-tbxxl" Oct 30 16:56:54.646704 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:54.646685 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67ba1b17-147c-4b27-b2bd-9660840a9778-tmp-dir\") pod \"test-model-ea77d8-kserve-7f798c774c-tbxxl\" (UID: \"67ba1b17-147c-4b27-b2bd-9660840a9778\") " pod="ai-tenant-e2e-inf-a-ea77d8/test-model-ea77d8-kserve-7f798c774c-tbxxl" Oct 30 16:56:54.649409 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:54.649091 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-ea77d8\"/\"test-model-ea77d8-kserve-self-signed-certs\"" Oct 30 16:56:54.649409 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:54.649245 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-ea77d8\"/\"openshift-service-ca.crt\"" Oct 30 16:56:54.649409 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:54.649284 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-ea77d8\"/\"kube-root-ca.crt\"" Oct 30 16:56:54.649409 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:54.649379 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-ea77d8\"/\"default-dockercfg-58mck\"" Oct 30 16:56:54.688780 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:54.688752 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ff1189-759ff754b7-gl6mj" Oct 30 16:56:54.748372 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:54.748342 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5cbbc24b-bf64-455e-9dc2-c34487840dfb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cbbc24b-bf64-455e-9dc2-c34487840dfb-plugins-config-volume\") pod \"5cbbc24b-bf64-455e-9dc2-c34487840dfb\" (UID: \"5cbbc24b-bf64-455e-9dc2-c34487840dfb\") " Oct 30 16:56:54.748542 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:54.748462 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cbbc24b-bf64-455e-9dc2-c34487840dfb-kube-api-access-n4vfj\" (UniqueName: \"kubernetes.io/projected/5cbbc24b-bf64-455e-9dc2-c34487840dfb-kube-api-access-n4vfj\") pod \"5cbbc24b-bf64-455e-9dc2-c34487840dfb\" (UID: \"5cbbc24b-bf64-455e-9dc2-c34487840dfb\") " Oct 30 16:56:54.748776 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:54.748753 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67ba1b17-147c-4b27-b2bd-9660840a9778-home\") pod \"test-model-ea77d8-kserve-7f798c774c-tbxxl\" (UID: \"67ba1b17-147c-4b27-b2bd-9660840a9778\") " pod="ai-tenant-e2e-inf-a-ea77d8/test-model-ea77d8-kserve-7f798c774c-tbxxl" Oct 30 16:56:54.748893 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:54.748825 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cbbc24b-bf64-455e-9dc2-c34487840dfb-plugins-config-volume" pod "5cbbc24b-bf64-455e-9dc2-c34487840dfb" (UID: "5cbbc24b-bf64-455e-9dc2-c34487840dfb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:56:54.749002 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:54.748965 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsb74\" (UniqueName: \"kubernetes.io/projected/67ba1b17-147c-4b27-b2bd-9660840a9778-kube-api-access-vsb74\") pod \"test-model-ea77d8-kserve-7f798c774c-tbxxl\" (UID: \"67ba1b17-147c-4b27-b2bd-9660840a9778\") " pod="ai-tenant-e2e-inf-a-ea77d8/test-model-ea77d8-kserve-7f798c774c-tbxxl" Oct 30 16:56:54.749089 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:54.749026 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67ba1b17-147c-4b27-b2bd-9660840a9778-model-cache\") pod \"test-model-ea77d8-kserve-7f798c774c-tbxxl\" (UID: \"67ba1b17-147c-4b27-b2bd-9660840a9778\") " pod="ai-tenant-e2e-inf-a-ea77d8/test-model-ea77d8-kserve-7f798c774c-tbxxl" Oct 30 16:56:54.749089 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:54.749070 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67ba1b17-147c-4b27-b2bd-9660840a9778-tls-certs\") pod \"test-model-ea77d8-kserve-7f798c774c-tbxxl\" (UID: \"67ba1b17-147c-4b27-b2bd-9660840a9778\") " pod="ai-tenant-e2e-inf-a-ea77d8/test-model-ea77d8-kserve-7f798c774c-tbxxl" Oct 30 16:56:54.749204 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:54.749124 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67ba1b17-147c-4b27-b2bd-9660840a9778-dshm\") pod \"test-model-ea77d8-kserve-7f798c774c-tbxxl\" (UID: \"67ba1b17-147c-4b27-b2bd-9660840a9778\") " pod="ai-tenant-e2e-inf-a-ea77d8/test-model-ea77d8-kserve-7f798c774c-tbxxl" Oct 30 16:56:54.749204 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:54.749155 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67ba1b17-147c-4b27-b2bd-9660840a9778-home\") pod \"test-model-ea77d8-kserve-7f798c774c-tbxxl\" (UID: \"67ba1b17-147c-4b27-b2bd-9660840a9778\") " pod="ai-tenant-e2e-inf-a-ea77d8/test-model-ea77d8-kserve-7f798c774c-tbxxl" Oct 30 16:56:54.749305 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:54.749204 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67ba1b17-147c-4b27-b2bd-9660840a9778-tmp-dir\") pod \"test-model-ea77d8-kserve-7f798c774c-tbxxl\" (UID: \"67ba1b17-147c-4b27-b2bd-9660840a9778\") " pod="ai-tenant-e2e-inf-a-ea77d8/test-model-ea77d8-kserve-7f798c774c-tbxxl" Oct 30 16:56:54.749429 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:54.749408 5052 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cbbc24b-bf64-455e-9dc2-c34487840dfb-plugins-config-volume\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:56:54.749740 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:54.749715 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67ba1b17-147c-4b27-b2bd-9660840a9778-tmp-dir\") pod \"test-model-ea77d8-kserve-7f798c774c-tbxxl\" (UID: \"67ba1b17-147c-4b27-b2bd-9660840a9778\") " pod="ai-tenant-e2e-inf-a-ea77d8/test-model-ea77d8-kserve-7f798c774c-tbxxl" Oct 30 16:56:54.749865 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:54.749779 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67ba1b17-147c-4b27-b2bd-9660840a9778-model-cache\") pod \"test-model-ea77d8-kserve-7f798c774c-tbxxl\" (UID: \"67ba1b17-147c-4b27-b2bd-9660840a9778\") " pod="ai-tenant-e2e-inf-a-ea77d8/test-model-ea77d8-kserve-7f798c774c-tbxxl" Oct 30 16:56:54.751311 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:54.751288 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cbbc24b-bf64-455e-9dc2-c34487840dfb-kube-api-access-n4vfj" pod "5cbbc24b-bf64-455e-9dc2-c34487840dfb" (UID: "5cbbc24b-bf64-455e-9dc2-c34487840dfb"). InnerVolumeSpecName "kube-api-access-n4vfj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:56:54.751546 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:54.751530 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67ba1b17-147c-4b27-b2bd-9660840a9778-dshm\") pod \"test-model-ea77d8-kserve-7f798c774c-tbxxl\" (UID: \"67ba1b17-147c-4b27-b2bd-9660840a9778\") " pod="ai-tenant-e2e-inf-a-ea77d8/test-model-ea77d8-kserve-7f798c774c-tbxxl" Oct 30 16:56:54.753125 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:54.753101 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67ba1b17-147c-4b27-b2bd-9660840a9778-tls-certs\") pod \"test-model-ea77d8-kserve-7f798c774c-tbxxl\" (UID: \"67ba1b17-147c-4b27-b2bd-9660840a9778\") " pod="ai-tenant-e2e-inf-a-ea77d8/test-model-ea77d8-kserve-7f798c774c-tbxxl" Oct 30 16:56:54.777597 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:54.774920 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsb74\" (UniqueName: \"kubernetes.io/projected/67ba1b17-147c-4b27-b2bd-9660840a9778-kube-api-access-vsb74\") pod \"test-model-ea77d8-kserve-7f798c774c-tbxxl\" (UID: \"67ba1b17-147c-4b27-b2bd-9660840a9778\") " pod="ai-tenant-e2e-inf-a-ea77d8/test-model-ea77d8-kserve-7f798c774c-tbxxl" Oct 30 16:56:54.883666 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:54.883370 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4vfj\" (UniqueName: \"kubernetes.io/projected/5cbbc24b-bf64-455e-9dc2-c34487840dfb-kube-api-access-n4vfj\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:56:55.016607 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:55.001557 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ea77d8/test-model-ea77d8-kserve-7f798c774c-tbxxl" Oct 30 16:56:55.390329 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:55.390301 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-ea77d8/test-model-ea77d8-kserve-7f798c774c-tbxxl"] Oct 30 16:56:55.391113 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:56:55.391088 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod67ba1b17_147c_4b27_b2bd_9660840a9778.slice/crio-eb9672e795462d24c9aa077aad92f4e9eb3b79eceda0f9b85f54165c35a07f8a WatchSource:0}: Error finding container eb9672e795462d24c9aa077aad92f4e9eb3b79eceda0f9b85f54165c35a07f8a: Status 404 returned error can't find the container with id eb9672e795462d24c9aa077aad92f4e9eb3b79eceda0f9b85f54165c35a07f8a Oct 30 16:56:55.466683 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:55.459369 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ff1189-759ff754b7-gl6mj" Oct 30 16:56:55.466683 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:55.459367 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ff1189-759ff754b7-gl6mj" event={"ID":"5cbbc24b-bf64-455e-9dc2-c34487840dfb","Type":"ContainerDied","Data":"c3696052e4802b084352abf0e87dc286a0bc458cef025c67f0b72ffe6804f9b2"} Oct 30 16:56:55.466683 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:55.459508 5052 scope.go:122] "RemoveContainer" containerID="321c963f3ddeb9e3048bc683bc1bd9dde745780526526d048f87684b21d0146f" Oct 30 16:56:55.466683 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:55.462424 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ea77d8/test-model-ea77d8-kserve-7f798c774c-tbxxl" event={"ID":"67ba1b17-147c-4b27-b2bd-9660840a9778","Type":"ContainerStarted","Data":"eb9672e795462d24c9aa077aad92f4e9eb3b79eceda0f9b85f54165c35a07f8a"} Oct 30 16:56:55.518952 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:55.518901 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ff1189-759ff754b7-gl6mj"] Oct 30 16:56:55.543083 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:55.543046 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ff1189-759ff754b7-gl6mj"] Oct 30 16:56:56.473043 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:56.473012 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ea77d8/test-model-ea77d8-kserve-7f798c774c-tbxxl" event={"ID":"67ba1b17-147c-4b27-b2bd-9660840a9778","Type":"ContainerStarted","Data":"ded388af59dacf976347a1cdb510b8d63807af86594aea32f40a2e10be67f2e0"} Oct 30 16:56:56.520797 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:56.520727 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-ea77d8/test-model-ea77d8-kserve-7f798c774c-tbxxl" podStartSLOduration=2.520710782 podStartE2EDuration="2.520710782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:56:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:56:56.517287092 +0000 UTC m=+2172.044622312" watchObservedRunningTime="2026-10-30 16:56:56.520710782 +0000 UTC m=+2172.048046002" Oct 30 16:56:56.600948 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:56.600918 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5cbbc24b-bf64-455e-9dc2-c34487840dfb" path="/var/lib/kubelet/pods/5cbbc24b-bf64-455e-9dc2-c34487840dfb/volumes" Oct 30 16:56:56.601411 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:56.601396 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d254a10-0b28-40b3-8240-e278e39c2f9b" path="/var/lib/kubelet/pods/6d254a10-0b28-40b3-8240-e278e39c2f9b/volumes" Oct 30 16:56:56.787251 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:56.787159 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-480a60-7c447cb756-6ct9j"] Oct 30 16:56:57.002126 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:57.002095 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-ea77d8/test-model-ea77d8-kserve-7f798c774c-tbxxl" Oct 30 16:56:58.454248 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:58.454217 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-55x7p" Oct 30 16:56:58.558770 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:58.558683 5052 generic.go:356] "Generic (PLEG): container finished" podID="52ba3acd-4de6-4ae2-8a20-49bf7f966ab0" containerID="f6e0da38fce207d9afc7ba78924333b92ac36829e21634dccc8f63a00a430af3" exitCode=0 Oct 30 16:56:58.558770 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:58.558728 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-55x7p" Oct 30 16:56:58.559011 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:58.558772 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-55x7p" event={"ID":"52ba3acd-4de6-4ae2-8a20-49bf7f966ab0","Type":"ContainerDied","Data":"f6e0da38fce207d9afc7ba78924333b92ac36829e21634dccc8f63a00a430af3"} Oct 30 16:56:58.559011 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:58.558828 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-55x7p" event={"ID":"52ba3acd-4de6-4ae2-8a20-49bf7f966ab0","Type":"ContainerDied","Data":"7a4f34ccca927aeb562bab8452a807c6e248cd8a012a113aac5c8b26e2ec2fdf"} Oct 30 16:56:58.559011 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:58.558859 5052 scope.go:122] "RemoveContainer" containerID="f6e0da38fce207d9afc7ba78924333b92ac36829e21634dccc8f63a00a430af3" Oct 30 16:56:58.578840 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:58.578789 5052 scope.go:122] "RemoveContainer" containerID="f6e0da38fce207d9afc7ba78924333b92ac36829e21634dccc8f63a00a430af3" Oct 30 16:56:58.579150 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:56:58.579129 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6e0da38fce207d9afc7ba78924333b92ac36829e21634dccc8f63a00a430af3\": container with ID starting with f6e0da38fce207d9afc7ba78924333b92ac36829e21634dccc8f63a00a430af3 not found: ID does not exist" containerID="f6e0da38fce207d9afc7ba78924333b92ac36829e21634dccc8f63a00a430af3" Oct 30 16:56:58.579311 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:58.579282 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6e0da38fce207d9afc7ba78924333b92ac36829e21634dccc8f63a00a430af3"} err="failed to get container status \"f6e0da38fce207d9afc7ba78924333b92ac36829e21634dccc8f63a00a430af3\": rpc error: code = NotFound desc = could not find container \"f6e0da38fce207d9afc7ba78924333b92ac36829e21634dccc8f63a00a430af3\": container with ID starting with f6e0da38fce207d9afc7ba78924333b92ac36829e21634dccc8f63a00a430af3 not found: ID does not exist" Oct 30 16:56:58.579421 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:58.579192 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52ba3acd-4de6-4ae2-8a20-49bf7f966ab0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52ba3acd-4de6-4ae2-8a20-49bf7f966ab0-istio-data\") pod \"52ba3acd-4de6-4ae2-8a20-49bf7f966ab0\" (UID: \"52ba3acd-4de6-4ae2-8a20-49bf7f966ab0\") " Oct 30 16:56:58.579541 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:58.579523 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52ba3acd-4de6-4ae2-8a20-49bf7f966ab0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52ba3acd-4de6-4ae2-8a20-49bf7f966ab0-workload-certs\") pod \"52ba3acd-4de6-4ae2-8a20-49bf7f966ab0\" (UID: \"52ba3acd-4de6-4ae2-8a20-49bf7f966ab0\") " Oct 30 16:56:58.579638 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:58.579411 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52ba3acd-4de6-4ae2-8a20-49bf7f966ab0-istio-data" pod "52ba3acd-4de6-4ae2-8a20-49bf7f966ab0" (UID: "52ba3acd-4de6-4ae2-8a20-49bf7f966ab0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:56:58.579638 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:58.579561 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52ba3acd-4de6-4ae2-8a20-49bf7f966ab0-kube-api-access-7l9bl\" (UniqueName: \"kubernetes.io/projected/52ba3acd-4de6-4ae2-8a20-49bf7f966ab0-kube-api-access-7l9bl\") pod \"52ba3acd-4de6-4ae2-8a20-49bf7f966ab0\" (UID: \"52ba3acd-4de6-4ae2-8a20-49bf7f966ab0\") " Oct 30 16:56:58.579786 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:58.579634 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52ba3acd-4de6-4ae2-8a20-49bf7f966ab0-istio-token\" (UniqueName: \"kubernetes.io/projected/52ba3acd-4de6-4ae2-8a20-49bf7f966ab0-istio-token\") pod \"52ba3acd-4de6-4ae2-8a20-49bf7f966ab0\" (UID: \"52ba3acd-4de6-4ae2-8a20-49bf7f966ab0\") " Oct 30 16:56:58.579786 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:58.579692 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52ba3acd-4de6-4ae2-8a20-49bf7f966ab0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52ba3acd-4de6-4ae2-8a20-49bf7f966ab0-istio-envoy\") pod \"52ba3acd-4de6-4ae2-8a20-49bf7f966ab0\" (UID: \"52ba3acd-4de6-4ae2-8a20-49bf7f966ab0\") " Oct 30 16:56:58.579888 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:58.579798 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52ba3acd-4de6-4ae2-8a20-49bf7f966ab0-workload-certs" pod "52ba3acd-4de6-4ae2-8a20-49bf7f966ab0" (UID: "52ba3acd-4de6-4ae2-8a20-49bf7f966ab0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:56:58.579945 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:58.579887 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52ba3acd-4de6-4ae2-8a20-49bf7f966ab0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52ba3acd-4de6-4ae2-8a20-49bf7f966ab0-istiod-ca-cert\") pod \"52ba3acd-4de6-4ae2-8a20-49bf7f966ab0\" (UID: \"52ba3acd-4de6-4ae2-8a20-49bf7f966ab0\") " Oct 30 16:56:58.579945 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:58.579935 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/52ba3acd-4de6-4ae2-8a20-49bf7f966ab0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52ba3acd-4de6-4ae2-8a20-49bf7f966ab0-istio-podinfo\") pod \"52ba3acd-4de6-4ae2-8a20-49bf7f966ab0\" (UID: \"52ba3acd-4de6-4ae2-8a20-49bf7f966ab0\") " Oct 30 16:56:58.580092 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:58.579989 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52ba3acd-4de6-4ae2-8a20-49bf7f966ab0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52ba3acd-4de6-4ae2-8a20-49bf7f966ab0-credential-socket\") pod \"52ba3acd-4de6-4ae2-8a20-49bf7f966ab0\" (UID: \"52ba3acd-4de6-4ae2-8a20-49bf7f966ab0\") " Oct 30 16:56:58.580092 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:58.580063 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52ba3acd-4de6-4ae2-8a20-49bf7f966ab0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52ba3acd-4de6-4ae2-8a20-49bf7f966ab0-workload-socket\") pod \"52ba3acd-4de6-4ae2-8a20-49bf7f966ab0\" (UID: \"52ba3acd-4de6-4ae2-8a20-49bf7f966ab0\") " Oct 30 16:56:58.580785 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:58.580760 5052 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52ba3acd-4de6-4ae2-8a20-49bf7f966ab0-istio-data\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:56:58.580888 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:58.580791 5052 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52ba3acd-4de6-4ae2-8a20-49bf7f966ab0-workload-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:56:58.581173 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:58.581108 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52ba3acd-4de6-4ae2-8a20-49bf7f966ab0-workload-socket" pod "52ba3acd-4de6-4ae2-8a20-49bf7f966ab0" (UID: "52ba3acd-4de6-4ae2-8a20-49bf7f966ab0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:56:58.581647 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:58.581619 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52ba3acd-4de6-4ae2-8a20-49bf7f966ab0-istiod-ca-cert" pod "52ba3acd-4de6-4ae2-8a20-49bf7f966ab0" (UID: "52ba3acd-4de6-4ae2-8a20-49bf7f966ab0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:56:58.582227 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:58.582195 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52ba3acd-4de6-4ae2-8a20-49bf7f966ab0-credential-socket" pod "52ba3acd-4de6-4ae2-8a20-49bf7f966ab0" (UID: "52ba3acd-4de6-4ae2-8a20-49bf7f966ab0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:56:58.582612 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:58.582539 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52ba3acd-4de6-4ae2-8a20-49bf7f966ab0-kube-api-access-7l9bl" pod "52ba3acd-4de6-4ae2-8a20-49bf7f966ab0" (UID: "52ba3acd-4de6-4ae2-8a20-49bf7f966ab0"). InnerVolumeSpecName "kube-api-access-7l9bl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:56:58.583928 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:58.583896 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/52ba3acd-4de6-4ae2-8a20-49bf7f966ab0-istio-podinfo" pod "52ba3acd-4de6-4ae2-8a20-49bf7f966ab0" (UID: "52ba3acd-4de6-4ae2-8a20-49bf7f966ab0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:56:58.583928 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:58.583913 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52ba3acd-4de6-4ae2-8a20-49bf7f966ab0-istio-token" pod "52ba3acd-4de6-4ae2-8a20-49bf7f966ab0" (UID: "52ba3acd-4de6-4ae2-8a20-49bf7f966ab0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:56:58.584114 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:58.583992 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52ba3acd-4de6-4ae2-8a20-49bf7f966ab0-istio-envoy" pod "52ba3acd-4de6-4ae2-8a20-49bf7f966ab0" (UID: "52ba3acd-4de6-4ae2-8a20-49bf7f966ab0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:56:58.682859 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:58.682819 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7l9bl\" (UniqueName: \"kubernetes.io/projected/52ba3acd-4de6-4ae2-8a20-49bf7f966ab0-kube-api-access-7l9bl\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:56:58.682859 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:58.682849 5052 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52ba3acd-4de6-4ae2-8a20-49bf7f966ab0-istio-token\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:56:58.683072 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:58.682878 5052 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52ba3acd-4de6-4ae2-8a20-49bf7f966ab0-istio-envoy\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:56:58.683072 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:58.682901 5052 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52ba3acd-4de6-4ae2-8a20-49bf7f966ab0-istiod-ca-cert\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:56:58.683072 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:58.682916 5052 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52ba3acd-4de6-4ae2-8a20-49bf7f966ab0-istio-podinfo\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:56:58.683072 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:58.682931 5052 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52ba3acd-4de6-4ae2-8a20-49bf7f966ab0-credential-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:56:58.683072 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:58.682946 5052 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52ba3acd-4de6-4ae2-8a20-49bf7f966ab0-workload-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:56:58.906237 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:58.906211 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-55x7p"] Oct 30 16:56:58.919000 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:56:58.918973 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-55x7p"] Oct 30 16:57:00.599721 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:00.599681 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52ba3acd-4de6-4ae2-8a20-49bf7f966ab0" path="/var/lib/kubelet/pods/52ba3acd-4de6-4ae2-8a20-49bf7f966ab0/volumes" Oct 30 16:57:01.736466 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:01.736433 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-3ee682-openshift-default-5f7887cb79-6tsdx"] Oct 30 16:57:01.736899 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:01.736654 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-3ee682-openshift-default-5f7887cb79-6tsdx" podUID="fe851cb7-f6e1-4baf-b75e-69018da63e35" containerName="istio-proxy" containerID="cri-o://e7214d0e42e8d6965f3d213fc9d90a7c8d038188814f10218c5a77a5a9a8b6d2" gracePeriod=30 Oct 30 16:57:02.548139 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:02.548092 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-480a60-7c447cb756-6ct9j" podUID="3b3da2b1-c132-453a-b8e9-512a43e613ff" containerName="payload-processing" containerID="cri-o://945f122630787d3d009e107bdd51c4b40193e6395f1aacc12a034321f9458c32" gracePeriod=25 Oct 30 16:57:05.008128 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:05.008092 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-ea77d8/test-model-ea77d8-kserve-7f798c774c-tbxxl" Oct 30 16:57:05.020764 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:05.020733 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-ea77d8/test-model-ea77d8-kserve-7f798c774c-tbxxl" Oct 30 16:57:05.628894 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:05.628860 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-ea77d8/test-model-ea77d8-kserve-7f798c774c-tbxxl" Oct 30 16:57:07.067434 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:07.067398 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3ee682-openshift-default-5f7887cb79-6tsdx" Oct 30 16:57:07.171434 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:07.171399 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe851cb7-f6e1-4baf-b75e-69018da63e35-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe851cb7-f6e1-4baf-b75e-69018da63e35-workload-certs\") pod \"fe851cb7-f6e1-4baf-b75e-69018da63e35\" (UID: \"fe851cb7-f6e1-4baf-b75e-69018da63e35\") " Oct 30 16:57:07.171625 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:07.171471 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe851cb7-f6e1-4baf-b75e-69018da63e35-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe851cb7-f6e1-4baf-b75e-69018da63e35-istiod-ca-cert\") pod \"fe851cb7-f6e1-4baf-b75e-69018da63e35\" (UID: \"fe851cb7-f6e1-4baf-b75e-69018da63e35\") " Oct 30 16:57:07.171625 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:07.171559 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe851cb7-f6e1-4baf-b75e-69018da63e35-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe851cb7-f6e1-4baf-b75e-69018da63e35-workload-socket\") pod \"fe851cb7-f6e1-4baf-b75e-69018da63e35\" (UID: \"fe851cb7-f6e1-4baf-b75e-69018da63e35\") " Oct 30 16:57:07.171771 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:07.171671 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe851cb7-f6e1-4baf-b75e-69018da63e35-istio-token\" (UniqueName: \"kubernetes.io/projected/fe851cb7-f6e1-4baf-b75e-69018da63e35-istio-token\") pod \"fe851cb7-f6e1-4baf-b75e-69018da63e35\" (UID: \"fe851cb7-f6e1-4baf-b75e-69018da63e35\") " Oct 30 16:57:07.171771 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:07.171683 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe851cb7-f6e1-4baf-b75e-69018da63e35-workload-certs" pod "fe851cb7-f6e1-4baf-b75e-69018da63e35" (UID: "fe851cb7-f6e1-4baf-b75e-69018da63e35"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:57:07.171771 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:07.171727 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe851cb7-f6e1-4baf-b75e-69018da63e35-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe851cb7-f6e1-4baf-b75e-69018da63e35-credential-socket\") pod \"fe851cb7-f6e1-4baf-b75e-69018da63e35\" (UID: \"fe851cb7-f6e1-4baf-b75e-69018da63e35\") " Oct 30 16:57:07.171771 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:07.171753 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe851cb7-f6e1-4baf-b75e-69018da63e35-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe851cb7-f6e1-4baf-b75e-69018da63e35-istio-envoy\") pod \"fe851cb7-f6e1-4baf-b75e-69018da63e35\" (UID: \"fe851cb7-f6e1-4baf-b75e-69018da63e35\") " Oct 30 16:57:07.171978 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:07.171860 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe851cb7-f6e1-4baf-b75e-69018da63e35-workload-socket" pod "fe851cb7-f6e1-4baf-b75e-69018da63e35" (UID: "fe851cb7-f6e1-4baf-b75e-69018da63e35"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:57:07.172038 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:07.171989 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe851cb7-f6e1-4baf-b75e-69018da63e35-kube-api-access-t8hgk\" (UniqueName: \"kubernetes.io/projected/fe851cb7-f6e1-4baf-b75e-69018da63e35-kube-api-access-t8hgk\") pod \"fe851cb7-f6e1-4baf-b75e-69018da63e35\" (UID: \"fe851cb7-f6e1-4baf-b75e-69018da63e35\") " Oct 30 16:57:07.172038 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:07.172005 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe851cb7-f6e1-4baf-b75e-69018da63e35-credential-socket" pod "fe851cb7-f6e1-4baf-b75e-69018da63e35" (UID: "fe851cb7-f6e1-4baf-b75e-69018da63e35"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:57:07.172126 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:07.172035 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe851cb7-f6e1-4baf-b75e-69018da63e35-istiod-ca-cert" pod "fe851cb7-f6e1-4baf-b75e-69018da63e35" (UID: "fe851cb7-f6e1-4baf-b75e-69018da63e35"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:57:07.172187 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:07.172133 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe851cb7-f6e1-4baf-b75e-69018da63e35-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe851cb7-f6e1-4baf-b75e-69018da63e35-istio-data\") pod \"fe851cb7-f6e1-4baf-b75e-69018da63e35\" (UID: \"fe851cb7-f6e1-4baf-b75e-69018da63e35\") " Oct 30 16:57:07.172187 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:07.172176 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fe851cb7-f6e1-4baf-b75e-69018da63e35-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe851cb7-f6e1-4baf-b75e-69018da63e35-istio-podinfo\") pod \"fe851cb7-f6e1-4baf-b75e-69018da63e35\" (UID: \"fe851cb7-f6e1-4baf-b75e-69018da63e35\") " Oct 30 16:57:07.172473 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:07.172445 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe851cb7-f6e1-4baf-b75e-69018da63e35-istio-data" pod "fe851cb7-f6e1-4baf-b75e-69018da63e35" (UID: "fe851cb7-f6e1-4baf-b75e-69018da63e35"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:57:07.173170 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:07.173139 5052 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe851cb7-f6e1-4baf-b75e-69018da63e35-istio-data\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:57:07.173280 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:07.173175 5052 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe851cb7-f6e1-4baf-b75e-69018da63e35-workload-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:57:07.173280 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:07.173189 5052 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe851cb7-f6e1-4baf-b75e-69018da63e35-istiod-ca-cert\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:57:07.173280 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:07.173202 5052 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe851cb7-f6e1-4baf-b75e-69018da63e35-workload-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:57:07.173280 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:07.173217 5052 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe851cb7-f6e1-4baf-b75e-69018da63e35-credential-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:57:07.174259 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:07.174235 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe851cb7-f6e1-4baf-b75e-69018da63e35-istio-envoy" pod "fe851cb7-f6e1-4baf-b75e-69018da63e35" (UID: "fe851cb7-f6e1-4baf-b75e-69018da63e35"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:57:07.174406 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:07.174382 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe851cb7-f6e1-4baf-b75e-69018da63e35-istio-token" pod "fe851cb7-f6e1-4baf-b75e-69018da63e35" (UID: "fe851cb7-f6e1-4baf-b75e-69018da63e35"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:57:07.174640 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:07.174617 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe851cb7-f6e1-4baf-b75e-69018da63e35-kube-api-access-t8hgk" pod "fe851cb7-f6e1-4baf-b75e-69018da63e35" (UID: "fe851cb7-f6e1-4baf-b75e-69018da63e35"). InnerVolumeSpecName "kube-api-access-t8hgk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:57:07.174754 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:07.174718 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fe851cb7-f6e1-4baf-b75e-69018da63e35-istio-podinfo" pod "fe851cb7-f6e1-4baf-b75e-69018da63e35" (UID: "fe851cb7-f6e1-4baf-b75e-69018da63e35"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:57:07.274901 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:07.274869 5052 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe851cb7-f6e1-4baf-b75e-69018da63e35-istio-token\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:57:07.274901 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:07.274894 5052 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe851cb7-f6e1-4baf-b75e-69018da63e35-istio-envoy\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:57:07.274901 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:07.274903 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8hgk\" (UniqueName: \"kubernetes.io/projected/fe851cb7-f6e1-4baf-b75e-69018da63e35-kube-api-access-t8hgk\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:57:07.275118 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:07.274911 5052 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe851cb7-f6e1-4baf-b75e-69018da63e35-istio-podinfo\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:57:07.635220 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:07.635185 5052 generic.go:356] "Generic (PLEG): container finished" podID="fe851cb7-f6e1-4baf-b75e-69018da63e35" containerID="e7214d0e42e8d6965f3d213fc9d90a7c8d038188814f10218c5a77a5a9a8b6d2" exitCode=0 Oct 30 16:57:07.635385 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:07.635233 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3ee682-openshift-default-5f7887cb79-6tsdx" Oct 30 16:57:07.635385 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:07.635264 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3ee682-openshift-default-5f7887cb79-6tsdx" event={"ID":"fe851cb7-f6e1-4baf-b75e-69018da63e35","Type":"ContainerDied","Data":"e7214d0e42e8d6965f3d213fc9d90a7c8d038188814f10218c5a77a5a9a8b6d2"} Oct 30 16:57:07.635385 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:07.635321 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3ee682-openshift-default-5f7887cb79-6tsdx" event={"ID":"fe851cb7-f6e1-4baf-b75e-69018da63e35","Type":"ContainerDied","Data":"cb4e5c5f06e0cd8d4a7c1809a3fd94a4a6f954b11430107aedbdaa11d7963ddf"} Oct 30 16:57:07.635385 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:07.635345 5052 scope.go:122] "RemoveContainer" containerID="e7214d0e42e8d6965f3d213fc9d90a7c8d038188814f10218c5a77a5a9a8b6d2" Oct 30 16:57:07.648030 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:07.648010 5052 scope.go:122] "RemoveContainer" containerID="2e130b1db7c8d1085dd61c563edc847dfa62b1e77499d4cdd0050cc8604b1682" Oct 30 16:57:07.658350 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:07.658320 5052 scope.go:122] "RemoveContainer" containerID="e7214d0e42e8d6965f3d213fc9d90a7c8d038188814f10218c5a77a5a9a8b6d2" Oct 30 16:57:07.658777 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:57:07.658753 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7214d0e42e8d6965f3d213fc9d90a7c8d038188814f10218c5a77a5a9a8b6d2\": container with ID starting with e7214d0e42e8d6965f3d213fc9d90a7c8d038188814f10218c5a77a5a9a8b6d2 not found: ID does not exist" containerID="e7214d0e42e8d6965f3d213fc9d90a7c8d038188814f10218c5a77a5a9a8b6d2" Oct 30 16:57:07.658863 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:07.658784 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7214d0e42e8d6965f3d213fc9d90a7c8d038188814f10218c5a77a5a9a8b6d2"} err="failed to get container status \"e7214d0e42e8d6965f3d213fc9d90a7c8d038188814f10218c5a77a5a9a8b6d2\": rpc error: code = NotFound desc = could not find container \"e7214d0e42e8d6965f3d213fc9d90a7c8d038188814f10218c5a77a5a9a8b6d2\": container with ID starting with e7214d0e42e8d6965f3d213fc9d90a7c8d038188814f10218c5a77a5a9a8b6d2 not found: ID does not exist" Oct 30 16:57:07.658863 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:07.658807 5052 scope.go:122] "RemoveContainer" containerID="2e130b1db7c8d1085dd61c563edc847dfa62b1e77499d4cdd0050cc8604b1682" Oct 30 16:57:07.659089 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:57:07.659070 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e130b1db7c8d1085dd61c563edc847dfa62b1e77499d4cdd0050cc8604b1682\": container with ID starting with 2e130b1db7c8d1085dd61c563edc847dfa62b1e77499d4cdd0050cc8604b1682 not found: ID does not exist" containerID="2e130b1db7c8d1085dd61c563edc847dfa62b1e77499d4cdd0050cc8604b1682" Oct 30 16:57:07.659137 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:07.659098 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e130b1db7c8d1085dd61c563edc847dfa62b1e77499d4cdd0050cc8604b1682"} err="failed to get container status \"2e130b1db7c8d1085dd61c563edc847dfa62b1e77499d4cdd0050cc8604b1682\": rpc error: code = NotFound desc = could not find container \"2e130b1db7c8d1085dd61c563edc847dfa62b1e77499d4cdd0050cc8604b1682\": container with ID starting with 2e130b1db7c8d1085dd61c563edc847dfa62b1e77499d4cdd0050cc8604b1682 not found: ID does not exist" Oct 30 16:57:07.693468 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:07.693438 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-3ee682-openshift-default-5f7887cb79-6tsdx"] Oct 30 16:57:07.756148 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:07.755781 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-3ee682-openshift-default-5f7887cb79-6tsdx"] Oct 30 16:57:08.601270 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:08.601234 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe851cb7-f6e1-4baf-b75e-69018da63e35" path="/var/lib/kubelet/pods/fe851cb7-f6e1-4baf-b75e-69018da63e35/volumes" Oct 30 16:57:09.052447 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:09.052404 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-480a60-7c447cb756-6ct9j" podUID="3b3da2b1-c132-453a-b8e9-512a43e613ff" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 16:57:10.847750 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:10.847717 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-f92caf/test-model-f92caf-kserve-6c694d4757-z2nrr"] Oct 30 16:57:10.848992 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:10.848964 5052 cpu_manager.go:414] "containerMap: removing container" podUID="52ba3acd-4de6-4ae2-8a20-49bf7f966ab0" containerName="istio-proxy" Oct 30 16:57:10.848992 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:10.848991 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52ba3acd-4de6-4ae2-8a20-49bf7f966ab0" containerName="istio-proxy" Oct 30 16:57:10.849193 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:10.849063 5052 cpu_manager.go:414] "containerMap: removing container" podUID="fe851cb7-f6e1-4baf-b75e-69018da63e35" containerName="istio-proxy" Oct 30 16:57:10.849193 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:10.849072 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe851cb7-f6e1-4baf-b75e-69018da63e35" containerName="istio-proxy" Oct 30 16:57:10.849193 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:10.849108 5052 cpu_manager.go:414] "containerMap: removing container" podUID="fe851cb7-f6e1-4baf-b75e-69018da63e35" containerName="istio-proxy" Oct 30 16:57:10.849193 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:10.849115 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe851cb7-f6e1-4baf-b75e-69018da63e35" containerName="istio-proxy" Oct 30 16:57:10.849193 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:10.849127 5052 cpu_manager.go:414] "containerMap: removing container" podUID="5cbbc24b-bf64-455e-9dc2-c34487840dfb" containerName="payload-pre-processing" Oct 30 16:57:10.849193 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:10.849134 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cbbc24b-bf64-455e-9dc2-c34487840dfb" containerName="payload-pre-processing" Oct 30 16:57:10.849517 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:10.849304 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe851cb7-f6e1-4baf-b75e-69018da63e35" containerName="istio-proxy" Oct 30 16:57:10.849517 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:10.849317 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cbbc24b-bf64-455e-9dc2-c34487840dfb" containerName="payload-pre-processing" Oct 30 16:57:10.849517 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:10.849331 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="52ba3acd-4de6-4ae2-8a20-49bf7f966ab0" containerName="istio-proxy" Oct 30 16:57:10.854935 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:10.854913 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f92caf/test-model-f92caf-kserve-6c694d4757-z2nrr" Oct 30 16:57:10.878037 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:10.876735 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-f92caf/test-model-f92caf-kserve-6c694d4757-z2nrr"] Oct 30 16:57:10.879679 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:10.879647 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-f92caf\"/\"kube-root-ca.crt\"" Oct 30 16:57:10.880414 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:10.880396 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-f92caf\"/\"openshift-service-ca.crt\"" Oct 30 16:57:10.886046 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:10.886027 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-f92caf\"/\"default-dockercfg-7fjv5\"" Oct 30 16:57:10.886147 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:10.886062 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-f92caf\"/\"test-model-f92caf-kserve-self-signed-certs\"" Oct 30 16:57:11.029971 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:11.029927 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9551ea7f-15f8-4907-aad5-f9160f7b604e-dshm\") pod \"test-model-f92caf-kserve-6c694d4757-z2nrr\" (UID: \"9551ea7f-15f8-4907-aad5-f9160f7b604e\") " pod="ai-tenant-e2e-inf-b-f92caf/test-model-f92caf-kserve-6c694d4757-z2nrr" Oct 30 16:57:11.030129 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:11.030085 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9551ea7f-15f8-4907-aad5-f9160f7b604e-tls-certs\") pod \"test-model-f92caf-kserve-6c694d4757-z2nrr\" (UID: \"9551ea7f-15f8-4907-aad5-f9160f7b604e\") " pod="ai-tenant-e2e-inf-b-f92caf/test-model-f92caf-kserve-6c694d4757-z2nrr" Oct 30 16:57:11.030129 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:11.030115 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9551ea7f-15f8-4907-aad5-f9160f7b604e-model-cache\") pod \"test-model-f92caf-kserve-6c694d4757-z2nrr\" (UID: \"9551ea7f-15f8-4907-aad5-f9160f7b604e\") " pod="ai-tenant-e2e-inf-b-f92caf/test-model-f92caf-kserve-6c694d4757-z2nrr" Oct 30 16:57:11.030240 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:11.030143 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9551ea7f-15f8-4907-aad5-f9160f7b604e-tmp-dir\") pod \"test-model-f92caf-kserve-6c694d4757-z2nrr\" (UID: \"9551ea7f-15f8-4907-aad5-f9160f7b604e\") " pod="ai-tenant-e2e-inf-b-f92caf/test-model-f92caf-kserve-6c694d4757-z2nrr" Oct 30 16:57:11.030280 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:11.030257 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9551ea7f-15f8-4907-aad5-f9160f7b604e-home\") pod \"test-model-f92caf-kserve-6c694d4757-z2nrr\" (UID: \"9551ea7f-15f8-4907-aad5-f9160f7b604e\") " pod="ai-tenant-e2e-inf-b-f92caf/test-model-f92caf-kserve-6c694d4757-z2nrr" Oct 30 16:57:11.030336 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:11.030322 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhws4\" (UniqueName: \"kubernetes.io/projected/9551ea7f-15f8-4907-aad5-f9160f7b604e-kube-api-access-xhws4\") pod \"test-model-f92caf-kserve-6c694d4757-z2nrr\" (UID: \"9551ea7f-15f8-4907-aad5-f9160f7b604e\") " pod="ai-tenant-e2e-inf-b-f92caf/test-model-f92caf-kserve-6c694d4757-z2nrr" Oct 30 16:57:11.166608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:11.164118 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9551ea7f-15f8-4907-aad5-f9160f7b604e-dshm\") pod \"test-model-f92caf-kserve-6c694d4757-z2nrr\" (UID: \"9551ea7f-15f8-4907-aad5-f9160f7b604e\") " pod="ai-tenant-e2e-inf-b-f92caf/test-model-f92caf-kserve-6c694d4757-z2nrr" Oct 30 16:57:11.166608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:11.164312 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9551ea7f-15f8-4907-aad5-f9160f7b604e-tls-certs\") pod \"test-model-f92caf-kserve-6c694d4757-z2nrr\" (UID: \"9551ea7f-15f8-4907-aad5-f9160f7b604e\") " pod="ai-tenant-e2e-inf-b-f92caf/test-model-f92caf-kserve-6c694d4757-z2nrr" Oct 30 16:57:11.166608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:11.164352 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9551ea7f-15f8-4907-aad5-f9160f7b604e-model-cache\") pod \"test-model-f92caf-kserve-6c694d4757-z2nrr\" (UID: \"9551ea7f-15f8-4907-aad5-f9160f7b604e\") " pod="ai-tenant-e2e-inf-b-f92caf/test-model-f92caf-kserve-6c694d4757-z2nrr" Oct 30 16:57:11.166608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:11.164400 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9551ea7f-15f8-4907-aad5-f9160f7b604e-tmp-dir\") pod \"test-model-f92caf-kserve-6c694d4757-z2nrr\" (UID: \"9551ea7f-15f8-4907-aad5-f9160f7b604e\") " pod="ai-tenant-e2e-inf-b-f92caf/test-model-f92caf-kserve-6c694d4757-z2nrr" Oct 30 16:57:11.166608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:11.164429 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9551ea7f-15f8-4907-aad5-f9160f7b604e-home\") pod \"test-model-f92caf-kserve-6c694d4757-z2nrr\" (UID: \"9551ea7f-15f8-4907-aad5-f9160f7b604e\") " pod="ai-tenant-e2e-inf-b-f92caf/test-model-f92caf-kserve-6c694d4757-z2nrr" Oct 30 16:57:11.166608 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:11.164510 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhws4\" (UniqueName: \"kubernetes.io/projected/9551ea7f-15f8-4907-aad5-f9160f7b604e-kube-api-access-xhws4\") pod \"test-model-f92caf-kserve-6c694d4757-z2nrr\" (UID: \"9551ea7f-15f8-4907-aad5-f9160f7b604e\") " pod="ai-tenant-e2e-inf-b-f92caf/test-model-f92caf-kserve-6c694d4757-z2nrr" Oct 30 16:57:11.178211 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:11.176094 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9551ea7f-15f8-4907-aad5-f9160f7b604e-model-cache\") pod \"test-model-f92caf-kserve-6c694d4757-z2nrr\" (UID: \"9551ea7f-15f8-4907-aad5-f9160f7b604e\") " pod="ai-tenant-e2e-inf-b-f92caf/test-model-f92caf-kserve-6c694d4757-z2nrr" Oct 30 16:57:11.201640 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:11.184937 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9551ea7f-15f8-4907-aad5-f9160f7b604e-tmp-dir\") pod \"test-model-f92caf-kserve-6c694d4757-z2nrr\" (UID: \"9551ea7f-15f8-4907-aad5-f9160f7b604e\") " pod="ai-tenant-e2e-inf-b-f92caf/test-model-f92caf-kserve-6c694d4757-z2nrr" Oct 30 16:57:11.201640 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:11.185203 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9551ea7f-15f8-4907-aad5-f9160f7b604e-home\") pod \"test-model-f92caf-kserve-6c694d4757-z2nrr\" (UID: \"9551ea7f-15f8-4907-aad5-f9160f7b604e\") " pod="ai-tenant-e2e-inf-b-f92caf/test-model-f92caf-kserve-6c694d4757-z2nrr" Oct 30 16:57:11.215611 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:11.202486 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhws4\" (UniqueName: \"kubernetes.io/projected/9551ea7f-15f8-4907-aad5-f9160f7b604e-kube-api-access-xhws4\") pod \"test-model-f92caf-kserve-6c694d4757-z2nrr\" (UID: \"9551ea7f-15f8-4907-aad5-f9160f7b604e\") " pod="ai-tenant-e2e-inf-b-f92caf/test-model-f92caf-kserve-6c694d4757-z2nrr" Oct 30 16:57:11.219602 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:11.218779 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9551ea7f-15f8-4907-aad5-f9160f7b604e-dshm\") pod \"test-model-f92caf-kserve-6c694d4757-z2nrr\" (UID: \"9551ea7f-15f8-4907-aad5-f9160f7b604e\") " pod="ai-tenant-e2e-inf-b-f92caf/test-model-f92caf-kserve-6c694d4757-z2nrr" Oct 30 16:57:11.230607 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:11.220942 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9551ea7f-15f8-4907-aad5-f9160f7b604e-tls-certs\") pod \"test-model-f92caf-kserve-6c694d4757-z2nrr\" (UID: \"9551ea7f-15f8-4907-aad5-f9160f7b604e\") " pod="ai-tenant-e2e-inf-b-f92caf/test-model-f92caf-kserve-6c694d4757-z2nrr" Oct 30 16:57:11.479233 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:11.479149 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f92caf/test-model-f92caf-kserve-6c694d4757-z2nrr" Oct 30 16:57:11.796773 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:11.796651 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-f92caf/test-model-f92caf-kserve-6c694d4757-z2nrr"] Oct 30 16:57:11.801963 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:57:11.801923 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9551ea7f_15f8_4907_aad5_f9160f7b604e.slice/crio-1e7114eed1c959cbac01f10408fb917d7935d75d6992b4a7649fe58de9d42d9e WatchSource:0}: Error finding container 1e7114eed1c959cbac01f10408fb917d7935d75d6992b4a7649fe58de9d42d9e: Status 404 returned error can't find the container with id 1e7114eed1c959cbac01f10408fb917d7935d75d6992b4a7649fe58de9d42d9e Oct 30 16:57:12.675681 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:12.675642 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f92caf/test-model-f92caf-kserve-6c694d4757-z2nrr" event={"ID":"9551ea7f-15f8-4907-aad5-f9160f7b604e","Type":"ContainerStarted","Data":"445476025c22533ea9898f34c5ed7d4ce87b3a94fe65563f418173b1076a6ceb"} Oct 30 16:57:12.675681 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:12.675688 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f92caf/test-model-f92caf-kserve-6c694d4757-z2nrr" event={"ID":"9551ea7f-15f8-4907-aad5-f9160f7b604e","Type":"ContainerStarted","Data":"1e7114eed1c959cbac01f10408fb917d7935d75d6992b4a7649fe58de9d42d9e"} Oct 30 16:57:12.718477 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:12.718423 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-f92caf/test-model-f92caf-kserve-6c694d4757-z2nrr" podStartSLOduration=2.718410491 podStartE2EDuration="2.718410491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:57:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:57:12.718225951 +0000 UTC m=+2188.245561171" watchObservedRunningTime="2026-10-30 16:57:12.718410491 +0000 UTC m=+2188.245745709" Oct 30 16:57:13.022526 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:13.022454 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-x5p9m"] Oct 30 16:57:13.023494 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:13.023470 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe851cb7-f6e1-4baf-b75e-69018da63e35" containerName="istio-proxy" Oct 30 16:57:13.028019 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:13.028002 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-x5p9m" Oct 30 16:57:13.068990 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:13.068957 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-bfecdd-openshift-default-dockercfg-2cb9m\"" Oct 30 16:57:13.075074 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:13.075046 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-x5p9m"] Oct 30 16:57:13.098503 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:13.098440 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6c3dee8-d01e-445d-b8a4-96b7f57493ec-credential-socket\") pod \"e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-x5p9m\" (UID: \"d6c3dee8-d01e-445d-b8a4-96b7f57493ec\") " pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-x5p9m" Oct 30 16:57:13.098666 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:13.098627 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6c3dee8-d01e-445d-b8a4-96b7f57493ec-workload-certs\") pod \"e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-x5p9m\" (UID: \"d6c3dee8-d01e-445d-b8a4-96b7f57493ec\") " pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-x5p9m" Oct 30 16:57:13.098751 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:13.098674 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6c3dee8-d01e-445d-b8a4-96b7f57493ec-istiod-ca-cert\") pod \"e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-x5p9m\" (UID: \"d6c3dee8-d01e-445d-b8a4-96b7f57493ec\") " pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-x5p9m" Oct 30 16:57:13.098812 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:13.098766 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6c3dee8-d01e-445d-b8a4-96b7f57493ec-istio-envoy\") pod \"e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-x5p9m\" (UID: \"d6c3dee8-d01e-445d-b8a4-96b7f57493ec\") " pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-x5p9m" Oct 30 16:57:13.098945 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:13.098923 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6c3dee8-d01e-445d-b8a4-96b7f57493ec-workload-socket\") pod \"e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-x5p9m\" (UID: \"d6c3dee8-d01e-445d-b8a4-96b7f57493ec\") " pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-x5p9m" Oct 30 16:57:13.099039 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:13.098966 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9gxx\" (UniqueName: \"kubernetes.io/projected/d6c3dee8-d01e-445d-b8a4-96b7f57493ec-kube-api-access-h9gxx\") pod \"e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-x5p9m\" (UID: \"d6c3dee8-d01e-445d-b8a4-96b7f57493ec\") " pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-x5p9m" Oct 30 16:57:13.099114 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:13.099073 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6c3dee8-d01e-445d-b8a4-96b7f57493ec-istio-data\") pod \"e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-x5p9m\" (UID: \"d6c3dee8-d01e-445d-b8a4-96b7f57493ec\") " pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-x5p9m" Oct 30 16:57:13.099238 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:13.099190 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6c3dee8-d01e-445d-b8a4-96b7f57493ec-istio-podinfo\") pod \"e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-x5p9m\" (UID: \"d6c3dee8-d01e-445d-b8a4-96b7f57493ec\") " pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-x5p9m" Oct 30 16:57:13.099712 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:13.099469 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6c3dee8-d01e-445d-b8a4-96b7f57493ec-istio-token\") pod \"e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-x5p9m\" (UID: \"d6c3dee8-d01e-445d-b8a4-96b7f57493ec\") " pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-x5p9m" Oct 30 16:57:13.210602 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:13.204986 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6c3dee8-d01e-445d-b8a4-96b7f57493ec-istio-podinfo\") pod \"e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-x5p9m\" (UID: \"d6c3dee8-d01e-445d-b8a4-96b7f57493ec\") " pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-x5p9m" Oct 30 16:57:13.210602 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:13.205121 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6c3dee8-d01e-445d-b8a4-96b7f57493ec-istio-token\") pod \"e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-x5p9m\" (UID: \"d6c3dee8-d01e-445d-b8a4-96b7f57493ec\") " pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-x5p9m" Oct 30 16:57:13.210602 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:13.205162 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6c3dee8-d01e-445d-b8a4-96b7f57493ec-credential-socket\") pod \"e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-x5p9m\" (UID: \"d6c3dee8-d01e-445d-b8a4-96b7f57493ec\") " pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-x5p9m" Oct 30 16:57:13.210602 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:13.205220 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6c3dee8-d01e-445d-b8a4-96b7f57493ec-workload-certs\") pod \"e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-x5p9m\" (UID: \"d6c3dee8-d01e-445d-b8a4-96b7f57493ec\") " pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-x5p9m" Oct 30 16:57:13.210602 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:13.205265 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6c3dee8-d01e-445d-b8a4-96b7f57493ec-istiod-ca-cert\") pod \"e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-x5p9m\" (UID: \"d6c3dee8-d01e-445d-b8a4-96b7f57493ec\") " pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-x5p9m" Oct 30 16:57:13.210602 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:13.205323 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6c3dee8-d01e-445d-b8a4-96b7f57493ec-istio-envoy\") pod \"e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-x5p9m\" (UID: \"d6c3dee8-d01e-445d-b8a4-96b7f57493ec\") " pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-x5p9m" Oct 30 16:57:13.210602 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:13.205546 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6c3dee8-d01e-445d-b8a4-96b7f57493ec-workload-socket\") pod \"e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-x5p9m\" (UID: \"d6c3dee8-d01e-445d-b8a4-96b7f57493ec\") " pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-x5p9m" Oct 30 16:57:13.210602 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:13.205610 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9gxx\" (UniqueName: \"kubernetes.io/projected/d6c3dee8-d01e-445d-b8a4-96b7f57493ec-kube-api-access-h9gxx\") pod \"e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-x5p9m\" (UID: \"d6c3dee8-d01e-445d-b8a4-96b7f57493ec\") " pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-x5p9m" Oct 30 16:57:13.210602 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:13.205673 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6c3dee8-d01e-445d-b8a4-96b7f57493ec-istio-data\") pod \"e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-x5p9m\" (UID: \"d6c3dee8-d01e-445d-b8a4-96b7f57493ec\") " pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-x5p9m" Oct 30 16:57:13.210602 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:13.206196 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6c3dee8-d01e-445d-b8a4-96b7f57493ec-istio-data\") pod \"e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-x5p9m\" (UID: \"d6c3dee8-d01e-445d-b8a4-96b7f57493ec\") " pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-x5p9m" Oct 30 16:57:13.228601 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:13.215444 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6c3dee8-d01e-445d-b8a4-96b7f57493ec-istio-podinfo\") pod \"e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-x5p9m\" (UID: \"d6c3dee8-d01e-445d-b8a4-96b7f57493ec\") " pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-x5p9m" Oct 30 16:57:13.228601 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:13.219183 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6c3dee8-d01e-445d-b8a4-96b7f57493ec-credential-socket\") pod \"e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-x5p9m\" (UID: \"d6c3dee8-d01e-445d-b8a4-96b7f57493ec\") " pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-x5p9m" Oct 30 16:57:13.228601 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:13.219432 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6c3dee8-d01e-445d-b8a4-96b7f57493ec-workload-certs\") pod \"e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-x5p9m\" (UID: \"d6c3dee8-d01e-445d-b8a4-96b7f57493ec\") " pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-x5p9m" Oct 30 16:57:13.229612 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:13.229167 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6c3dee8-d01e-445d-b8a4-96b7f57493ec-istiod-ca-cert\") pod \"e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-x5p9m\" (UID: \"d6c3dee8-d01e-445d-b8a4-96b7f57493ec\") " pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-x5p9m" Oct 30 16:57:13.229612 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:13.229398 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6c3dee8-d01e-445d-b8a4-96b7f57493ec-workload-socket\") pod \"e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-x5p9m\" (UID: \"d6c3dee8-d01e-445d-b8a4-96b7f57493ec\") " pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-x5p9m" Oct 30 16:57:13.258624 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:13.255329 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6c3dee8-d01e-445d-b8a4-96b7f57493ec-istio-envoy\") pod \"e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-x5p9m\" (UID: \"d6c3dee8-d01e-445d-b8a4-96b7f57493ec\") " pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-x5p9m" Oct 30 16:57:13.293652 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:13.277196 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6c3dee8-d01e-445d-b8a4-96b7f57493ec-istio-token\") pod \"e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-x5p9m\" (UID: \"d6c3dee8-d01e-445d-b8a4-96b7f57493ec\") " pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-x5p9m" Oct 30 16:57:13.316223 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:13.295382 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9gxx\" (UniqueName: \"kubernetes.io/projected/d6c3dee8-d01e-445d-b8a4-96b7f57493ec-kube-api-access-h9gxx\") pod \"e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-x5p9m\" (UID: \"d6c3dee8-d01e-445d-b8a4-96b7f57493ec\") " pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-x5p9m" Oct 30 16:57:13.385607 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:13.384919 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-x5p9m" Oct 30 16:57:13.480351 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:13.480313 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-f92caf/test-model-f92caf-kserve-6c694d4757-z2nrr" Oct 30 16:57:13.780243 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:13.780215 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-x5p9m"] Oct 30 16:57:13.781092 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:57:13.781067 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6c3dee8_d01e_445d_b8a4_96b7f57493ec.slice/crio-cb5d5641f049456fb180ce2ebcd6cc9c9e03816356f63c9a0b3b2ae6dd4501a9 WatchSource:0}: Error finding container cb5d5641f049456fb180ce2ebcd6cc9c9e03816356f63c9a0b3b2ae6dd4501a9: Status 404 returned error can't find the container with id cb5d5641f049456fb180ce2ebcd6cc9c9e03816356f63c9a0b3b2ae6dd4501a9 Oct 30 16:57:13.784563 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:13.784526 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:57:13.784701 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:13.784650 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:57:13.784769 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:13.784701 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:57:14.745099 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:14.741360 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-x5p9m" event={"ID":"d6c3dee8-d01e-445d-b8a4-96b7f57493ec","Type":"ContainerStarted","Data":"6e801b7a1159f0947dc899e0f8a3597f7d099583f63a7e759acb93e810488d39"} Oct 30 16:57:14.745099 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:14.741423 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-x5p9m" event={"ID":"d6c3dee8-d01e-445d-b8a4-96b7f57493ec","Type":"ContainerStarted","Data":"cb5d5641f049456fb180ce2ebcd6cc9c9e03816356f63c9a0b3b2ae6dd4501a9"} Oct 30 16:57:14.805471 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:14.805421 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-x5p9m" podStartSLOduration=2.805407976 podStartE2EDuration="2.805407976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:57:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:57:14.801602131 +0000 UTC m=+2190.328937381" watchObservedRunningTime="2026-10-30 16:57:14.805407976 +0000 UTC m=+2190.332743194" Oct 30 16:57:15.385588 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:15.385534 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-x5p9m" Oct 30 16:57:15.391662 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:15.391636 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-x5p9m" Oct 30 16:57:15.769645 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:15.769249 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-x5p9m" Oct 30 16:57:15.771143 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:15.771124 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-x5p9m" Oct 30 16:57:18.847225 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:57:18.847185 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 16:57:19.052997 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:19.052960 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-480a60-7c447cb756-6ct9j" podUID="3b3da2b1-c132-453a-b8e9-512a43e613ff" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 16:57:20.749603 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:20.689409 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-ea77d8/test-model-ea77d8-kserve-7f798c774c-tbxxl" podUID="67ba1b17-147c-4b27-b2bd-9660840a9778" containerName="main" probeResult="failure" output="Get \"https://10.132.0.171:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 16:57:20.895604 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:20.866674 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx" podUID="bd662f3a-659f-477c-b8d5-94c64288c0c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:57:20.895604 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:20.868973 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/distinct-w5-967fac-kserve-69cbcccd8b-b85bn" podUID="8f6b9354-378f-4451-9ff7-be7f7fec12f2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:57:20.922364 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:20.911950 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-750dcb/distinct-w2-750dcb-kserve-86bbdfcbb-bpn77" podUID="c45190af-d795-4e13-8f2f-670b2c597658" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:57:20.922364 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:20.917295 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d693d0/distinct-2-w6-d693d0-kserve-6498fdc474-dpbjj" podUID="4b98b8ff-72cd-4ddb-a4b8-67078162e611" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:57:20.922364 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:20.921703 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-17f60f/test-model-17f60f-kserve-6d99c5d4f5-n2vss" podUID="06b94bcd-8c5a-4143-a936-34d62f4189b0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.169:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:57:20.922364 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:20.922054 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-9c1e26/test-model-9c1e26-kserve-7cc944cb5f-xjljc" podUID="17327d7e-8d87-4461-be7f-1ed4c1d17701" containerName="main" probeResult="failure" output="Get \"https://10.132.0.168:8000/ready\": context deadline exceeded" Oct 30 16:57:21.074752 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:21.074655 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-689899/test-model-689899-kserve-5f8945cf6f-925q2" podUID="cbc3abfb-2bcf-4133-9a49-65b29f06f12a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.170:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:57:21.489630 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:21.484715 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-f92caf/test-model-f92caf-kserve-6c694d4757-z2nrr" Oct 30 16:57:21.522022 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:21.521623 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-f92caf/test-model-f92caf-kserve-6c694d4757-z2nrr" Oct 30 16:57:22.006130 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:22.006092 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-7xqp2"] Oct 30 16:57:22.006677 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:22.006393 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-7xqp2" podUID="f43327c0-fce9-4bdf-ae16-2111377c71ad" containerName="istio-proxy" containerID="cri-o://fb4ec2bde8c8f4328e3878bdcf85cb0e0bb3f2d0735307e9e6413b622b30c2ea" gracePeriod=30 Oct 30 16:57:22.163045 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:22.156605 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-8cxdf"] Oct 30 16:57:22.163045 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:22.156847 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-8cxdf" podUID="dd213278-0e61-4779-a644-abfed99fbb9a" containerName="istio-proxy" containerID="cri-o://e9052ffa09f683f1ff1770365e18bfcd37e8a9ba23795838e0125f2898f05cd4" gracePeriod=30 Oct 30 16:57:22.167125 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:22.167095 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-f92caf/test-model-f92caf-kserve-6c694d4757-z2nrr" Oct 30 16:57:22.167334 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:22.167308 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cp9m8"] Oct 30 16:57:22.167563 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:22.167520 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cp9m8" podUID="ee44984b-c448-4580-aa5a-637c73d09e19" containerName="istio-proxy" containerID="cri-o://d46b8129f7de87e1ddf19c8dd248d0ec3e2f04a3ccf0bf6d3f1a6876fbb75a06" gracePeriod=30 Oct 30 16:57:23.008138 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:23.008110 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-480a60-7c447cb756-6ct9j" Oct 30 16:57:23.133138 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:23.133094 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b3da2b1-c132-453a-b8e9-512a43e613ff-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b3da2b1-c132-453a-b8e9-512a43e613ff-plugins-config-volume\") pod \"3b3da2b1-c132-453a-b8e9-512a43e613ff\" (UID: \"3b3da2b1-c132-453a-b8e9-512a43e613ff\") " Oct 30 16:57:23.133402 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:23.133383 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b3da2b1-c132-453a-b8e9-512a43e613ff-kube-api-access-445bh\" (UniqueName: \"kubernetes.io/projected/3b3da2b1-c132-453a-b8e9-512a43e613ff-kube-api-access-445bh\") pod \"3b3da2b1-c132-453a-b8e9-512a43e613ff\" (UID: \"3b3da2b1-c132-453a-b8e9-512a43e613ff\") " Oct 30 16:57:23.133570 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:23.133540 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b3da2b1-c132-453a-b8e9-512a43e613ff-plugins-config-volume" pod "3b3da2b1-c132-453a-b8e9-512a43e613ff" (UID: "3b3da2b1-c132-453a-b8e9-512a43e613ff"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:57:23.135823 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:23.135797 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b3da2b1-c132-453a-b8e9-512a43e613ff-kube-api-access-445bh" pod "3b3da2b1-c132-453a-b8e9-512a43e613ff" (UID: "3b3da2b1-c132-453a-b8e9-512a43e613ff"). InnerVolumeSpecName "kube-api-access-445bh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:57:23.138714 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:23.138692 5052 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b3da2b1-c132-453a-b8e9-512a43e613ff-plugins-config-volume\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:57:23.138786 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:23.138720 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-445bh\" (UniqueName: \"kubernetes.io/projected/3b3da2b1-c132-453a-b8e9-512a43e613ff-kube-api-access-445bh\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:57:23.161224 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:23.161193 5052 generic.go:356] "Generic (PLEG): container finished" podID="3b3da2b1-c132-453a-b8e9-512a43e613ff" containerID="945f122630787d3d009e107bdd51c4b40193e6395f1aacc12a034321f9458c32" exitCode=0 Oct 30 16:57:23.161350 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:23.161245 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-480a60-7c447cb756-6ct9j" Oct 30 16:57:23.161350 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:23.161279 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-480a60-7c447cb756-6ct9j" event={"ID":"3b3da2b1-c132-453a-b8e9-512a43e613ff","Type":"ContainerDied","Data":"945f122630787d3d009e107bdd51c4b40193e6395f1aacc12a034321f9458c32"} Oct 30 16:57:23.161350 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:23.161336 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-480a60-7c447cb756-6ct9j" event={"ID":"3b3da2b1-c132-453a-b8e9-512a43e613ff","Type":"ContainerDied","Data":"d8aaf5c925e4a9ba6e7f4d1baa25dda32fcf84acb9b967e01bc64a4c50a99117"} Oct 30 16:57:23.161508 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:23.161363 5052 scope.go:122] "RemoveContainer" containerID="945f122630787d3d009e107bdd51c4b40193e6395f1aacc12a034321f9458c32" Oct 30 16:57:23.174673 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:23.174655 5052 scope.go:122] "RemoveContainer" containerID="945f122630787d3d009e107bdd51c4b40193e6395f1aacc12a034321f9458c32" Oct 30 16:57:23.174968 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:57:23.174948 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"945f122630787d3d009e107bdd51c4b40193e6395f1aacc12a034321f9458c32\": container with ID starting with 945f122630787d3d009e107bdd51c4b40193e6395f1aacc12a034321f9458c32 not found: ID does not exist" containerID="945f122630787d3d009e107bdd51c4b40193e6395f1aacc12a034321f9458c32" Oct 30 16:57:23.175056 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:23.174973 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"945f122630787d3d009e107bdd51c4b40193e6395f1aacc12a034321f9458c32"} err="failed to get container status \"945f122630787d3d009e107bdd51c4b40193e6395f1aacc12a034321f9458c32\": rpc error: code = NotFound desc = could not find container \"945f122630787d3d009e107bdd51c4b40193e6395f1aacc12a034321f9458c32\": container with ID starting with 945f122630787d3d009e107bdd51c4b40193e6395f1aacc12a034321f9458c32 not found: ID does not exist" Oct 30 16:57:23.220292 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:23.220240 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-480a60-7c447cb756-6ct9j"] Oct 30 16:57:23.225632 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:23.225610 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-480a60-7c447cb756-6ct9j"] Oct 30 16:57:24.617679 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:24.617641 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b3da2b1-c132-453a-b8e9-512a43e613ff" path="/var/lib/kubelet/pods/3b3da2b1-c132-453a-b8e9-512a43e613ff/volumes" Oct 30 16:57:27.210675 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:27.210643 5052 generic.go:356] "Generic (PLEG): container finished" podID="f43327c0-fce9-4bdf-ae16-2111377c71ad" containerID="fb4ec2bde8c8f4328e3878bdcf85cb0e0bb3f2d0735307e9e6413b622b30c2ea" exitCode=0 Oct 30 16:57:27.211078 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:27.210897 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-7xqp2" event={"ID":"f43327c0-fce9-4bdf-ae16-2111377c71ad","Type":"ContainerDied","Data":"fb4ec2bde8c8f4328e3878bdcf85cb0e0bb3f2d0735307e9e6413b622b30c2ea"} Oct 30 16:57:27.393025 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:27.392681 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-7xqp2" Oct 30 16:57:27.481531 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:27.481501 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-8cxdf" Oct 30 16:57:27.504279 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:27.504192 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cp9m8" Oct 30 16:57:27.533277 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:27.533179 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f43327c0-fce9-4bdf-ae16-2111377c71ad-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f43327c0-fce9-4bdf-ae16-2111377c71ad-workload-socket\") pod \"f43327c0-fce9-4bdf-ae16-2111377c71ad\" (UID: \"f43327c0-fce9-4bdf-ae16-2111377c71ad\") " Oct 30 16:57:27.533277 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:27.533237 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f43327c0-fce9-4bdf-ae16-2111377c71ad-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f43327c0-fce9-4bdf-ae16-2111377c71ad-credential-socket\") pod \"f43327c0-fce9-4bdf-ae16-2111377c71ad\" (UID: \"f43327c0-fce9-4bdf-ae16-2111377c71ad\") " Oct 30 16:57:27.533514 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:27.533279 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f43327c0-fce9-4bdf-ae16-2111377c71ad-istio-token\" (UniqueName: \"kubernetes.io/projected/f43327c0-fce9-4bdf-ae16-2111377c71ad-istio-token\") pod \"f43327c0-fce9-4bdf-ae16-2111377c71ad\" (UID: \"f43327c0-fce9-4bdf-ae16-2111377c71ad\") " Oct 30 16:57:27.533514 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:27.533435 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f43327c0-fce9-4bdf-ae16-2111377c71ad-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f43327c0-fce9-4bdf-ae16-2111377c71ad-istiod-ca-cert\") pod \"f43327c0-fce9-4bdf-ae16-2111377c71ad\" (UID: \"f43327c0-fce9-4bdf-ae16-2111377c71ad\") " Oct 30 16:57:27.533514 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:27.533471 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f43327c0-fce9-4bdf-ae16-2111377c71ad-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f43327c0-fce9-4bdf-ae16-2111377c71ad-istio-podinfo\") pod \"f43327c0-fce9-4bdf-ae16-2111377c71ad\" (UID: \"f43327c0-fce9-4bdf-ae16-2111377c71ad\") " Oct 30 16:57:27.533697 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:27.533626 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f43327c0-fce9-4bdf-ae16-2111377c71ad-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f43327c0-fce9-4bdf-ae16-2111377c71ad-istio-data\") pod \"f43327c0-fce9-4bdf-ae16-2111377c71ad\" (UID: \"f43327c0-fce9-4bdf-ae16-2111377c71ad\") " Oct 30 16:57:27.534026 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:27.533900 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f43327c0-fce9-4bdf-ae16-2111377c71ad-credential-socket" pod "f43327c0-fce9-4bdf-ae16-2111377c71ad" (UID: "f43327c0-fce9-4bdf-ae16-2111377c71ad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:57:27.534026 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:27.533951 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f43327c0-fce9-4bdf-ae16-2111377c71ad-workload-socket" pod "f43327c0-fce9-4bdf-ae16-2111377c71ad" (UID: "f43327c0-fce9-4bdf-ae16-2111377c71ad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:57:27.534026 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:27.533992 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f43327c0-fce9-4bdf-ae16-2111377c71ad-istio-data" pod "f43327c0-fce9-4bdf-ae16-2111377c71ad" (UID: "f43327c0-fce9-4bdf-ae16-2111377c71ad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:57:27.534287 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:27.534264 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f43327c0-fce9-4bdf-ae16-2111377c71ad-istiod-ca-cert" pod "f43327c0-fce9-4bdf-ae16-2111377c71ad" (UID: "f43327c0-fce9-4bdf-ae16-2111377c71ad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:57:27.534390 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:27.534374 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f43327c0-fce9-4bdf-ae16-2111377c71ad-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f43327c0-fce9-4bdf-ae16-2111377c71ad-workload-certs\") pod \"f43327c0-fce9-4bdf-ae16-2111377c71ad\" (UID: \"f43327c0-fce9-4bdf-ae16-2111377c71ad\") " Oct 30 16:57:27.534448 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:27.534417 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f43327c0-fce9-4bdf-ae16-2111377c71ad-kube-api-access-d2ggn\" (UniqueName: \"kubernetes.io/projected/f43327c0-fce9-4bdf-ae16-2111377c71ad-kube-api-access-d2ggn\") pod \"f43327c0-fce9-4bdf-ae16-2111377c71ad\" (UID: \"f43327c0-fce9-4bdf-ae16-2111377c71ad\") " Oct 30 16:57:27.534684 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:27.534490 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f43327c0-fce9-4bdf-ae16-2111377c71ad-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f43327c0-fce9-4bdf-ae16-2111377c71ad-istio-envoy\") pod \"f43327c0-fce9-4bdf-ae16-2111377c71ad\" (UID: \"f43327c0-fce9-4bdf-ae16-2111377c71ad\") " Oct 30 16:57:27.534752 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:27.534694 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f43327c0-fce9-4bdf-ae16-2111377c71ad-workload-certs" pod "f43327c0-fce9-4bdf-ae16-2111377c71ad" (UID: "f43327c0-fce9-4bdf-ae16-2111377c71ad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:57:27.536624 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:27.535557 5052 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f43327c0-fce9-4bdf-ae16-2111377c71ad-istio-data\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:57:27.536624 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:27.535634 5052 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f43327c0-fce9-4bdf-ae16-2111377c71ad-workload-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:57:27.536624 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:27.535650 5052 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f43327c0-fce9-4bdf-ae16-2111377c71ad-workload-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:57:27.536624 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:27.535664 5052 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f43327c0-fce9-4bdf-ae16-2111377c71ad-credential-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:57:27.536624 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:27.535678 5052 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f43327c0-fce9-4bdf-ae16-2111377c71ad-istiod-ca-cert\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:57:27.536624 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:27.536335 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f43327c0-fce9-4bdf-ae16-2111377c71ad-istio-token" pod "f43327c0-fce9-4bdf-ae16-2111377c71ad" (UID: "f43327c0-fce9-4bdf-ae16-2111377c71ad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:57:27.537410 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:27.537389 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f43327c0-fce9-4bdf-ae16-2111377c71ad-kube-api-access-d2ggn" pod "f43327c0-fce9-4bdf-ae16-2111377c71ad" (UID: "f43327c0-fce9-4bdf-ae16-2111377c71ad"). InnerVolumeSpecName "kube-api-access-d2ggn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:57:27.537571 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:27.537553 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f43327c0-fce9-4bdf-ae16-2111377c71ad-istio-podinfo" pod "f43327c0-fce9-4bdf-ae16-2111377c71ad" (UID: "f43327c0-fce9-4bdf-ae16-2111377c71ad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:57:27.537831 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:27.537809 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f43327c0-fce9-4bdf-ae16-2111377c71ad-istio-envoy" pod "f43327c0-fce9-4bdf-ae16-2111377c71ad" (UID: "f43327c0-fce9-4bdf-ae16-2111377c71ad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:57:27.636828 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:27.636793 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee44984b-c448-4580-aa5a-637c73d09e19-kube-api-access-69n2v\" (UniqueName: \"kubernetes.io/projected/ee44984b-c448-4580-aa5a-637c73d09e19-kube-api-access-69n2v\") pod \"ee44984b-c448-4580-aa5a-637c73d09e19\" (UID: \"ee44984b-c448-4580-aa5a-637c73d09e19\") " Oct 30 16:57:27.637009 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:27.636877 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee44984b-c448-4580-aa5a-637c73d09e19-istio-token\" (UniqueName: \"kubernetes.io/projected/ee44984b-c448-4580-aa5a-637c73d09e19-istio-token\") pod \"ee44984b-c448-4580-aa5a-637c73d09e19\" (UID: \"ee44984b-c448-4580-aa5a-637c73d09e19\") " Oct 30 16:57:27.637009 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:27.636928 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd213278-0e61-4779-a644-abfed99fbb9a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd213278-0e61-4779-a644-abfed99fbb9a-workload-socket\") pod \"dd213278-0e61-4779-a644-abfed99fbb9a\" (UID: \"dd213278-0e61-4779-a644-abfed99fbb9a\") " Oct 30 16:57:27.637009 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:27.636957 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd213278-0e61-4779-a644-abfed99fbb9a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd213278-0e61-4779-a644-abfed99fbb9a-workload-certs\") pod \"dd213278-0e61-4779-a644-abfed99fbb9a\" (UID: \"dd213278-0e61-4779-a644-abfed99fbb9a\") " Oct 30 16:57:27.637009 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:27.637000 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dd213278-0e61-4779-a644-abfed99fbb9a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd213278-0e61-4779-a644-abfed99fbb9a-istio-podinfo\") pod \"dd213278-0e61-4779-a644-abfed99fbb9a\" (UID: \"dd213278-0e61-4779-a644-abfed99fbb9a\") " Oct 30 16:57:27.637267 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:27.637039 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee44984b-c448-4580-aa5a-637c73d09e19-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee44984b-c448-4580-aa5a-637c73d09e19-workload-certs\") pod \"ee44984b-c448-4580-aa5a-637c73d09e19\" (UID: \"ee44984b-c448-4580-aa5a-637c73d09e19\") " Oct 30 16:57:27.637267 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:27.637065 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ee44984b-c448-4580-aa5a-637c73d09e19-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee44984b-c448-4580-aa5a-637c73d09e19-istio-podinfo\") pod \"ee44984b-c448-4580-aa5a-637c73d09e19\" (UID: \"ee44984b-c448-4580-aa5a-637c73d09e19\") " Oct 30 16:57:27.637267 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:27.637158 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd213278-0e61-4779-a644-abfed99fbb9a-kube-api-access-crjgx\" (UniqueName: \"kubernetes.io/projected/dd213278-0e61-4779-a644-abfed99fbb9a-kube-api-access-crjgx\") pod \"dd213278-0e61-4779-a644-abfed99fbb9a\" (UID: \"dd213278-0e61-4779-a644-abfed99fbb9a\") " Oct 30 16:57:27.637267 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:27.637192 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd213278-0e61-4779-a644-abfed99fbb9a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd213278-0e61-4779-a644-abfed99fbb9a-istio-envoy\") pod \"dd213278-0e61-4779-a644-abfed99fbb9a\" (UID: \"dd213278-0e61-4779-a644-abfed99fbb9a\") " Oct 30 16:57:27.637267 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:27.637220 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee44984b-c448-4580-aa5a-637c73d09e19-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee44984b-c448-4580-aa5a-637c73d09e19-istio-envoy\") pod \"ee44984b-c448-4580-aa5a-637c73d09e19\" (UID: \"ee44984b-c448-4580-aa5a-637c73d09e19\") " Oct 30 16:57:27.637649 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:27.637273 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd213278-0e61-4779-a644-abfed99fbb9a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd213278-0e61-4779-a644-abfed99fbb9a-istiod-ca-cert\") pod \"dd213278-0e61-4779-a644-abfed99fbb9a\" (UID: \"dd213278-0e61-4779-a644-abfed99fbb9a\") " Oct 30 16:57:27.637649 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:27.637316 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee44984b-c448-4580-aa5a-637c73d09e19-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee44984b-c448-4580-aa5a-637c73d09e19-credential-socket\") pod \"ee44984b-c448-4580-aa5a-637c73d09e19\" (UID: \"ee44984b-c448-4580-aa5a-637c73d09e19\") " Oct 30 16:57:27.637649 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:27.637354 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd213278-0e61-4779-a644-abfed99fbb9a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd213278-0e61-4779-a644-abfed99fbb9a-istio-data\") pod \"dd213278-0e61-4779-a644-abfed99fbb9a\" (UID: \"dd213278-0e61-4779-a644-abfed99fbb9a\") " Oct 30 16:57:27.637649 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:27.637395 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee44984b-c448-4580-aa5a-637c73d09e19-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee44984b-c448-4580-aa5a-637c73d09e19-workload-socket\") pod \"ee44984b-c448-4580-aa5a-637c73d09e19\" (UID: \"ee44984b-c448-4580-aa5a-637c73d09e19\") " Oct 30 16:57:27.637649 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:27.637404 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd213278-0e61-4779-a644-abfed99fbb9a-workload-certs" pod "dd213278-0e61-4779-a644-abfed99fbb9a" (UID: "dd213278-0e61-4779-a644-abfed99fbb9a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:57:27.637649 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:27.637523 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd213278-0e61-4779-a644-abfed99fbb9a-istio-token\" (UniqueName: \"kubernetes.io/projected/dd213278-0e61-4779-a644-abfed99fbb9a-istio-token\") pod \"dd213278-0e61-4779-a644-abfed99fbb9a\" (UID: \"dd213278-0e61-4779-a644-abfed99fbb9a\") " Oct 30 16:57:27.637649 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:27.637613 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee44984b-c448-4580-aa5a-637c73d09e19-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee44984b-c448-4580-aa5a-637c73d09e19-istiod-ca-cert\") pod \"ee44984b-c448-4580-aa5a-637c73d09e19\" (UID: \"ee44984b-c448-4580-aa5a-637c73d09e19\") " Oct 30 16:57:27.638032 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:27.637685 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd213278-0e61-4779-a644-abfed99fbb9a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd213278-0e61-4779-a644-abfed99fbb9a-credential-socket\") pod \"dd213278-0e61-4779-a644-abfed99fbb9a\" (UID: \"dd213278-0e61-4779-a644-abfed99fbb9a\") " Oct 30 16:57:27.638032 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:27.637732 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee44984b-c448-4580-aa5a-637c73d09e19-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee44984b-c448-4580-aa5a-637c73d09e19-istio-data\") pod \"ee44984b-c448-4580-aa5a-637c73d09e19\" (UID: \"ee44984b-c448-4580-aa5a-637c73d09e19\") " Oct 30 16:57:27.638164 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:27.638130 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd213278-0e61-4779-a644-abfed99fbb9a-workload-socket" pod "dd213278-0e61-4779-a644-abfed99fbb9a" (UID: "dd213278-0e61-4779-a644-abfed99fbb9a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:57:27.638790 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:27.638763 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee44984b-c448-4580-aa5a-637c73d09e19-credential-socket" pod "ee44984b-c448-4580-aa5a-637c73d09e19" (UID: "ee44984b-c448-4580-aa5a-637c73d09e19"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:57:27.638906 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:27.638847 5052 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f43327c0-fce9-4bdf-ae16-2111377c71ad-istio-token\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:57:27.638906 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:27.638873 5052 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd213278-0e61-4779-a644-abfed99fbb9a-workload-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:57:27.638906 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:27.638888 5052 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd213278-0e61-4779-a644-abfed99fbb9a-workload-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:57:27.638906 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:27.638905 5052 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f43327c0-fce9-4bdf-ae16-2111377c71ad-istio-podinfo\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:57:27.639173 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:27.638920 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2ggn\" (UniqueName: \"kubernetes.io/projected/f43327c0-fce9-4bdf-ae16-2111377c71ad-kube-api-access-d2ggn\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:57:27.639173 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:27.638938 5052 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f43327c0-fce9-4bdf-ae16-2111377c71ad-istio-envoy\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:57:27.639173 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:27.639166 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd213278-0e61-4779-a644-abfed99fbb9a-istio-data" pod "dd213278-0e61-4779-a644-abfed99fbb9a" (UID: "dd213278-0e61-4779-a644-abfed99fbb9a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:57:27.639642 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:27.639362 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee44984b-c448-4580-aa5a-637c73d09e19-workload-socket" pod "ee44984b-c448-4580-aa5a-637c73d09e19" (UID: "ee44984b-c448-4580-aa5a-637c73d09e19"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:57:27.641119 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:27.641090 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee44984b-c448-4580-aa5a-637c73d09e19-workload-certs" pod "ee44984b-c448-4580-aa5a-637c73d09e19" (UID: "ee44984b-c448-4580-aa5a-637c73d09e19"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:57:27.641229 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:27.641132 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee44984b-c448-4580-aa5a-637c73d09e19-kube-api-access-69n2v" pod "ee44984b-c448-4580-aa5a-637c73d09e19" (UID: "ee44984b-c448-4580-aa5a-637c73d09e19"). InnerVolumeSpecName "kube-api-access-69n2v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:57:27.641342 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:27.641312 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd213278-0e61-4779-a644-abfed99fbb9a-credential-socket" pod "dd213278-0e61-4779-a644-abfed99fbb9a" (UID: "dd213278-0e61-4779-a644-abfed99fbb9a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:57:27.641633 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:27.641602 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee44984b-c448-4580-aa5a-637c73d09e19-istiod-ca-cert" pod "ee44984b-c448-4580-aa5a-637c73d09e19" (UID: "ee44984b-c448-4580-aa5a-637c73d09e19"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:57:27.641633 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:27.641617 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee44984b-c448-4580-aa5a-637c73d09e19-istio-data" pod "ee44984b-c448-4580-aa5a-637c73d09e19" (UID: "ee44984b-c448-4580-aa5a-637c73d09e19"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:57:27.641795 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:27.641702 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd213278-0e61-4779-a644-abfed99fbb9a-istio-token" pod "dd213278-0e61-4779-a644-abfed99fbb9a" (UID: "dd213278-0e61-4779-a644-abfed99fbb9a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:57:27.641795 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:27.641728 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd213278-0e61-4779-a644-abfed99fbb9a-istiod-ca-cert" pod "dd213278-0e61-4779-a644-abfed99fbb9a" (UID: "dd213278-0e61-4779-a644-abfed99fbb9a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:57:27.641938 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:27.641917 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ee44984b-c448-4580-aa5a-637c73d09e19-istio-podinfo" pod "ee44984b-c448-4580-aa5a-637c73d09e19" (UID: "ee44984b-c448-4580-aa5a-637c73d09e19"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:57:27.641984 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:27.641938 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd213278-0e61-4779-a644-abfed99fbb9a-istio-envoy" pod "dd213278-0e61-4779-a644-abfed99fbb9a" (UID: "dd213278-0e61-4779-a644-abfed99fbb9a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:57:27.642039 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:27.642018 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee44984b-c448-4580-aa5a-637c73d09e19-istio-envoy" pod "ee44984b-c448-4580-aa5a-637c73d09e19" (UID: "ee44984b-c448-4580-aa5a-637c73d09e19"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:57:27.642076 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:27.642041 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dd213278-0e61-4779-a644-abfed99fbb9a-istio-podinfo" pod "dd213278-0e61-4779-a644-abfed99fbb9a" (UID: "dd213278-0e61-4779-a644-abfed99fbb9a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:57:27.642462 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:27.642445 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee44984b-c448-4580-aa5a-637c73d09e19-istio-token" pod "ee44984b-c448-4580-aa5a-637c73d09e19" (UID: "ee44984b-c448-4580-aa5a-637c73d09e19"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:57:27.643778 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:27.643761 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd213278-0e61-4779-a644-abfed99fbb9a-kube-api-access-crjgx" pod "dd213278-0e61-4779-a644-abfed99fbb9a" (UID: "dd213278-0e61-4779-a644-abfed99fbb9a"). InnerVolumeSpecName "kube-api-access-crjgx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:57:27.756660 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:27.755718 5052 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee44984b-c448-4580-aa5a-637c73d09e19-istio-token\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:57:27.756660 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:27.755758 5052 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd213278-0e61-4779-a644-abfed99fbb9a-istio-podinfo\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:57:27.756660 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:27.755772 5052 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee44984b-c448-4580-aa5a-637c73d09e19-workload-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:57:27.756660 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:27.755785 5052 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee44984b-c448-4580-aa5a-637c73d09e19-istio-podinfo\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:57:27.756660 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:27.755800 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crjgx\" (UniqueName: \"kubernetes.io/projected/dd213278-0e61-4779-a644-abfed99fbb9a-kube-api-access-crjgx\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:57:27.756660 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:27.755815 5052 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd213278-0e61-4779-a644-abfed99fbb9a-istio-envoy\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:57:27.756660 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:27.755826 5052 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee44984b-c448-4580-aa5a-637c73d09e19-istio-envoy\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:57:27.756660 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:27.755841 5052 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd213278-0e61-4779-a644-abfed99fbb9a-istiod-ca-cert\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:57:27.756660 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:27.755853 5052 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee44984b-c448-4580-aa5a-637c73d09e19-credential-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:57:27.756660 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:27.755865 5052 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd213278-0e61-4779-a644-abfed99fbb9a-istio-data\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:57:27.756660 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:27.755877 5052 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee44984b-c448-4580-aa5a-637c73d09e19-workload-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:57:27.756660 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:27.755889 5052 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd213278-0e61-4779-a644-abfed99fbb9a-istio-token\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:57:27.756660 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:27.755902 5052 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee44984b-c448-4580-aa5a-637c73d09e19-istiod-ca-cert\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:57:27.756660 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:27.755914 5052 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd213278-0e61-4779-a644-abfed99fbb9a-credential-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:57:27.756660 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:27.755926 5052 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee44984b-c448-4580-aa5a-637c73d09e19-istio-data\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:57:27.756660 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:27.755938 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69n2v\" (UniqueName: \"kubernetes.io/projected/ee44984b-c448-4580-aa5a-637c73d09e19-kube-api-access-69n2v\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:57:28.220822 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:28.220790 5052 generic.go:356] "Generic (PLEG): container finished" podID="dd213278-0e61-4779-a644-abfed99fbb9a" containerID="e9052ffa09f683f1ff1770365e18bfcd37e8a9ba23795838e0125f2898f05cd4" exitCode=0 Oct 30 16:57:28.221251 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:28.220847 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-8cxdf" Oct 30 16:57:28.221251 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:28.220870 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-8cxdf" event={"ID":"dd213278-0e61-4779-a644-abfed99fbb9a","Type":"ContainerDied","Data":"e9052ffa09f683f1ff1770365e18bfcd37e8a9ba23795838e0125f2898f05cd4"} Oct 30 16:57:28.221251 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:28.220931 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-8cxdf" event={"ID":"dd213278-0e61-4779-a644-abfed99fbb9a","Type":"ContainerDied","Data":"ef6b6c3b26a22d318ee4a4e89a3682754e4bbda33aaedc11922a1635ba756638"} Oct 30 16:57:28.221251 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:28.220964 5052 scope.go:122] "RemoveContainer" containerID="e9052ffa09f683f1ff1770365e18bfcd37e8a9ba23795838e0125f2898f05cd4" Oct 30 16:57:28.222405 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:28.222391 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-7xqp2" Oct 30 16:57:28.222523 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:28.222448 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-7xqp2" event={"ID":"f43327c0-fce9-4bdf-ae16-2111377c71ad","Type":"ContainerDied","Data":"72375590bd1ed1f62b754af8e457832e830d7d5e5dd823caf636b52e4b9c4f42"} Oct 30 16:57:28.224105 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:28.224081 5052 generic.go:356] "Generic (PLEG): container finished" podID="ee44984b-c448-4580-aa5a-637c73d09e19" containerID="d46b8129f7de87e1ddf19c8dd248d0ec3e2f04a3ccf0bf6d3f1a6876fbb75a06" exitCode=0 Oct 30 16:57:28.224239 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:28.224128 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cp9m8" event={"ID":"ee44984b-c448-4580-aa5a-637c73d09e19","Type":"ContainerDied","Data":"d46b8129f7de87e1ddf19c8dd248d0ec3e2f04a3ccf0bf6d3f1a6876fbb75a06"} Oct 30 16:57:28.224239 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:28.224159 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cp9m8" event={"ID":"ee44984b-c448-4580-aa5a-637c73d09e19","Type":"ContainerDied","Data":"92f82870a9f063cacf8016a3fb8ef10ad6b9615e4b43bf5dc0970c414ee94074"} Oct 30 16:57:28.224239 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:28.224189 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cp9m8" Oct 30 16:57:28.232765 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:28.232734 5052 scope.go:122] "RemoveContainer" containerID="e9052ffa09f683f1ff1770365e18bfcd37e8a9ba23795838e0125f2898f05cd4" Oct 30 16:57:28.233037 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:57:28.233016 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9052ffa09f683f1ff1770365e18bfcd37e8a9ba23795838e0125f2898f05cd4\": container with ID starting with e9052ffa09f683f1ff1770365e18bfcd37e8a9ba23795838e0125f2898f05cd4 not found: ID does not exist" containerID="e9052ffa09f683f1ff1770365e18bfcd37e8a9ba23795838e0125f2898f05cd4" Oct 30 16:57:28.233115 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:28.233044 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9052ffa09f683f1ff1770365e18bfcd37e8a9ba23795838e0125f2898f05cd4"} err="failed to get container status \"e9052ffa09f683f1ff1770365e18bfcd37e8a9ba23795838e0125f2898f05cd4\": rpc error: code = NotFound desc = could not find container \"e9052ffa09f683f1ff1770365e18bfcd37e8a9ba23795838e0125f2898f05cd4\": container with ID starting with e9052ffa09f683f1ff1770365e18bfcd37e8a9ba23795838e0125f2898f05cd4 not found: ID does not exist" Oct 30 16:57:28.233115 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:28.233061 5052 scope.go:122] "RemoveContainer" containerID="fb4ec2bde8c8f4328e3878bdcf85cb0e0bb3f2d0735307e9e6413b622b30c2ea" Oct 30 16:57:28.245236 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:28.245209 5052 scope.go:122] "RemoveContainer" containerID="d46b8129f7de87e1ddf19c8dd248d0ec3e2f04a3ccf0bf6d3f1a6876fbb75a06" Oct 30 16:57:28.255940 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:28.255921 5052 scope.go:122] "RemoveContainer" containerID="d46b8129f7de87e1ddf19c8dd248d0ec3e2f04a3ccf0bf6d3f1a6876fbb75a06" Oct 30 16:57:28.256222 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:57:28.256204 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d46b8129f7de87e1ddf19c8dd248d0ec3e2f04a3ccf0bf6d3f1a6876fbb75a06\": container with ID starting with d46b8129f7de87e1ddf19c8dd248d0ec3e2f04a3ccf0bf6d3f1a6876fbb75a06 not found: ID does not exist" containerID="d46b8129f7de87e1ddf19c8dd248d0ec3e2f04a3ccf0bf6d3f1a6876fbb75a06" Oct 30 16:57:28.256281 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:28.256229 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d46b8129f7de87e1ddf19c8dd248d0ec3e2f04a3ccf0bf6d3f1a6876fbb75a06"} err="failed to get container status \"d46b8129f7de87e1ddf19c8dd248d0ec3e2f04a3ccf0bf6d3f1a6876fbb75a06\": rpc error: code = NotFound desc = could not find container \"d46b8129f7de87e1ddf19c8dd248d0ec3e2f04a3ccf0bf6d3f1a6876fbb75a06\": container with ID starting with d46b8129f7de87e1ddf19c8dd248d0ec3e2f04a3ccf0bf6d3f1a6876fbb75a06 not found: ID does not exist" Oct 30 16:57:28.311095 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:28.310869 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cp9m8"] Oct 30 16:57:28.323420 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:28.323392 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cp9m8"] Oct 30 16:57:28.350935 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:28.350911 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-7xqp2"] Oct 30 16:57:28.362487 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:28.355729 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-7xqp2"] Oct 30 16:57:28.395144 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:28.395113 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-8cxdf"] Oct 30 16:57:28.403452 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:28.403427 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-8cxdf"] Oct 30 16:57:28.600591 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:28.600499 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd213278-0e61-4779-a644-abfed99fbb9a" path="/var/lib/kubelet/pods/dd213278-0e61-4779-a644-abfed99fbb9a/volumes" Oct 30 16:57:28.601042 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:28.601026 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee44984b-c448-4580-aa5a-637c73d09e19" path="/var/lib/kubelet/pods/ee44984b-c448-4580-aa5a-637c73d09e19/volumes" Oct 30 16:57:28.601483 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:28.601470 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f43327c0-fce9-4bdf-ae16-2111377c71ad" path="/var/lib/kubelet/pods/f43327c0-fce9-4bdf-ae16-2111377c71ad/volumes" Oct 30 16:57:33.685272 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:33.685236 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-b4xqs"] Oct 30 16:57:33.686079 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:33.686063 5052 cpu_manager.go:414] "containerMap: removing container" podUID="dd213278-0e61-4779-a644-abfed99fbb9a" containerName="istio-proxy" Oct 30 16:57:33.686079 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:33.686079 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd213278-0e61-4779-a644-abfed99fbb9a" containerName="istio-proxy" Oct 30 16:57:33.686184 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:33.686087 5052 cpu_manager.go:414] "containerMap: removing container" podUID="ee44984b-c448-4580-aa5a-637c73d09e19" containerName="istio-proxy" Oct 30 16:57:33.686184 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:33.686093 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee44984b-c448-4580-aa5a-637c73d09e19" containerName="istio-proxy" Oct 30 16:57:33.686184 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:33.686116 5052 cpu_manager.go:414] "containerMap: removing container" podUID="f43327c0-fce9-4bdf-ae16-2111377c71ad" containerName="istio-proxy" Oct 30 16:57:33.686184 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:33.686121 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f43327c0-fce9-4bdf-ae16-2111377c71ad" containerName="istio-proxy" Oct 30 16:57:33.686184 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:33.686157 5052 cpu_manager.go:414] "containerMap: removing container" podUID="3b3da2b1-c132-453a-b8e9-512a43e613ff" containerName="payload-processing" Oct 30 16:57:33.686184 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:33.686163 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b3da2b1-c132-453a-b8e9-512a43e613ff" containerName="payload-processing" Oct 30 16:57:33.686385 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:33.686345 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd213278-0e61-4779-a644-abfed99fbb9a" containerName="istio-proxy" Oct 30 16:57:33.686385 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:33.686360 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee44984b-c448-4580-aa5a-637c73d09e19" containerName="istio-proxy" Oct 30 16:57:33.686385 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:33.686374 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="f43327c0-fce9-4bdf-ae16-2111377c71ad" containerName="istio-proxy" Oct 30 16:57:33.686515 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:33.686397 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b3da2b1-c132-453a-b8e9-512a43e613ff" containerName="payload-processing" Oct 30 16:57:33.693284 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:33.693258 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-b4xqs" Oct 30 16:57:33.717291 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:33.717258 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-b4xqs"] Oct 30 16:57:33.781694 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:33.781657 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/157dcd12-c7ff-488a-9b49-fc2a36012728-workload-socket\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-b4xqs\" (UID: \"157dcd12-c7ff-488a-9b49-fc2a36012728\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-b4xqs" Oct 30 16:57:33.781892 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:33.781846 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/157dcd12-c7ff-488a-9b49-fc2a36012728-istiod-ca-cert\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-b4xqs\" (UID: \"157dcd12-c7ff-488a-9b49-fc2a36012728\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-b4xqs" Oct 30 16:57:33.781956 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:33.781913 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jfmp\" (UniqueName: \"kubernetes.io/projected/157dcd12-c7ff-488a-9b49-fc2a36012728-kube-api-access-6jfmp\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-b4xqs\" (UID: \"157dcd12-c7ff-488a-9b49-fc2a36012728\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-b4xqs" Oct 30 16:57:33.782043 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:33.782021 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/157dcd12-c7ff-488a-9b49-fc2a36012728-workload-certs\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-b4xqs\" (UID: \"157dcd12-c7ff-488a-9b49-fc2a36012728\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-b4xqs" Oct 30 16:57:33.782135 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:33.782117 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/157dcd12-c7ff-488a-9b49-fc2a36012728-istio-token\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-b4xqs\" (UID: \"157dcd12-c7ff-488a-9b49-fc2a36012728\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-b4xqs" Oct 30 16:57:33.782200 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:33.782169 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/157dcd12-c7ff-488a-9b49-fc2a36012728-istio-podinfo\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-b4xqs\" (UID: \"157dcd12-c7ff-488a-9b49-fc2a36012728\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-b4xqs" Oct 30 16:57:33.782264 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:33.782250 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/157dcd12-c7ff-488a-9b49-fc2a36012728-credential-socket\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-b4xqs\" (UID: \"157dcd12-c7ff-488a-9b49-fc2a36012728\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-b4xqs" Oct 30 16:57:33.782337 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:33.782323 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/157dcd12-c7ff-488a-9b49-fc2a36012728-istio-envoy\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-b4xqs\" (UID: \"157dcd12-c7ff-488a-9b49-fc2a36012728\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-b4xqs" Oct 30 16:57:33.782390 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:33.782368 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/157dcd12-c7ff-488a-9b49-fc2a36012728-istio-data\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-b4xqs\" (UID: \"157dcd12-c7ff-488a-9b49-fc2a36012728\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-b4xqs" Oct 30 16:57:33.884432 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:33.884398 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/157dcd12-c7ff-488a-9b49-fc2a36012728-istiod-ca-cert\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-b4xqs\" (UID: \"157dcd12-c7ff-488a-9b49-fc2a36012728\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-b4xqs" Oct 30 16:57:33.884635 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:33.884474 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jfmp\" (UniqueName: \"kubernetes.io/projected/157dcd12-c7ff-488a-9b49-fc2a36012728-kube-api-access-6jfmp\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-b4xqs\" (UID: \"157dcd12-c7ff-488a-9b49-fc2a36012728\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-b4xqs" Oct 30 16:57:33.884635 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:33.884523 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/157dcd12-c7ff-488a-9b49-fc2a36012728-workload-certs\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-b4xqs\" (UID: \"157dcd12-c7ff-488a-9b49-fc2a36012728\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-b4xqs" Oct 30 16:57:33.884635 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:33.884595 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/157dcd12-c7ff-488a-9b49-fc2a36012728-istio-token\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-b4xqs\" (UID: \"157dcd12-c7ff-488a-9b49-fc2a36012728\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-b4xqs" Oct 30 16:57:33.884799 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:33.884641 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/157dcd12-c7ff-488a-9b49-fc2a36012728-istio-podinfo\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-b4xqs\" (UID: \"157dcd12-c7ff-488a-9b49-fc2a36012728\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-b4xqs" Oct 30 16:57:33.884799 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:33.884680 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/157dcd12-c7ff-488a-9b49-fc2a36012728-credential-socket\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-b4xqs\" (UID: \"157dcd12-c7ff-488a-9b49-fc2a36012728\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-b4xqs" Oct 30 16:57:33.884914 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:33.884807 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/157dcd12-c7ff-488a-9b49-fc2a36012728-istio-envoy\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-b4xqs\" (UID: \"157dcd12-c7ff-488a-9b49-fc2a36012728\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-b4xqs" Oct 30 16:57:33.884914 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:33.884833 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/157dcd12-c7ff-488a-9b49-fc2a36012728-istio-data\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-b4xqs\" (UID: \"157dcd12-c7ff-488a-9b49-fc2a36012728\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-b4xqs" Oct 30 16:57:33.884914 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:33.884904 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/157dcd12-c7ff-488a-9b49-fc2a36012728-workload-socket\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-b4xqs\" (UID: \"157dcd12-c7ff-488a-9b49-fc2a36012728\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-b4xqs" Oct 30 16:57:33.885072 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:33.884996 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/157dcd12-c7ff-488a-9b49-fc2a36012728-workload-certs\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-b4xqs\" (UID: \"157dcd12-c7ff-488a-9b49-fc2a36012728\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-b4xqs" Oct 30 16:57:33.885286 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:33.885257 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/157dcd12-c7ff-488a-9b49-fc2a36012728-workload-socket\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-b4xqs\" (UID: \"157dcd12-c7ff-488a-9b49-fc2a36012728\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-b4xqs" Oct 30 16:57:33.885400 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:33.885321 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/157dcd12-c7ff-488a-9b49-fc2a36012728-istio-data\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-b4xqs\" (UID: \"157dcd12-c7ff-488a-9b49-fc2a36012728\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-b4xqs" Oct 30 16:57:33.885400 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:33.885370 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/157dcd12-c7ff-488a-9b49-fc2a36012728-credential-socket\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-b4xqs\" (UID: \"157dcd12-c7ff-488a-9b49-fc2a36012728\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-b4xqs" Oct 30 16:57:33.885650 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:33.885437 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/157dcd12-c7ff-488a-9b49-fc2a36012728-istiod-ca-cert\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-b4xqs\" (UID: \"157dcd12-c7ff-488a-9b49-fc2a36012728\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-b4xqs" Oct 30 16:57:33.887270 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:33.887246 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/157dcd12-c7ff-488a-9b49-fc2a36012728-istio-envoy\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-b4xqs\" (UID: \"157dcd12-c7ff-488a-9b49-fc2a36012728\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-b4xqs" Oct 30 16:57:33.887357 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:33.887316 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/157dcd12-c7ff-488a-9b49-fc2a36012728-istio-podinfo\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-b4xqs\" (UID: \"157dcd12-c7ff-488a-9b49-fc2a36012728\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-b4xqs" Oct 30 16:57:33.909099 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:33.909075 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jfmp\" (UniqueName: \"kubernetes.io/projected/157dcd12-c7ff-488a-9b49-fc2a36012728-kube-api-access-6jfmp\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-b4xqs\" (UID: \"157dcd12-c7ff-488a-9b49-fc2a36012728\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-b4xqs" Oct 30 16:57:33.909224 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:33.909205 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/157dcd12-c7ff-488a-9b49-fc2a36012728-istio-token\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-b4xqs\" (UID: \"157dcd12-c7ff-488a-9b49-fc2a36012728\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-b4xqs" Oct 30 16:57:34.021338 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:34.021259 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-b4xqs" Oct 30 16:57:34.365446 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:34.365418 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-b4xqs"] Oct 30 16:57:34.366057 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:57:34.366025 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod157dcd12_c7ff_488a_9b49_fc2a36012728.slice/crio-90788c96c84390861bd52774e75de7a46e46cf5a307b620f220e5218f94d943f WatchSource:0}: Error finding container 90788c96c84390861bd52774e75de7a46e46cf5a307b620f220e5218f94d943f: Status 404 returned error can't find the container with id 90788c96c84390861bd52774e75de7a46e46cf5a307b620f220e5218f94d943f Oct 30 16:57:34.369406 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:34.369371 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:57:34.369495 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:34.369443 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:57:34.369495 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:34.369473 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:57:35.304926 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:35.304893 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-b4xqs" event={"ID":"157dcd12-c7ff-488a-9b49-fc2a36012728","Type":"ContainerStarted","Data":"8ff156124dae0f864e03d67d92029d77e36dfc3b4a12774dddd2f52482683256"} Oct 30 16:57:35.305400 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:35.304942 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-b4xqs" event={"ID":"157dcd12-c7ff-488a-9b49-fc2a36012728","Type":"ContainerStarted","Data":"90788c96c84390861bd52774e75de7a46e46cf5a307b620f220e5218f94d943f"} Oct 30 16:57:35.377697 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:35.377644 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-b4xqs" podStartSLOduration=2.377630546 podStartE2EDuration="2.377630546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:57:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:57:35.374250713 +0000 UTC m=+2210.901585927" watchObservedRunningTime="2026-10-30 16:57:35.377630546 +0000 UTC m=+2210.904965764" Oct 30 16:57:36.022390 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:36.022354 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-b4xqs" Oct 30 16:57:36.024047 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:36.024022 5052 patch_prober.go:29] interesting pod/e2e-inf-b-689899-openshift-default-66cbd45f99-b4xqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Oct 30 16:57:36.024137 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:36.024072 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-b4xqs" podUID="157dcd12-c7ff-488a-9b49-fc2a36012728" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Oct 30 16:57:36.856603 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:36.852955 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-ea77d8-openshift-default-68fd566499-wqdpf"] Oct 30 16:57:36.862974 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:36.862950 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-ea77d8-openshift-default-68fd566499-wqdpf"] Oct 30 16:57:36.863100 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:36.863040 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ea77d8-openshift-default-68fd566499-wqdpf" Oct 30 16:57:36.879779 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:36.879752 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-ea77d8-openshift-default-dockercfg-hkvs8\"" Oct 30 16:57:36.963640 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:36.963605 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e-credential-socket\") pod \"e2e-inf-a-ea77d8-openshift-default-68fd566499-wqdpf\" (UID: \"2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e\") " pod="openshift-ingress/e2e-inf-a-ea77d8-openshift-default-68fd566499-wqdpf" Oct 30 16:57:36.963793 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:36.963649 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e-istio-envoy\") pod \"e2e-inf-a-ea77d8-openshift-default-68fd566499-wqdpf\" (UID: \"2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e\") " pod="openshift-ingress/e2e-inf-a-ea77d8-openshift-default-68fd566499-wqdpf" Oct 30 16:57:36.963793 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:36.963738 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e-istiod-ca-cert\") pod \"e2e-inf-a-ea77d8-openshift-default-68fd566499-wqdpf\" (UID: \"2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e\") " pod="openshift-ingress/e2e-inf-a-ea77d8-openshift-default-68fd566499-wqdpf" Oct 30 16:57:36.963886 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:36.963855 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e-workload-certs\") pod \"e2e-inf-a-ea77d8-openshift-default-68fd566499-wqdpf\" (UID: \"2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e\") " pod="openshift-ingress/e2e-inf-a-ea77d8-openshift-default-68fd566499-wqdpf" Oct 30 16:57:36.963983 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:36.963969 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e-istio-data\") pod \"e2e-inf-a-ea77d8-openshift-default-68fd566499-wqdpf\" (UID: \"2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e\") " pod="openshift-ingress/e2e-inf-a-ea77d8-openshift-default-68fd566499-wqdpf" Oct 30 16:57:36.964020 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:36.963998 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5dpw\" (UniqueName: \"kubernetes.io/projected/2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e-kube-api-access-r5dpw\") pod \"e2e-inf-a-ea77d8-openshift-default-68fd566499-wqdpf\" (UID: \"2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e\") " pod="openshift-ingress/e2e-inf-a-ea77d8-openshift-default-68fd566499-wqdpf" Oct 30 16:57:36.964052 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:36.964026 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e-workload-socket\") pod \"e2e-inf-a-ea77d8-openshift-default-68fd566499-wqdpf\" (UID: \"2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e\") " pod="openshift-ingress/e2e-inf-a-ea77d8-openshift-default-68fd566499-wqdpf" Oct 30 16:57:36.964110 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:36.964096 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e-istio-token\") pod \"e2e-inf-a-ea77d8-openshift-default-68fd566499-wqdpf\" (UID: \"2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e\") " pod="openshift-ingress/e2e-inf-a-ea77d8-openshift-default-68fd566499-wqdpf" Oct 30 16:57:36.964183 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:36.964172 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e-istio-podinfo\") pod \"e2e-inf-a-ea77d8-openshift-default-68fd566499-wqdpf\" (UID: \"2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e\") " pod="openshift-ingress/e2e-inf-a-ea77d8-openshift-default-68fd566499-wqdpf" Oct 30 16:57:37.022447 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:37.022411 5052 patch_prober.go:29] interesting pod/e2e-inf-b-689899-openshift-default-66cbd45f99-b4xqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Oct 30 16:57:37.022637 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:37.022463 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-b4xqs" podUID="157dcd12-c7ff-488a-9b49-fc2a36012728" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Oct 30 16:57:37.065552 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:37.065519 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e-workload-certs\") pod \"e2e-inf-a-ea77d8-openshift-default-68fd566499-wqdpf\" (UID: \"2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e\") " pod="openshift-ingress/e2e-inf-a-ea77d8-openshift-default-68fd566499-wqdpf" Oct 30 16:57:37.065761 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:37.065642 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e-istio-data\") pod \"e2e-inf-a-ea77d8-openshift-default-68fd566499-wqdpf\" (UID: \"2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e\") " pod="openshift-ingress/e2e-inf-a-ea77d8-openshift-default-68fd566499-wqdpf" Oct 30 16:57:37.065761 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:37.065668 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5dpw\" (UniqueName: \"kubernetes.io/projected/2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e-kube-api-access-r5dpw\") pod \"e2e-inf-a-ea77d8-openshift-default-68fd566499-wqdpf\" (UID: \"2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e\") " pod="openshift-ingress/e2e-inf-a-ea77d8-openshift-default-68fd566499-wqdpf" Oct 30 16:57:37.065761 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:37.065700 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e-workload-socket\") pod \"e2e-inf-a-ea77d8-openshift-default-68fd566499-wqdpf\" (UID: \"2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e\") " pod="openshift-ingress/e2e-inf-a-ea77d8-openshift-default-68fd566499-wqdpf" Oct 30 16:57:37.065761 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:37.065721 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e-istio-token\") pod \"e2e-inf-a-ea77d8-openshift-default-68fd566499-wqdpf\" (UID: \"2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e\") " pod="openshift-ingress/e2e-inf-a-ea77d8-openshift-default-68fd566499-wqdpf" Oct 30 16:57:37.065995 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:37.065825 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e-istio-podinfo\") pod \"e2e-inf-a-ea77d8-openshift-default-68fd566499-wqdpf\" (UID: \"2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e\") " pod="openshift-ingress/e2e-inf-a-ea77d8-openshift-default-68fd566499-wqdpf" Oct 30 16:57:37.065995 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:37.065964 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e-credential-socket\") pod \"e2e-inf-a-ea77d8-openshift-default-68fd566499-wqdpf\" (UID: \"2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e\") " pod="openshift-ingress/e2e-inf-a-ea77d8-openshift-default-68fd566499-wqdpf" Oct 30 16:57:37.066162 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:37.066003 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e-istio-envoy\") pod \"e2e-inf-a-ea77d8-openshift-default-68fd566499-wqdpf\" (UID: \"2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e\") " pod="openshift-ingress/e2e-inf-a-ea77d8-openshift-default-68fd566499-wqdpf" Oct 30 16:57:37.066162 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:37.066026 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e-istio-data\") pod \"e2e-inf-a-ea77d8-openshift-default-68fd566499-wqdpf\" (UID: \"2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e\") " pod="openshift-ingress/e2e-inf-a-ea77d8-openshift-default-68fd566499-wqdpf" Oct 30 16:57:37.066162 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:37.066048 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e-istiod-ca-cert\") pod \"e2e-inf-a-ea77d8-openshift-default-68fd566499-wqdpf\" (UID: \"2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e\") " pod="openshift-ingress/e2e-inf-a-ea77d8-openshift-default-68fd566499-wqdpf" Oct 30 16:57:37.066310 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:37.066230 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e-workload-socket\") pod \"e2e-inf-a-ea77d8-openshift-default-68fd566499-wqdpf\" (UID: \"2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e\") " pod="openshift-ingress/e2e-inf-a-ea77d8-openshift-default-68fd566499-wqdpf" Oct 30 16:57:37.066310 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:37.066264 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e-credential-socket\") pod \"e2e-inf-a-ea77d8-openshift-default-68fd566499-wqdpf\" (UID: \"2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e\") " pod="openshift-ingress/e2e-inf-a-ea77d8-openshift-default-68fd566499-wqdpf" Oct 30 16:57:37.066426 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:37.066346 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e-workload-certs\") pod \"e2e-inf-a-ea77d8-openshift-default-68fd566499-wqdpf\" (UID: \"2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e\") " pod="openshift-ingress/e2e-inf-a-ea77d8-openshift-default-68fd566499-wqdpf" Oct 30 16:57:37.066762 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:37.066741 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e-istiod-ca-cert\") pod \"e2e-inf-a-ea77d8-openshift-default-68fd566499-wqdpf\" (UID: \"2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e\") " pod="openshift-ingress/e2e-inf-a-ea77d8-openshift-default-68fd566499-wqdpf" Oct 30 16:57:37.068348 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:37.068322 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e-istio-envoy\") pod \"e2e-inf-a-ea77d8-openshift-default-68fd566499-wqdpf\" (UID: \"2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e\") " pod="openshift-ingress/e2e-inf-a-ea77d8-openshift-default-68fd566499-wqdpf" Oct 30 16:57:37.068851 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:37.068827 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e-istio-podinfo\") pod \"e2e-inf-a-ea77d8-openshift-default-68fd566499-wqdpf\" (UID: \"2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e\") " pod="openshift-ingress/e2e-inf-a-ea77d8-openshift-default-68fd566499-wqdpf" Oct 30 16:57:37.091418 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:37.091394 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e-istio-token\") pod \"e2e-inf-a-ea77d8-openshift-default-68fd566499-wqdpf\" (UID: \"2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e\") " pod="openshift-ingress/e2e-inf-a-ea77d8-openshift-default-68fd566499-wqdpf" Oct 30 16:57:37.091730 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:37.091706 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5dpw\" (UniqueName: \"kubernetes.io/projected/2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e-kube-api-access-r5dpw\") pod \"e2e-inf-a-ea77d8-openshift-default-68fd566499-wqdpf\" (UID: \"2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e\") " pod="openshift-ingress/e2e-inf-a-ea77d8-openshift-default-68fd566499-wqdpf" Oct 30 16:57:37.187358 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:37.187329 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ea77d8-openshift-default-68fd566499-wqdpf" Oct 30 16:57:37.555882 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:57:37.555840 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b5e7ff8_4590_4afd_bb81_cc7df5dfd20e.slice/crio-079834d7714bff7816beafeafe0d878be4ac03097a0186c2e7d5ff2311a00a20 WatchSource:0}: Error finding container 079834d7714bff7816beafeafe0d878be4ac03097a0186c2e7d5ff2311a00a20: Status 404 returned error can't find the container with id 079834d7714bff7816beafeafe0d878be4ac03097a0186c2e7d5ff2311a00a20 Oct 30 16:57:37.557718 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:37.557682 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-ea77d8-openshift-default-68fd566499-wqdpf"] Oct 30 16:57:37.559391 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:37.559353 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:57:37.559487 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:37.559436 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:57:37.559487 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:37.559481 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:57:38.022299 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:38.022261 5052 patch_prober.go:29] interesting pod/e2e-inf-b-689899-openshift-default-66cbd45f99-b4xqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Oct 30 16:57:38.022802 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:38.022332 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-b4xqs" podUID="157dcd12-c7ff-488a-9b49-fc2a36012728" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Oct 30 16:57:38.344299 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:38.344197 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ea77d8-openshift-default-68fd566499-wqdpf" event={"ID":"2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e","Type":"ContainerStarted","Data":"16c232ff5edc96e38866097a543394dd5e754303930d96105ed7f288fee430f1"} Oct 30 16:57:38.344611 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:38.344568 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ea77d8-openshift-default-68fd566499-wqdpf" event={"ID":"2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e","Type":"ContainerStarted","Data":"079834d7714bff7816beafeafe0d878be4ac03097a0186c2e7d5ff2311a00a20"} Oct 30 16:57:39.021626 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:39.021556 5052 patch_prober.go:29] interesting pod/e2e-inf-b-689899-openshift-default-66cbd45f99-b4xqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Oct 30 16:57:39.021812 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:39.021652 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-b4xqs" podUID="157dcd12-c7ff-488a-9b49-fc2a36012728" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Oct 30 16:57:39.188086 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:39.188050 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-ea77d8-openshift-default-68fd566499-wqdpf" Oct 30 16:57:39.189253 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:39.189229 5052 patch_prober.go:29] interesting pod/e2e-inf-a-ea77d8-openshift-default-68fd566499-wqdpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Oct 30 16:57:39.189313 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:39.189281 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ea77d8-openshift-default-68fd566499-wqdpf" podUID="2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Oct 30 16:57:40.021995 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:40.021959 5052 patch_prober.go:29] interesting pod/e2e-inf-b-689899-openshift-default-66cbd45f99-b4xqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Oct 30 16:57:40.022271 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:40.022028 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-b4xqs" podUID="157dcd12-c7ff-488a-9b49-fc2a36012728" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Oct 30 16:57:40.187937 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:40.187785 5052 patch_prober.go:29] interesting pod/e2e-inf-a-ea77d8-openshift-default-68fd566499-wqdpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Oct 30 16:57:40.187937 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:40.187866 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ea77d8-openshift-default-68fd566499-wqdpf" podUID="2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Oct 30 16:57:41.021727 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:41.021692 5052 patch_prober.go:29] interesting pod/e2e-inf-b-689899-openshift-default-66cbd45f99-b4xqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Oct 30 16:57:41.021899 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:41.021752 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-b4xqs" podUID="157dcd12-c7ff-488a-9b49-fc2a36012728" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Oct 30 16:57:41.187837 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:41.187796 5052 patch_prober.go:29] interesting pod/e2e-inf-a-ea77d8-openshift-default-68fd566499-wqdpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Oct 30 16:57:41.188037 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:41.187863 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ea77d8-openshift-default-68fd566499-wqdpf" podUID="2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Oct 30 16:57:42.027295 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:42.027263 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-b4xqs" Oct 30 16:57:42.031605 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:42.028684 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-b4xqs" Oct 30 16:57:42.047603 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:42.040533 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-b4xqs" Oct 30 16:57:42.089822 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:42.089768 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-ea77d8-openshift-default-68fd566499-wqdpf" podStartSLOduration=6.089754103 podStartE2EDuration="6.089754103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:57:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:57:38.428038797 +0000 UTC m=+2213.955374028" watchObservedRunningTime="2026-10-30 16:57:42.089754103 +0000 UTC m=+2217.617089330" Oct 30 16:57:42.188657 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:42.188620 5052 patch_prober.go:29] interesting pod/e2e-inf-a-ea77d8-openshift-default-68fd566499-wqdpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Oct 30 16:57:42.188854 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:42.188685 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ea77d8-openshift-default-68fd566499-wqdpf" podUID="2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Oct 30 16:57:43.187766 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:43.187735 5052 patch_prober.go:29] interesting pod/e2e-inf-a-ea77d8-openshift-default-68fd566499-wqdpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Oct 30 16:57:43.188176 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:43.187784 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ea77d8-openshift-default-68fd566499-wqdpf" podUID="2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Oct 30 16:57:44.376379 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:44.376339 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-ea77d8-openshift-default-68fd566499-wqdpf" Oct 30 16:57:44.377558 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:44.377529 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-ea77d8-openshift-default-68fd566499-wqdpf" Oct 30 16:57:44.378802 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:44.378782 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-ea77d8-openshift-default-68fd566499-wqdpf" Oct 30 16:57:45.226098 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:45.226067 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-qq9qb"] Oct 30 16:57:45.232398 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:45.232375 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-qq9qb" Oct 30 16:57:45.253948 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:45.253919 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-qq9qb"] Oct 30 16:57:45.262249 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:45.262217 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-3ed10a-openshift-default-dockercfg-gt5hj\"" Oct 30 16:57:45.349281 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:45.349238 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f571ed90-9a31-47d2-8c4b-9bc57a9975b4-istio-token\") pod \"e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-qq9qb\" (UID: \"f571ed90-9a31-47d2-8c4b-9bc57a9975b4\") " pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-qq9qb" Oct 30 16:57:45.349485 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:45.349359 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f571ed90-9a31-47d2-8c4b-9bc57a9975b4-credential-socket\") pod \"e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-qq9qb\" (UID: \"f571ed90-9a31-47d2-8c4b-9bc57a9975b4\") " pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-qq9qb" Oct 30 16:57:45.349594 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:45.349553 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f571ed90-9a31-47d2-8c4b-9bc57a9975b4-istio-envoy\") pod \"e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-qq9qb\" (UID: \"f571ed90-9a31-47d2-8c4b-9bc57a9975b4\") " pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-qq9qb" Oct 30 16:57:45.349666 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:45.349607 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dql4s\" (UniqueName: \"kubernetes.io/projected/f571ed90-9a31-47d2-8c4b-9bc57a9975b4-kube-api-access-dql4s\") pod \"e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-qq9qb\" (UID: \"f571ed90-9a31-47d2-8c4b-9bc57a9975b4\") " pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-qq9qb" Oct 30 16:57:45.349747 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:45.349730 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f571ed90-9a31-47d2-8c4b-9bc57a9975b4-workload-certs\") pod \"e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-qq9qb\" (UID: \"f571ed90-9a31-47d2-8c4b-9bc57a9975b4\") " pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-qq9qb" Oct 30 16:57:45.349808 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:45.349764 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f571ed90-9a31-47d2-8c4b-9bc57a9975b4-istio-podinfo\") pod \"e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-qq9qb\" (UID: \"f571ed90-9a31-47d2-8c4b-9bc57a9975b4\") " pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-qq9qb" Oct 30 16:57:45.349867 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:45.349830 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f571ed90-9a31-47d2-8c4b-9bc57a9975b4-istiod-ca-cert\") pod \"e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-qq9qb\" (UID: \"f571ed90-9a31-47d2-8c4b-9bc57a9975b4\") " pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-qq9qb" Oct 30 16:57:45.349927 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:45.349876 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f571ed90-9a31-47d2-8c4b-9bc57a9975b4-istio-data\") pod \"e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-qq9qb\" (UID: \"f571ed90-9a31-47d2-8c4b-9bc57a9975b4\") " pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-qq9qb" Oct 30 16:57:45.349985 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:45.349943 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f571ed90-9a31-47d2-8c4b-9bc57a9975b4-workload-socket\") pod \"e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-qq9qb\" (UID: \"f571ed90-9a31-47d2-8c4b-9bc57a9975b4\") " pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-qq9qb" Oct 30 16:57:45.452026 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:45.451993 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f571ed90-9a31-47d2-8c4b-9bc57a9975b4-istio-envoy\") pod \"e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-qq9qb\" (UID: \"f571ed90-9a31-47d2-8c4b-9bc57a9975b4\") " pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-qq9qb" Oct 30 16:57:45.452026 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:45.452028 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dql4s\" (UniqueName: \"kubernetes.io/projected/f571ed90-9a31-47d2-8c4b-9bc57a9975b4-kube-api-access-dql4s\") pod \"e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-qq9qb\" (UID: \"f571ed90-9a31-47d2-8c4b-9bc57a9975b4\") " pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-qq9qb" Oct 30 16:57:45.452572 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:45.452076 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f571ed90-9a31-47d2-8c4b-9bc57a9975b4-workload-certs\") pod \"e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-qq9qb\" (UID: \"f571ed90-9a31-47d2-8c4b-9bc57a9975b4\") " pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-qq9qb" Oct 30 16:57:45.452572 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:45.452093 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f571ed90-9a31-47d2-8c4b-9bc57a9975b4-istio-podinfo\") pod \"e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-qq9qb\" (UID: \"f571ed90-9a31-47d2-8c4b-9bc57a9975b4\") " pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-qq9qb" Oct 30 16:57:45.452572 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:45.452121 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f571ed90-9a31-47d2-8c4b-9bc57a9975b4-istiod-ca-cert\") pod \"e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-qq9qb\" (UID: \"f571ed90-9a31-47d2-8c4b-9bc57a9975b4\") " pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-qq9qb" Oct 30 16:57:45.452572 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:45.452278 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f571ed90-9a31-47d2-8c4b-9bc57a9975b4-istio-data\") pod \"e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-qq9qb\" (UID: \"f571ed90-9a31-47d2-8c4b-9bc57a9975b4\") " pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-qq9qb" Oct 30 16:57:45.452572 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:45.452381 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f571ed90-9a31-47d2-8c4b-9bc57a9975b4-workload-socket\") pod \"e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-qq9qb\" (UID: \"f571ed90-9a31-47d2-8c4b-9bc57a9975b4\") " pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-qq9qb" Oct 30 16:57:45.452572 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:45.452462 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f571ed90-9a31-47d2-8c4b-9bc57a9975b4-istio-token\") pod \"e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-qq9qb\" (UID: \"f571ed90-9a31-47d2-8c4b-9bc57a9975b4\") " pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-qq9qb" Oct 30 16:57:45.452572 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:45.452550 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f571ed90-9a31-47d2-8c4b-9bc57a9975b4-credential-socket\") pod \"e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-qq9qb\" (UID: \"f571ed90-9a31-47d2-8c4b-9bc57a9975b4\") " pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-qq9qb" Oct 30 16:57:45.452572 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:45.452549 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f571ed90-9a31-47d2-8c4b-9bc57a9975b4-workload-certs\") pod \"e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-qq9qb\" (UID: \"f571ed90-9a31-47d2-8c4b-9bc57a9975b4\") " pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-qq9qb" Oct 30 16:57:45.453055 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:45.452674 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f571ed90-9a31-47d2-8c4b-9bc57a9975b4-istio-data\") pod \"e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-qq9qb\" (UID: \"f571ed90-9a31-47d2-8c4b-9bc57a9975b4\") " pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-qq9qb" Oct 30 16:57:45.453055 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:45.452763 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f571ed90-9a31-47d2-8c4b-9bc57a9975b4-workload-socket\") pod \"e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-qq9qb\" (UID: \"f571ed90-9a31-47d2-8c4b-9bc57a9975b4\") " pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-qq9qb" Oct 30 16:57:45.453055 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:45.452883 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f571ed90-9a31-47d2-8c4b-9bc57a9975b4-istiod-ca-cert\") pod \"e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-qq9qb\" (UID: \"f571ed90-9a31-47d2-8c4b-9bc57a9975b4\") " pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-qq9qb" Oct 30 16:57:45.453055 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:45.453008 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f571ed90-9a31-47d2-8c4b-9bc57a9975b4-credential-socket\") pod \"e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-qq9qb\" (UID: \"f571ed90-9a31-47d2-8c4b-9bc57a9975b4\") " pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-qq9qb" Oct 30 16:57:45.454599 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:45.454565 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f571ed90-9a31-47d2-8c4b-9bc57a9975b4-istio-envoy\") pod \"e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-qq9qb\" (UID: \"f571ed90-9a31-47d2-8c4b-9bc57a9975b4\") " pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-qq9qb" Oct 30 16:57:45.454754 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:45.454733 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f571ed90-9a31-47d2-8c4b-9bc57a9975b4-istio-podinfo\") pod \"e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-qq9qb\" (UID: \"f571ed90-9a31-47d2-8c4b-9bc57a9975b4\") " pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-qq9qb" Oct 30 16:57:45.476594 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:45.476512 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f571ed90-9a31-47d2-8c4b-9bc57a9975b4-istio-token\") pod \"e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-qq9qb\" (UID: \"f571ed90-9a31-47d2-8c4b-9bc57a9975b4\") " pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-qq9qb" Oct 30 16:57:45.476714 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:45.476625 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dql4s\" (UniqueName: \"kubernetes.io/projected/f571ed90-9a31-47d2-8c4b-9bc57a9975b4-kube-api-access-dql4s\") pod \"e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-qq9qb\" (UID: \"f571ed90-9a31-47d2-8c4b-9bc57a9975b4\") " pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-qq9qb" Oct 30 16:57:45.549177 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:45.549137 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-qq9qb" Oct 30 16:57:45.886293 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:57:45.886262 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf571ed90_9a31_47d2_8c4b_9bc57a9975b4.slice/crio-b79e8f86366e86afc44f9006fbdbc47d81debcc7c643a90905be645ad1adf857 WatchSource:0}: Error finding container b79e8f86366e86afc44f9006fbdbc47d81debcc7c643a90905be645ad1adf857: Status 404 returned error can't find the container with id b79e8f86366e86afc44f9006fbdbc47d81debcc7c643a90905be645ad1adf857 Oct 30 16:57:45.886633 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:45.886607 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-qq9qb"] Oct 30 16:57:45.889157 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:45.889126 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:57:45.889241 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:45.889191 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:57:45.889241 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:45.889221 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:57:46.440507 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:46.440468 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-qq9qb" event={"ID":"f571ed90-9a31-47d2-8c4b-9bc57a9975b4","Type":"ContainerStarted","Data":"a5b140946fb5276862c19f77e56a6bbd4bb9251ce4893f5c6cea9e11d34476b3"} Oct 30 16:57:46.440710 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:46.440523 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-qq9qb" event={"ID":"f571ed90-9a31-47d2-8c4b-9bc57a9975b4","Type":"ContainerStarted","Data":"b79e8f86366e86afc44f9006fbdbc47d81debcc7c643a90905be645ad1adf857"} Oct 30 16:57:46.517273 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:46.517168 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-qq9qb" podStartSLOduration=1.517153368 podStartE2EDuration="1.517153368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:57:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:57:46.501780454 +0000 UTC m=+2222.029115683" watchObservedRunningTime="2026-10-30 16:57:46.517153368 +0000 UTC m=+2222.044488587" Oct 30 16:57:46.553370 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:46.553329 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-qq9qb" Oct 30 16:57:46.555565 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:46.555535 5052 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-qq9qb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body= Oct 30 16:57:46.555736 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:46.555658 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-qq9qb" podUID="f571ed90-9a31-47d2-8c4b-9bc57a9975b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" Oct 30 16:57:47.553196 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:47.553155 5052 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-qq9qb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body= Oct 30 16:57:47.553729 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:47.553219 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-qq9qb" podUID="f571ed90-9a31-47d2-8c4b-9bc57a9975b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" Oct 30 16:57:48.553003 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:48.552964 5052 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-qq9qb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body= Oct 30 16:57:48.553217 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:48.553026 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-qq9qb" podUID="f571ed90-9a31-47d2-8c4b-9bc57a9975b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" Oct 30 16:57:48.863865 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:57:48.863775 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 16:57:49.553040 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:49.552998 5052 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-qq9qb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body= Oct 30 16:57:49.553225 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:49.553068 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-qq9qb" podUID="f571ed90-9a31-47d2-8c4b-9bc57a9975b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" Oct 30 16:57:49.756772 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:49.756738 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-480a60-openshift-default-fb894cdbc-4zzvx"] Oct 30 16:57:49.757144 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:49.757120 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-480a60-openshift-default-fb894cdbc-4zzvx" podUID="957040df-7379-406f-8398-573f7ea3bde6" containerName="istio-proxy" containerID="cri-o://05766a0cccc395d4f43886ad8d0fc6a646396ea11c0e582683be77aa7237fcb1" gracePeriod=30 Oct 30 16:57:50.554186 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:50.554139 5052 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-qq9qb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body= Oct 30 16:57:50.554833 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:50.554802 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-qq9qb" podUID="f571ed90-9a31-47d2-8c4b-9bc57a9975b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" Oct 30 16:57:51.553017 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:51.552974 5052 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-qq9qb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body= Oct 30 16:57:51.553224 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:51.553043 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-qq9qb" podUID="f571ed90-9a31-47d2-8c4b-9bc57a9975b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" Oct 30 16:57:52.550287 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:52.550247 5052 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-qq9qb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body= Oct 30 16:57:52.550711 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:52.550304 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-qq9qb" podUID="f571ed90-9a31-47d2-8c4b-9bc57a9975b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" Oct 30 16:57:53.555787 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:53.555757 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-qq9qb" Oct 30 16:57:53.556250 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:53.556236 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-qq9qb" Oct 30 16:57:53.557062 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:53.557033 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-qq9qb" Oct 30 16:57:55.087206 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:55.087178 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-480a60-openshift-default-fb894cdbc-4zzvx" Oct 30 16:57:55.178073 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:55.178013 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/957040df-7379-406f-8398-573f7ea3bde6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/957040df-7379-406f-8398-573f7ea3bde6-workload-socket\") pod \"957040df-7379-406f-8398-573f7ea3bde6\" (UID: \"957040df-7379-406f-8398-573f7ea3bde6\") " Oct 30 16:57:55.178287 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:55.178267 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/957040df-7379-406f-8398-573f7ea3bde6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/957040df-7379-406f-8398-573f7ea3bde6-istio-podinfo\") pod \"957040df-7379-406f-8398-573f7ea3bde6\" (UID: \"957040df-7379-406f-8398-573f7ea3bde6\") " Oct 30 16:57:55.178566 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:55.178337 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/957040df-7379-406f-8398-573f7ea3bde6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/957040df-7379-406f-8398-573f7ea3bde6-credential-socket\") pod \"957040df-7379-406f-8398-573f7ea3bde6\" (UID: \"957040df-7379-406f-8398-573f7ea3bde6\") " Oct 30 16:57:55.178566 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:55.178561 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/957040df-7379-406f-8398-573f7ea3bde6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/957040df-7379-406f-8398-573f7ea3bde6-istio-envoy\") pod \"957040df-7379-406f-8398-573f7ea3bde6\" (UID: \"957040df-7379-406f-8398-573f7ea3bde6\") " Oct 30 16:57:55.178744 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:55.178636 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/957040df-7379-406f-8398-573f7ea3bde6-istio-token\" (UniqueName: \"kubernetes.io/projected/957040df-7379-406f-8398-573f7ea3bde6-istio-token\") pod \"957040df-7379-406f-8398-573f7ea3bde6\" (UID: \"957040df-7379-406f-8398-573f7ea3bde6\") " Oct 30 16:57:55.178744 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:55.178671 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/957040df-7379-406f-8398-573f7ea3bde6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/957040df-7379-406f-8398-573f7ea3bde6-istio-data\") pod \"957040df-7379-406f-8398-573f7ea3bde6\" (UID: \"957040df-7379-406f-8398-573f7ea3bde6\") " Oct 30 16:57:55.178744 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:55.178703 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/957040df-7379-406f-8398-573f7ea3bde6-kube-api-access-gfhq6\" (UniqueName: \"kubernetes.io/projected/957040df-7379-406f-8398-573f7ea3bde6-kube-api-access-gfhq6\") pod \"957040df-7379-406f-8398-573f7ea3bde6\" (UID: \"957040df-7379-406f-8398-573f7ea3bde6\") " Oct 30 16:57:55.178910 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:55.178743 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/957040df-7379-406f-8398-573f7ea3bde6-workload-socket" pod "957040df-7379-406f-8398-573f7ea3bde6" (UID: "957040df-7379-406f-8398-573f7ea3bde6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:57:55.178910 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:55.178755 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/957040df-7379-406f-8398-573f7ea3bde6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/957040df-7379-406f-8398-573f7ea3bde6-workload-certs\") pod \"957040df-7379-406f-8398-573f7ea3bde6\" (UID: \"957040df-7379-406f-8398-573f7ea3bde6\") " Oct 30 16:57:55.178910 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:55.178832 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/957040df-7379-406f-8398-573f7ea3bde6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/957040df-7379-406f-8398-573f7ea3bde6-istiod-ca-cert\") pod \"957040df-7379-406f-8398-573f7ea3bde6\" (UID: \"957040df-7379-406f-8398-573f7ea3bde6\") " Oct 30 16:57:55.179314 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:55.179292 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/957040df-7379-406f-8398-573f7ea3bde6-workload-certs" pod "957040df-7379-406f-8398-573f7ea3bde6" (UID: "957040df-7379-406f-8398-573f7ea3bde6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:57:55.179570 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:55.179549 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/957040df-7379-406f-8398-573f7ea3bde6-istio-data" pod "957040df-7379-406f-8398-573f7ea3bde6" (UID: "957040df-7379-406f-8398-573f7ea3bde6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:57:55.179872 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:55.179843 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/957040df-7379-406f-8398-573f7ea3bde6-istiod-ca-cert" pod "957040df-7379-406f-8398-573f7ea3bde6" (UID: "957040df-7379-406f-8398-573f7ea3bde6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:57:55.180047 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:55.179951 5052 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/957040df-7379-406f-8398-573f7ea3bde6-istio-data\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:57:55.180047 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:55.179975 5052 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/957040df-7379-406f-8398-573f7ea3bde6-workload-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:57:55.180047 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:55.179987 5052 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/957040df-7379-406f-8398-573f7ea3bde6-workload-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:57:55.180451 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:55.180432 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/957040df-7379-406f-8398-573f7ea3bde6-credential-socket" pod "957040df-7379-406f-8398-573f7ea3bde6" (UID: "957040df-7379-406f-8398-573f7ea3bde6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:57:55.183563 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:55.183538 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/957040df-7379-406f-8398-573f7ea3bde6-istio-podinfo" pod "957040df-7379-406f-8398-573f7ea3bde6" (UID: "957040df-7379-406f-8398-573f7ea3bde6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:57:55.187182 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:55.187142 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/957040df-7379-406f-8398-573f7ea3bde6-istio-envoy" pod "957040df-7379-406f-8398-573f7ea3bde6" (UID: "957040df-7379-406f-8398-573f7ea3bde6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:57:55.187324 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:55.187234 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/957040df-7379-406f-8398-573f7ea3bde6-istio-token" pod "957040df-7379-406f-8398-573f7ea3bde6" (UID: "957040df-7379-406f-8398-573f7ea3bde6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:57:55.188974 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:55.188949 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/957040df-7379-406f-8398-573f7ea3bde6-kube-api-access-gfhq6" pod "957040df-7379-406f-8398-573f7ea3bde6" (UID: "957040df-7379-406f-8398-573f7ea3bde6"). InnerVolumeSpecName "kube-api-access-gfhq6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:57:55.281670 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:55.281573 5052 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/957040df-7379-406f-8398-573f7ea3bde6-istio-podinfo\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:57:55.281670 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:55.281623 5052 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/957040df-7379-406f-8398-573f7ea3bde6-credential-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:57:55.281670 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:55.281640 5052 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/957040df-7379-406f-8398-573f7ea3bde6-istio-envoy\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:57:55.281670 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:55.281653 5052 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/957040df-7379-406f-8398-573f7ea3bde6-istio-token\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:57:55.281670 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:55.281666 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfhq6\" (UniqueName: \"kubernetes.io/projected/957040df-7379-406f-8398-573f7ea3bde6-kube-api-access-gfhq6\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:57:55.281946 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:55.281676 5052 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/957040df-7379-406f-8398-573f7ea3bde6-istiod-ca-cert\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:57:55.525167 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:55.525131 5052 generic.go:356] "Generic (PLEG): container finished" podID="957040df-7379-406f-8398-573f7ea3bde6" containerID="05766a0cccc395d4f43886ad8d0fc6a646396ea11c0e582683be77aa7237fcb1" exitCode=0 Oct 30 16:57:55.525366 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:55.525205 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-480a60-openshift-default-fb894cdbc-4zzvx" Oct 30 16:57:55.525366 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:55.525221 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-480a60-openshift-default-fb894cdbc-4zzvx" event={"ID":"957040df-7379-406f-8398-573f7ea3bde6","Type":"ContainerDied","Data":"05766a0cccc395d4f43886ad8d0fc6a646396ea11c0e582683be77aa7237fcb1"} Oct 30 16:57:55.525366 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:55.525290 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-480a60-openshift-default-fb894cdbc-4zzvx" event={"ID":"957040df-7379-406f-8398-573f7ea3bde6","Type":"ContainerDied","Data":"b63be741b6dfc493e84f4ea4a7006d579ce9962da82d374172149723f89690ae"} Oct 30 16:57:55.525366 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:55.525322 5052 scope.go:122] "RemoveContainer" containerID="05766a0cccc395d4f43886ad8d0fc6a646396ea11c0e582683be77aa7237fcb1" Oct 30 16:57:55.539116 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:55.539074 5052 scope.go:122] "RemoveContainer" containerID="05766a0cccc395d4f43886ad8d0fc6a646396ea11c0e582683be77aa7237fcb1" Oct 30 16:57:55.539516 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:57:55.539485 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05766a0cccc395d4f43886ad8d0fc6a646396ea11c0e582683be77aa7237fcb1\": container with ID starting with 05766a0cccc395d4f43886ad8d0fc6a646396ea11c0e582683be77aa7237fcb1 not found: ID does not exist" containerID="05766a0cccc395d4f43886ad8d0fc6a646396ea11c0e582683be77aa7237fcb1" Oct 30 16:57:55.539620 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:55.539526 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05766a0cccc395d4f43886ad8d0fc6a646396ea11c0e582683be77aa7237fcb1"} err="failed to get container status \"05766a0cccc395d4f43886ad8d0fc6a646396ea11c0e582683be77aa7237fcb1\": rpc error: code = NotFound desc = could not find container \"05766a0cccc395d4f43886ad8d0fc6a646396ea11c0e582683be77aa7237fcb1\": container with ID starting with 05766a0cccc395d4f43886ad8d0fc6a646396ea11c0e582683be77aa7237fcb1 not found: ID does not exist" Oct 30 16:57:55.584091 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:55.584061 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-480a60-openshift-default-fb894cdbc-4zzvx"] Oct 30 16:57:55.597235 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:55.597203 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-480a60-openshift-default-fb894cdbc-4zzvx"] Oct 30 16:57:56.601387 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:57:56.601348 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="957040df-7379-406f-8398-573f7ea3bde6" path="/var/lib/kubelet/pods/957040df-7379-406f-8398-573f7ea3bde6/volumes" Oct 30 16:58:12.751039 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:58:12.751002 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cdwcj"] Oct 30 16:58:18.842200 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:58:18.842162 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 16:58:48.844659 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:58:48.844605 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 16:58:55.858889 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:58:55.858849 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-h5v86"] Oct 30 16:58:55.860160 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:58:55.860127 5052 cpu_manager.go:414] "containerMap: removing container" podUID="957040df-7379-406f-8398-573f7ea3bde6" containerName="istio-proxy" Oct 30 16:58:55.860160 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:58:55.860155 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="957040df-7379-406f-8398-573f7ea3bde6" containerName="istio-proxy" Oct 30 16:58:55.860491 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:58:55.860474 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="957040df-7379-406f-8398-573f7ea3bde6" containerName="istio-proxy" Oct 30 16:58:55.867604 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:58:55.867567 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-h5v86" Oct 30 16:58:55.896892 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:58:55.896851 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-h5v86"] Oct 30 16:58:55.927283 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:58:55.927245 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-9e533f-openshift-default-dockercfg-x62c6\"" Oct 30 16:58:55.946158 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:58:55.946116 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/029cd866-9c51-45c8-9ca4-4c16578e12c9-istio-envoy\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-h5v86\" (UID: \"029cd866-9c51-45c8-9ca4-4c16578e12c9\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-h5v86" Oct 30 16:58:55.946460 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:58:55.946438 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/029cd866-9c51-45c8-9ca4-4c16578e12c9-credential-socket\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-h5v86\" (UID: \"029cd866-9c51-45c8-9ca4-4c16578e12c9\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-h5v86" Oct 30 16:58:55.946742 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:58:55.946719 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/029cd866-9c51-45c8-9ca4-4c16578e12c9-istio-token\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-h5v86\" (UID: \"029cd866-9c51-45c8-9ca4-4c16578e12c9\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-h5v86" Oct 30 16:58:55.946894 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:58:55.946876 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/029cd866-9c51-45c8-9ca4-4c16578e12c9-workload-certs\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-h5v86\" (UID: \"029cd866-9c51-45c8-9ca4-4c16578e12c9\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-h5v86" Oct 30 16:58:55.947062 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:58:55.947048 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/029cd866-9c51-45c8-9ca4-4c16578e12c9-istio-podinfo\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-h5v86\" (UID: \"029cd866-9c51-45c8-9ca4-4c16578e12c9\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-h5v86" Oct 30 16:58:55.947160 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:58:55.947148 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/029cd866-9c51-45c8-9ca4-4c16578e12c9-istiod-ca-cert\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-h5v86\" (UID: \"029cd866-9c51-45c8-9ca4-4c16578e12c9\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-h5v86" Oct 30 16:58:55.947284 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:58:55.947267 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56bhv\" (UniqueName: \"kubernetes.io/projected/029cd866-9c51-45c8-9ca4-4c16578e12c9-kube-api-access-56bhv\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-h5v86\" (UID: \"029cd866-9c51-45c8-9ca4-4c16578e12c9\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-h5v86" Oct 30 16:58:55.947529 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:58:55.947514 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/029cd866-9c51-45c8-9ca4-4c16578e12c9-istio-data\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-h5v86\" (UID: \"029cd866-9c51-45c8-9ca4-4c16578e12c9\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-h5v86" Oct 30 16:58:55.947664 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:58:55.947647 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/029cd866-9c51-45c8-9ca4-4c16578e12c9-workload-socket\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-h5v86\" (UID: \"029cd866-9c51-45c8-9ca4-4c16578e12c9\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-h5v86" Oct 30 16:58:56.095610 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:58:56.058931 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/029cd866-9c51-45c8-9ca4-4c16578e12c9-istio-token\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-h5v86\" (UID: \"029cd866-9c51-45c8-9ca4-4c16578e12c9\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-h5v86" Oct 30 16:58:56.096846 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:58:56.096812 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/029cd866-9c51-45c8-9ca4-4c16578e12c9-workload-certs\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-h5v86\" (UID: \"029cd866-9c51-45c8-9ca4-4c16578e12c9\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-h5v86" Oct 30 16:58:56.097141 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:58:56.097121 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/029cd866-9c51-45c8-9ca4-4c16578e12c9-istio-podinfo\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-h5v86\" (UID: \"029cd866-9c51-45c8-9ca4-4c16578e12c9\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-h5v86" Oct 30 16:58:56.097271 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:58:56.097254 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/029cd866-9c51-45c8-9ca4-4c16578e12c9-istiod-ca-cert\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-h5v86\" (UID: \"029cd866-9c51-45c8-9ca4-4c16578e12c9\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-h5v86" Oct 30 16:58:56.097420 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:58:56.097401 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56bhv\" (UniqueName: \"kubernetes.io/projected/029cd866-9c51-45c8-9ca4-4c16578e12c9-kube-api-access-56bhv\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-h5v86\" (UID: \"029cd866-9c51-45c8-9ca4-4c16578e12c9\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-h5v86" Oct 30 16:58:56.097747 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:58:56.097725 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/029cd866-9c51-45c8-9ca4-4c16578e12c9-istio-data\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-h5v86\" (UID: \"029cd866-9c51-45c8-9ca4-4c16578e12c9\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-h5v86" Oct 30 16:58:56.097919 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:58:56.097901 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/029cd866-9c51-45c8-9ca4-4c16578e12c9-workload-socket\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-h5v86\" (UID: \"029cd866-9c51-45c8-9ca4-4c16578e12c9\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-h5v86" Oct 30 16:58:56.098106 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:58:56.098090 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/029cd866-9c51-45c8-9ca4-4c16578e12c9-istio-envoy\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-h5v86\" (UID: \"029cd866-9c51-45c8-9ca4-4c16578e12c9\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-h5v86" Oct 30 16:58:56.111880 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:58:56.111802 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/029cd866-9c51-45c8-9ca4-4c16578e12c9-credential-socket\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-h5v86\" (UID: \"029cd866-9c51-45c8-9ca4-4c16578e12c9\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-h5v86" Oct 30 16:58:56.133426 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:58:56.112557 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/029cd866-9c51-45c8-9ca4-4c16578e12c9-credential-socket\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-h5v86\" (UID: \"029cd866-9c51-45c8-9ca4-4c16578e12c9\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-h5v86" Oct 30 16:58:56.133696 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:58:56.123759 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/029cd866-9c51-45c8-9ca4-4c16578e12c9-workload-certs\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-h5v86\" (UID: \"029cd866-9c51-45c8-9ca4-4c16578e12c9\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-h5v86" Oct 30 16:58:56.133834 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:58:56.127904 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/029cd866-9c51-45c8-9ca4-4c16578e12c9-istio-data\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-h5v86\" (UID: \"029cd866-9c51-45c8-9ca4-4c16578e12c9\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-h5v86" Oct 30 16:58:56.133925 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:58:56.128135 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/029cd866-9c51-45c8-9ca4-4c16578e12c9-workload-socket\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-h5v86\" (UID: \"029cd866-9c51-45c8-9ca4-4c16578e12c9\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-h5v86" Oct 30 16:58:56.134722 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:58:56.134694 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/029cd866-9c51-45c8-9ca4-4c16578e12c9-istiod-ca-cert\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-h5v86\" (UID: \"029cd866-9c51-45c8-9ca4-4c16578e12c9\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-h5v86" Oct 30 16:58:56.221601 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:58:56.211151 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56bhv\" (UniqueName: \"kubernetes.io/projected/029cd866-9c51-45c8-9ca4-4c16578e12c9-kube-api-access-56bhv\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-h5v86\" (UID: \"029cd866-9c51-45c8-9ca4-4c16578e12c9\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-h5v86" Oct 30 16:58:56.221601 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:58:56.214771 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/029cd866-9c51-45c8-9ca4-4c16578e12c9-istio-envoy\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-h5v86\" (UID: \"029cd866-9c51-45c8-9ca4-4c16578e12c9\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-h5v86" Oct 30 16:58:56.244605 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:58:56.237397 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/029cd866-9c51-45c8-9ca4-4c16578e12c9-istio-podinfo\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-h5v86\" (UID: \"029cd866-9c51-45c8-9ca4-4c16578e12c9\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-h5v86" Oct 30 16:58:56.247451 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:58:56.247334 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/029cd866-9c51-45c8-9ca4-4c16578e12c9-istio-token\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-h5v86\" (UID: \"029cd866-9c51-45c8-9ca4-4c16578e12c9\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-h5v86" Oct 30 16:58:56.490049 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:58:56.489993 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-h5v86" Oct 30 16:58:56.873335 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:58:56.873302 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-h5v86"] Oct 30 16:58:56.873741 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:58:56.873718 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod029cd866_9c51_45c8_9ca4_4c16578e12c9.slice/crio-4bb7056c10742d4e8e78844d1a85d29ec079404f04bccc5ee33ce7fbd0f429d0 WatchSource:0}: Error finding container 4bb7056c10742d4e8e78844d1a85d29ec079404f04bccc5ee33ce7fbd0f429d0: Status 404 returned error can't find the container with id 4bb7056c10742d4e8e78844d1a85d29ec079404f04bccc5ee33ce7fbd0f429d0 Oct 30 16:58:56.876482 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:58:56.876453 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:58:56.876544 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:58:56.876517 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:58:56.876595 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:58:56.876549 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:58:57.279549 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:58:57.279512 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-h5v86" event={"ID":"029cd866-9c51-45c8-9ca4-4c16578e12c9","Type":"ContainerStarted","Data":"e3c6f7aeea864977164007842e03e5ab7d8c32ea1a7105943fd2a500d7d7fafd"} Oct 30 16:58:57.279763 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:58:57.279565 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-h5v86" event={"ID":"029cd866-9c51-45c8-9ca4-4c16578e12c9","Type":"ContainerStarted","Data":"4bb7056c10742d4e8e78844d1a85d29ec079404f04bccc5ee33ce7fbd0f429d0"} Oct 30 16:58:57.333732 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:58:57.333675 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-h5v86" podStartSLOduration=2.333657468 podStartE2EDuration="2.333657468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:58:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:58:57.330055625 +0000 UTC m=+2292.857390845" watchObservedRunningTime="2026-10-30 16:58:57.333657468 +0000 UTC m=+2292.860992687" Oct 30 16:58:57.491603 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:58:57.491234 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-h5v86" Oct 30 16:58:57.529764 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:58:57.526925 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-h5v86" Oct 30 16:58:58.288259 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:58:58.288230 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-h5v86" Oct 30 16:58:58.289119 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:58:58.289098 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-h5v86" Oct 30 16:58:59.595027 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:58:59.594983 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-a59502/test-model-a59502-kserve-798b8c7859-5kptw"] Oct 30 16:59:02.280712 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:02.280676 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9e533f-85c9dd5979-nrmbf"] Oct 30 16:59:02.285822 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:02.285805 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9e533f-85c9dd5979-nrmbf" Oct 30 16:59:02.307263 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:02.307237 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9e533f-85c9dd5979-nrmbf"] Oct 30 16:59:02.314852 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:02.314826 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-9e533f\"" Oct 30 16:59:02.315014 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:02.314897 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-9e533f-dockercfg-mk68d\"" Oct 30 16:59:02.336893 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:02.336856 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9rhc\" (UniqueName: \"kubernetes.io/projected/8054915f-a3d1-44d0-ab47-4ca0161255da-kube-api-access-h9rhc\") pod \"payload-pre-processing-e2e-shared-b-w0-9e533f-85c9dd5979-nrmbf\" (UID: \"8054915f-a3d1-44d0-ab47-4ca0161255da\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9e533f-85c9dd5979-nrmbf" Oct 30 16:59:02.371211 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:02.353998 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8054915f-a3d1-44d0-ab47-4ca0161255da-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-9e533f-85c9dd5979-nrmbf\" (UID: \"8054915f-a3d1-44d0-ab47-4ca0161255da\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9e533f-85c9dd5979-nrmbf" Oct 30 16:59:02.638727 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:02.627618 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-9e533f-57fd8bd864-t6vr2"] Oct 30 16:59:02.652658 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:02.652088 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9rhc\" (UniqueName: \"kubernetes.io/projected/8054915f-a3d1-44d0-ab47-4ca0161255da-kube-api-access-h9rhc\") pod \"payload-pre-processing-e2e-shared-b-w0-9e533f-85c9dd5979-nrmbf\" (UID: \"8054915f-a3d1-44d0-ab47-4ca0161255da\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9e533f-85c9dd5979-nrmbf" Oct 30 16:59:02.653016 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:02.652990 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8054915f-a3d1-44d0-ab47-4ca0161255da-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-9e533f-85c9dd5979-nrmbf\" (UID: \"8054915f-a3d1-44d0-ab47-4ca0161255da\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9e533f-85c9dd5979-nrmbf" Oct 30 16:59:02.663659 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:02.662485 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8054915f-a3d1-44d0-ab47-4ca0161255da-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-9e533f-85c9dd5979-nrmbf\" (UID: \"8054915f-a3d1-44d0-ab47-4ca0161255da\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9e533f-85c9dd5979-nrmbf" Oct 30 16:59:02.668022 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:02.667985 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-9e533f-57fd8bd864-t6vr2" Oct 30 16:59:02.702631 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:02.696619 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9rhc\" (UniqueName: \"kubernetes.io/projected/8054915f-a3d1-44d0-ab47-4ca0161255da-kube-api-access-h9rhc\") pod \"payload-pre-processing-e2e-shared-b-w0-9e533f-85c9dd5979-nrmbf\" (UID: \"8054915f-a3d1-44d0-ab47-4ca0161255da\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9e533f-85c9dd5979-nrmbf" Oct 30 16:59:02.702631 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:02.697889 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-9e533f-57fd8bd864-t6vr2"] Oct 30 16:59:02.762117 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:02.762083 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4963d7c3-cee3-430c-8e1c-156c79a75a3d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-9e533f-57fd8bd864-t6vr2\" (UID: \"4963d7c3-cee3-430c-8e1c-156c79a75a3d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-9e533f-57fd8bd864-t6vr2" Oct 30 16:59:02.762399 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:02.762370 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gg4gh\" (UniqueName: \"kubernetes.io/projected/4963d7c3-cee3-430c-8e1c-156c79a75a3d-kube-api-access-gg4gh\") pod \"payload-processing-e2e-shared-b-w0-9e533f-57fd8bd864-t6vr2\" (UID: \"4963d7c3-cee3-430c-8e1c-156c79a75a3d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-9e533f-57fd8bd864-t6vr2" Oct 30 16:59:02.864572 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:02.864535 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4963d7c3-cee3-430c-8e1c-156c79a75a3d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-9e533f-57fd8bd864-t6vr2\" (UID: \"4963d7c3-cee3-430c-8e1c-156c79a75a3d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-9e533f-57fd8bd864-t6vr2" Oct 30 16:59:02.864772 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:02.864712 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gg4gh\" (UniqueName: \"kubernetes.io/projected/4963d7c3-cee3-430c-8e1c-156c79a75a3d-kube-api-access-gg4gh\") pod \"payload-processing-e2e-shared-b-w0-9e533f-57fd8bd864-t6vr2\" (UID: \"4963d7c3-cee3-430c-8e1c-156c79a75a3d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-9e533f-57fd8bd864-t6vr2" Oct 30 16:59:02.865335 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:02.865315 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4963d7c3-cee3-430c-8e1c-156c79a75a3d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-9e533f-57fd8bd864-t6vr2\" (UID: \"4963d7c3-cee3-430c-8e1c-156c79a75a3d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-9e533f-57fd8bd864-t6vr2" Oct 30 16:59:02.894090 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:02.894022 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gg4gh\" (UniqueName: \"kubernetes.io/projected/4963d7c3-cee3-430c-8e1c-156c79a75a3d-kube-api-access-gg4gh\") pod \"payload-processing-e2e-shared-b-w0-9e533f-57fd8bd864-t6vr2\" (UID: \"4963d7c3-cee3-430c-8e1c-156c79a75a3d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-9e533f-57fd8bd864-t6vr2" Oct 30 16:59:02.901934 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:02.901909 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9e533f-85c9dd5979-nrmbf" Oct 30 16:59:03.013777 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:03.013738 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-9e533f-57fd8bd864-t6vr2" Oct 30 16:59:03.224749 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:03.224719 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9e533f-85c9dd5979-nrmbf"] Oct 30 16:59:03.226165 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:59:03.226123 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8054915f_a3d1_44d0_ab47_4ca0161255da.slice/crio-bc86857811cdcc3c0fb6aff9dd8e1b00f24b4c33fc1c723cd530def86c39f283 WatchSource:0}: Error finding container bc86857811cdcc3c0fb6aff9dd8e1b00f24b4c33fc1c723cd530def86c39f283: Status 404 returned error can't find the container with id bc86857811cdcc3c0fb6aff9dd8e1b00f24b4c33fc1c723cd530def86c39f283 Oct 30 16:59:03.329562 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:03.329524 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9e533f-85c9dd5979-nrmbf" event={"ID":"8054915f-a3d1-44d0-ab47-4ca0161255da","Type":"ContainerStarted","Data":"9c88eb46a20297d4d538b40dbd7a013fa2d48b920ea4aa9562ce897a8daa31c4"} Oct 30 16:59:03.329991 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:03.329606 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9e533f-85c9dd5979-nrmbf" event={"ID":"8054915f-a3d1-44d0-ab47-4ca0161255da","Type":"ContainerStarted","Data":"bc86857811cdcc3c0fb6aff9dd8e1b00f24b4c33fc1c723cd530def86c39f283"} Oct 30 16:59:03.329991 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:03.329640 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9e533f-85c9dd5979-nrmbf" Oct 30 16:59:03.349808 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:03.349786 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-9e533f-57fd8bd864-t6vr2"] Oct 30 16:59:03.350405 ip-10-0-131-136 kubenswrapper[5052]: W1030 16:59:03.350381 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4963d7c3_cee3_430c_8e1c_156c79a75a3d.slice/crio-873dcca7d92d2269123cdcfd3c37fa81438dcff6a0936513b4a924f70e9b0b65 WatchSource:0}: Error finding container 873dcca7d92d2269123cdcfd3c37fa81438dcff6a0936513b4a924f70e9b0b65: Status 404 returned error can't find the container with id 873dcca7d92d2269123cdcfd3c37fa81438dcff6a0936513b4a924f70e9b0b65 Oct 30 16:59:03.374242 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:03.374189 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9e533f-85c9dd5979-nrmbf" podStartSLOduration=1.374172441 podStartE2EDuration="1.374172441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:59:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:59:03.372594772 +0000 UTC m=+2298.899930010" watchObservedRunningTime="2026-10-30 16:59:03.374172441 +0000 UTC m=+2298.901507658" Oct 30 16:59:04.339763 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:04.339719 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-9e533f-57fd8bd864-t6vr2" event={"ID":"4963d7c3-cee3-430c-8e1c-156c79a75a3d","Type":"ContainerStarted","Data":"3e52bc09f7330f22593ced39b6dad44b40e6f6b9adb426971337f4627e150ffa"} Oct 30 16:59:04.340261 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:04.339782 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-9e533f-57fd8bd864-t6vr2" event={"ID":"4963d7c3-cee3-430c-8e1c-156c79a75a3d","Type":"ContainerStarted","Data":"873dcca7d92d2269123cdcfd3c37fa81438dcff6a0936513b4a924f70e9b0b65"} Oct 30 16:59:04.340261 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:04.340016 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-9e533f-57fd8bd864-t6vr2" Oct 30 16:59:04.390794 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:04.390749 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-9e533f-57fd8bd864-t6vr2" podStartSLOduration=2.390734725 podStartE2EDuration="2.390734725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:59:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:59:04.38739228 +0000 UTC m=+2299.914727511" watchObservedRunningTime="2026-10-30 16:59:04.390734725 +0000 UTC m=+2299.918069943" Oct 30 16:59:10.290605 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:10.289666 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/distinct-2-w4-263f00-kserve-8464897cf5-q8ck8" podUID="15ed43ef-5cdd-489a-a70e-063239822ff3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 16:59:14.022789 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:14.022750 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-9c1e26/test-model-9c1e26-kserve-7cc944cb5f-xjljc"] Oct 30 16:59:14.343294 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:14.343218 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9e533f-85c9dd5979-nrmbf" Oct 30 16:59:14.625420 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:14.625325 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-a59502/test-model-a59502-kserve-798b8c7859-5kptw" podUID="c2915da3-9830-4751-b58f-837063130fa1" containerName="main" containerID="cri-o://f371d71aca03863c6c26020f9da6660406f29ea1434a5941db18646238321ccd" gracePeriod=45 Oct 30 16:59:14.903333 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:14.902380 5052 generic.go:356] "Generic (PLEG): container finished" podID="c2915da3-9830-4751-b58f-837063130fa1" containerID="f371d71aca03863c6c26020f9da6660406f29ea1434a5941db18646238321ccd" exitCode=0 Oct 30 16:59:14.903333 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:14.902564 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a59502/test-model-a59502-kserve-798b8c7859-5kptw" event={"ID":"c2915da3-9830-4751-b58f-837063130fa1","Type":"ContainerDied","Data":"f371d71aca03863c6c26020f9da6660406f29ea1434a5941db18646238321ccd"} Oct 30 16:59:14.949275 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:14.949246 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a59502/test-model-a59502-kserve-798b8c7859-5kptw" Oct 30 16:59:15.112114 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:15.111470 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2915da3-9830-4751-b58f-837063130fa1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2915da3-9830-4751-b58f-837063130fa1-model-cache\") pod \"c2915da3-9830-4751-b58f-837063130fa1\" (UID: \"c2915da3-9830-4751-b58f-837063130fa1\") " Oct 30 16:59:15.112735 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:15.112149 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2915da3-9830-4751-b58f-837063130fa1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2915da3-9830-4751-b58f-837063130fa1-tmp-dir\") pod \"c2915da3-9830-4751-b58f-837063130fa1\" (UID: \"c2915da3-9830-4751-b58f-837063130fa1\") " Oct 30 16:59:15.112735 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:15.112189 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2915da3-9830-4751-b58f-837063130fa1-tls-certs\" (UniqueName: \"kubernetes.io/secret/c2915da3-9830-4751-b58f-837063130fa1-tls-certs\") pod \"c2915da3-9830-4751-b58f-837063130fa1\" (UID: \"c2915da3-9830-4751-b58f-837063130fa1\") " Oct 30 16:59:15.112891 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:15.112864 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2915da3-9830-4751-b58f-837063130fa1-kube-api-access-gsnfg\" (UniqueName: \"kubernetes.io/projected/c2915da3-9830-4751-b58f-837063130fa1-kube-api-access-gsnfg\") pod \"c2915da3-9830-4751-b58f-837063130fa1\" (UID: \"c2915da3-9830-4751-b58f-837063130fa1\") " Oct 30 16:59:15.113798 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:15.113765 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2915da3-9830-4751-b58f-837063130fa1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2915da3-9830-4751-b58f-837063130fa1-dshm\") pod \"c2915da3-9830-4751-b58f-837063130fa1\" (UID: \"c2915da3-9830-4751-b58f-837063130fa1\") " Oct 30 16:59:15.113924 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:15.113910 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2915da3-9830-4751-b58f-837063130fa1-home\" (UniqueName: \"kubernetes.io/empty-dir/c2915da3-9830-4751-b58f-837063130fa1-home\") pod \"c2915da3-9830-4751-b58f-837063130fa1\" (UID: \"c2915da3-9830-4751-b58f-837063130fa1\") " Oct 30 16:59:15.114221 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:15.114194 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2915da3-9830-4751-b58f-837063130fa1-model-cache" pod "c2915da3-9830-4751-b58f-837063130fa1" (UID: "c2915da3-9830-4751-b58f-837063130fa1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:59:15.114626 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:15.114570 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2915da3-9830-4751-b58f-837063130fa1-tmp-dir" pod "c2915da3-9830-4751-b58f-837063130fa1" (UID: "c2915da3-9830-4751-b58f-837063130fa1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:59:15.114872 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:15.114848 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2915da3-9830-4751-b58f-837063130fa1-home" pod "c2915da3-9830-4751-b58f-837063130fa1" (UID: "c2915da3-9830-4751-b58f-837063130fa1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:59:15.116516 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:15.116477 5052 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2915da3-9830-4751-b58f-837063130fa1-model-cache\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:59:15.116715 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:15.116690 5052 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2915da3-9830-4751-b58f-837063130fa1-tmp-dir\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:59:15.116897 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:15.116885 5052 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2915da3-9830-4751-b58f-837063130fa1-home\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:59:15.119417 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:15.118863 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2915da3-9830-4751-b58f-837063130fa1-tls-certs" pod "c2915da3-9830-4751-b58f-837063130fa1" (UID: "c2915da3-9830-4751-b58f-837063130fa1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:59:15.119417 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:15.118935 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2915da3-9830-4751-b58f-837063130fa1-dshm" pod "c2915da3-9830-4751-b58f-837063130fa1" (UID: "c2915da3-9830-4751-b58f-837063130fa1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:59:15.119799 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:15.119775 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2915da3-9830-4751-b58f-837063130fa1-kube-api-access-gsnfg" pod "c2915da3-9830-4751-b58f-837063130fa1" (UID: "c2915da3-9830-4751-b58f-837063130fa1"). InnerVolumeSpecName "kube-api-access-gsnfg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:59:15.218989 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:15.218919 5052 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2915da3-9830-4751-b58f-837063130fa1-tls-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:59:15.218989 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:15.218946 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsnfg\" (UniqueName: \"kubernetes.io/projected/c2915da3-9830-4751-b58f-837063130fa1-kube-api-access-gsnfg\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:59:15.218989 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:15.218956 5052 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2915da3-9830-4751-b58f-837063130fa1-dshm\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:59:15.350333 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:15.350306 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-9e533f-57fd8bd864-t6vr2" Oct 30 16:59:15.912813 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:15.912779 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a59502/test-model-a59502-kserve-798b8c7859-5kptw" event={"ID":"c2915da3-9830-4751-b58f-837063130fa1","Type":"ContainerDied","Data":"a6518d714ff909bc376b93e79bba17751e040fa195f45b4c0598b1b080a59284"} Oct 30 16:59:15.913013 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:15.912821 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a59502/test-model-a59502-kserve-798b8c7859-5kptw" Oct 30 16:59:15.913013 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:15.912839 5052 scope.go:122] "RemoveContainer" containerID="f371d71aca03863c6c26020f9da6660406f29ea1434a5941db18646238321ccd" Oct 30 16:59:15.976661 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:15.976621 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-a59502/test-model-a59502-kserve-798b8c7859-5kptw"] Oct 30 16:59:15.988698 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:15.988667 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-a59502/test-model-a59502-kserve-798b8c7859-5kptw"] Oct 30 16:59:16.600732 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:16.600697 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2915da3-9830-4751-b58f-837063130fa1" path="/var/lib/kubelet/pods/c2915da3-9830-4751-b58f-837063130fa1/volumes" Oct 30 16:59:18.841976 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:59:18.841940 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 16:59:29.048327 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:29.048259 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-9c1e26/test-model-9c1e26-kserve-7cc944cb5f-xjljc" podUID="17327d7e-8d87-4461-be7f-1ed4c1d17701" containerName="main" containerID="cri-o://3aea17113c5222245dffb11430a8b3d00ed45d767d3571ef83b44f9ce2933d4b" gracePeriod=45 Oct 30 16:59:29.344905 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:29.344877 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9c1e26/test-model-9c1e26-kserve-7cc944cb5f-xjljc" Oct 30 16:59:29.491446 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:29.491413 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17327d7e-8d87-4461-be7f-1ed4c1d17701-home\" (UniqueName: \"kubernetes.io/empty-dir/17327d7e-8d87-4461-be7f-1ed4c1d17701-home\") pod \"17327d7e-8d87-4461-be7f-1ed4c1d17701\" (UID: \"17327d7e-8d87-4461-be7f-1ed4c1d17701\") " Oct 30 16:59:29.491675 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:29.491648 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17327d7e-8d87-4461-be7f-1ed4c1d17701-kube-api-access-zzrk4\" (UniqueName: \"kubernetes.io/projected/17327d7e-8d87-4461-be7f-1ed4c1d17701-kube-api-access-zzrk4\") pod \"17327d7e-8d87-4461-be7f-1ed4c1d17701\" (UID: \"17327d7e-8d87-4461-be7f-1ed4c1d17701\") " Oct 30 16:59:29.491738 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:29.491669 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17327d7e-8d87-4461-be7f-1ed4c1d17701-home" pod "17327d7e-8d87-4461-be7f-1ed4c1d17701" (UID: "17327d7e-8d87-4461-be7f-1ed4c1d17701"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:59:29.491738 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:29.491681 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17327d7e-8d87-4461-be7f-1ed4c1d17701-tls-certs\" (UniqueName: \"kubernetes.io/secret/17327d7e-8d87-4461-be7f-1ed4c1d17701-tls-certs\") pod \"17327d7e-8d87-4461-be7f-1ed4c1d17701\" (UID: \"17327d7e-8d87-4461-be7f-1ed4c1d17701\") " Oct 30 16:59:29.491829 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:29.491815 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17327d7e-8d87-4461-be7f-1ed4c1d17701-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17327d7e-8d87-4461-be7f-1ed4c1d17701-tmp-dir\") pod \"17327d7e-8d87-4461-be7f-1ed4c1d17701\" (UID: \"17327d7e-8d87-4461-be7f-1ed4c1d17701\") " Oct 30 16:59:29.491865 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:29.491858 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17327d7e-8d87-4461-be7f-1ed4c1d17701-dshm\" (UniqueName: \"kubernetes.io/empty-dir/17327d7e-8d87-4461-be7f-1ed4c1d17701-dshm\") pod \"17327d7e-8d87-4461-be7f-1ed4c1d17701\" (UID: \"17327d7e-8d87-4461-be7f-1ed4c1d17701\") " Oct 30 16:59:29.491951 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:29.491934 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17327d7e-8d87-4461-be7f-1ed4c1d17701-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17327d7e-8d87-4461-be7f-1ed4c1d17701-model-cache\") pod \"17327d7e-8d87-4461-be7f-1ed4c1d17701\" (UID: \"17327d7e-8d87-4461-be7f-1ed4c1d17701\") " Oct 30 16:59:29.492139 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:29.492064 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17327d7e-8d87-4461-be7f-1ed4c1d17701-tmp-dir" pod "17327d7e-8d87-4461-be7f-1ed4c1d17701" (UID: "17327d7e-8d87-4461-be7f-1ed4c1d17701"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:59:29.492271 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:29.492245 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17327d7e-8d87-4461-be7f-1ed4c1d17701-model-cache" pod "17327d7e-8d87-4461-be7f-1ed4c1d17701" (UID: "17327d7e-8d87-4461-be7f-1ed4c1d17701"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:59:29.492699 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:29.492679 5052 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17327d7e-8d87-4461-be7f-1ed4c1d17701-tmp-dir\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:59:29.492699 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:29.492700 5052 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17327d7e-8d87-4461-be7f-1ed4c1d17701-model-cache\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:59:29.492854 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:29.492713 5052 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17327d7e-8d87-4461-be7f-1ed4c1d17701-home\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:59:29.494082 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:29.494063 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17327d7e-8d87-4461-be7f-1ed4c1d17701-kube-api-access-zzrk4" pod "17327d7e-8d87-4461-be7f-1ed4c1d17701" (UID: "17327d7e-8d87-4461-be7f-1ed4c1d17701"). InnerVolumeSpecName "kube-api-access-zzrk4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:59:29.494243 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:29.494223 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17327d7e-8d87-4461-be7f-1ed4c1d17701-tls-certs" pod "17327d7e-8d87-4461-be7f-1ed4c1d17701" (UID: "17327d7e-8d87-4461-be7f-1ed4c1d17701"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:59:29.494309 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:29.494251 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17327d7e-8d87-4461-be7f-1ed4c1d17701-dshm" pod "17327d7e-8d87-4461-be7f-1ed4c1d17701" (UID: "17327d7e-8d87-4461-be7f-1ed4c1d17701"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:59:29.593870 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:29.593792 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzrk4\" (UniqueName: \"kubernetes.io/projected/17327d7e-8d87-4461-be7f-1ed4c1d17701-kube-api-access-zzrk4\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:59:29.593870 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:29.593821 5052 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17327d7e-8d87-4461-be7f-1ed4c1d17701-tls-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:59:29.593870 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:29.593830 5052 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17327d7e-8d87-4461-be7f-1ed4c1d17701-dshm\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:59:30.080363 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:30.080328 5052 generic.go:356] "Generic (PLEG): container finished" podID="17327d7e-8d87-4461-be7f-1ed4c1d17701" containerID="3aea17113c5222245dffb11430a8b3d00ed45d767d3571ef83b44f9ce2933d4b" exitCode=0 Oct 30 16:59:30.080824 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:30.080402 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9c1e26/test-model-9c1e26-kserve-7cc944cb5f-xjljc" Oct 30 16:59:30.080824 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:30.080418 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9c1e26/test-model-9c1e26-kserve-7cc944cb5f-xjljc" event={"ID":"17327d7e-8d87-4461-be7f-1ed4c1d17701","Type":"ContainerDied","Data":"3aea17113c5222245dffb11430a8b3d00ed45d767d3571ef83b44f9ce2933d4b"} Oct 30 16:59:30.080824 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:30.080465 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9c1e26/test-model-9c1e26-kserve-7cc944cb5f-xjljc" event={"ID":"17327d7e-8d87-4461-be7f-1ed4c1d17701","Type":"ContainerDied","Data":"e9e2fe5b128b5646ac69b708b860798e2095ec11d1653a461d743fc3605e6d7d"} Oct 30 16:59:30.080824 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:30.080483 5052 scope.go:122] "RemoveContainer" containerID="3aea17113c5222245dffb11430a8b3d00ed45d767d3571ef83b44f9ce2933d4b" Oct 30 16:59:30.092772 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:30.092755 5052 scope.go:122] "RemoveContainer" containerID="3aea17113c5222245dffb11430a8b3d00ed45d767d3571ef83b44f9ce2933d4b" Oct 30 16:59:30.093052 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:59:30.093033 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3aea17113c5222245dffb11430a8b3d00ed45d767d3571ef83b44f9ce2933d4b\": container with ID starting with 3aea17113c5222245dffb11430a8b3d00ed45d767d3571ef83b44f9ce2933d4b not found: ID does not exist" containerID="3aea17113c5222245dffb11430a8b3d00ed45d767d3571ef83b44f9ce2933d4b" Oct 30 16:59:30.093112 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:30.093059 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3aea17113c5222245dffb11430a8b3d00ed45d767d3571ef83b44f9ce2933d4b"} err="failed to get container status \"3aea17113c5222245dffb11430a8b3d00ed45d767d3571ef83b44f9ce2933d4b\": rpc error: code = NotFound desc = could not find container \"3aea17113c5222245dffb11430a8b3d00ed45d767d3571ef83b44f9ce2933d4b\": container with ID starting with 3aea17113c5222245dffb11430a8b3d00ed45d767d3571ef83b44f9ce2933d4b not found: ID does not exist" Oct 30 16:59:30.139210 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:30.139179 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-9c1e26/test-model-9c1e26-kserve-7cc944cb5f-xjljc"] Oct 30 16:59:30.151856 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:30.151835 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-9c1e26/test-model-9c1e26-kserve-7cc944cb5f-xjljc"] Oct 30 16:59:30.600211 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:30.600177 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17327d7e-8d87-4461-be7f-1ed4c1d17701" path="/var/lib/kubelet/pods/17327d7e-8d87-4461-be7f-1ed4c1d17701/volumes" Oct 30 16:59:45.956291 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:45.956253 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-17f60f/test-model-17f60f-kserve-6d99c5d4f5-n2vss"] Oct 30 16:59:48.842021 ip-10-0-131-136 kubenswrapper[5052]: E1030 16:59:48.841979 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 16:59:52.646114 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:52.646078 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a59502-openshift-default-74889b7c5f-9m8g5"] Oct 30 16:59:52.646636 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:52.646343 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-a59502-openshift-default-74889b7c5f-9m8g5" podUID="d7f77c25-893f-40a2-a406-1891790e4cb5" containerName="istio-proxy" containerID="cri-o://360a67d2a2b0835a961c6a60c20a1225eae3dddaf4be89423b1fa3345dad7cfa" gracePeriod=30 Oct 30 16:59:55.459667 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:55.459634 5052 patch_prober.go:29] interesting pod/e2e-inf-a-a59502-openshift-default-74889b7c5f-9m8g5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 16:59:55.460048 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:55.459693 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-a59502-openshift-default-74889b7c5f-9m8g5" podUID="d7f77c25-893f-40a2-a406-1891790e4cb5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 16:59:55.827403 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:55.827319 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cdwcj"] Oct 30 16:59:58.702215 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:58.702169 5052 generic.go:356] "Generic (PLEG): container finished" podID="d7f77c25-893f-40a2-a406-1891790e4cb5" containerID="360a67d2a2b0835a961c6a60c20a1225eae3dddaf4be89423b1fa3345dad7cfa" exitCode=0 Oct 30 16:59:58.711274 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:58.711233 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a59502-openshift-default-74889b7c5f-9m8g5" event={"ID":"d7f77c25-893f-40a2-a406-1891790e4cb5","Type":"ContainerDied","Data":"360a67d2a2b0835a961c6a60c20a1225eae3dddaf4be89423b1fa3345dad7cfa"} Oct 30 16:59:58.711397 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:58.711313 5052 scope.go:122] "RemoveContainer" containerID="bede44d865571edbe12a4667fdc6a7856a8ffd6bebc876fb78a0e14ca5361bbe" Oct 30 16:59:58.771614 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:58.757453 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a59502-openshift-default-74889b7c5f-9m8g5" Oct 30 16:59:58.849601 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:58.848844 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7f77c25-893f-40a2-a406-1891790e4cb5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7f77c25-893f-40a2-a406-1891790e4cb5-workload-certs\") pod \"d7f77c25-893f-40a2-a406-1891790e4cb5\" (UID: \"d7f77c25-893f-40a2-a406-1891790e4cb5\") " Oct 30 16:59:58.849601 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:58.848929 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7f77c25-893f-40a2-a406-1891790e4cb5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7f77c25-893f-40a2-a406-1891790e4cb5-istiod-ca-cert\") pod \"d7f77c25-893f-40a2-a406-1891790e4cb5\" (UID: \"d7f77c25-893f-40a2-a406-1891790e4cb5\") " Oct 30 16:59:58.849601 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:58.848971 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7f77c25-893f-40a2-a406-1891790e4cb5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7f77c25-893f-40a2-a406-1891790e4cb5-credential-socket\") pod \"d7f77c25-893f-40a2-a406-1891790e4cb5\" (UID: \"d7f77c25-893f-40a2-a406-1891790e4cb5\") " Oct 30 16:59:58.849601 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:58.849116 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7f77c25-893f-40a2-a406-1891790e4cb5-kube-api-access-d5cn6\" (UniqueName: \"kubernetes.io/projected/d7f77c25-893f-40a2-a406-1891790e4cb5-kube-api-access-d5cn6\") pod \"d7f77c25-893f-40a2-a406-1891790e4cb5\" (UID: \"d7f77c25-893f-40a2-a406-1891790e4cb5\") " Oct 30 16:59:58.849601 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:58.849175 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7f77c25-893f-40a2-a406-1891790e4cb5-istio-token\" (UniqueName: \"kubernetes.io/projected/d7f77c25-893f-40a2-a406-1891790e4cb5-istio-token\") pod \"d7f77c25-893f-40a2-a406-1891790e4cb5\" (UID: \"d7f77c25-893f-40a2-a406-1891790e4cb5\") " Oct 30 16:59:58.849601 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:58.849280 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7f77c25-893f-40a2-a406-1891790e4cb5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7f77c25-893f-40a2-a406-1891790e4cb5-workload-socket\") pod \"d7f77c25-893f-40a2-a406-1891790e4cb5\" (UID: \"d7f77c25-893f-40a2-a406-1891790e4cb5\") " Oct 30 16:59:58.849601 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:58.849324 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7f77c25-893f-40a2-a406-1891790e4cb5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7f77c25-893f-40a2-a406-1891790e4cb5-istio-envoy\") pod \"d7f77c25-893f-40a2-a406-1891790e4cb5\" (UID: \"d7f77c25-893f-40a2-a406-1891790e4cb5\") " Oct 30 16:59:58.849601 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:58.849361 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d7f77c25-893f-40a2-a406-1891790e4cb5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7f77c25-893f-40a2-a406-1891790e4cb5-istio-podinfo\") pod \"d7f77c25-893f-40a2-a406-1891790e4cb5\" (UID: \"d7f77c25-893f-40a2-a406-1891790e4cb5\") " Oct 30 16:59:58.849601 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:58.849434 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7f77c25-893f-40a2-a406-1891790e4cb5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7f77c25-893f-40a2-a406-1891790e4cb5-istio-data\") pod \"d7f77c25-893f-40a2-a406-1891790e4cb5\" (UID: \"d7f77c25-893f-40a2-a406-1891790e4cb5\") " Oct 30 16:59:58.853599 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:58.850508 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7f77c25-893f-40a2-a406-1891790e4cb5-istiod-ca-cert" pod "d7f77c25-893f-40a2-a406-1891790e4cb5" (UID: "d7f77c25-893f-40a2-a406-1891790e4cb5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:59:58.853599 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:58.850812 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7f77c25-893f-40a2-a406-1891790e4cb5-workload-certs" pod "d7f77c25-893f-40a2-a406-1891790e4cb5" (UID: "d7f77c25-893f-40a2-a406-1891790e4cb5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:59:58.853599 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:58.851002 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7f77c25-893f-40a2-a406-1891790e4cb5-workload-socket" pod "d7f77c25-893f-40a2-a406-1891790e4cb5" (UID: "d7f77c25-893f-40a2-a406-1891790e4cb5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:59:58.853599 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:58.851900 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7f77c25-893f-40a2-a406-1891790e4cb5-istio-data" pod "d7f77c25-893f-40a2-a406-1891790e4cb5" (UID: "d7f77c25-893f-40a2-a406-1891790e4cb5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:59:58.853599 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:58.852393 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7f77c25-893f-40a2-a406-1891790e4cb5-credential-socket" pod "d7f77c25-893f-40a2-a406-1891790e4cb5" (UID: "d7f77c25-893f-40a2-a406-1891790e4cb5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:59:58.853599 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:58.852844 5052 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7f77c25-893f-40a2-a406-1891790e4cb5-istiod-ca-cert\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:59:58.853599 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:58.852864 5052 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7f77c25-893f-40a2-a406-1891790e4cb5-credential-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:59:58.853599 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:58.852876 5052 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7f77c25-893f-40a2-a406-1891790e4cb5-workload-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:59:58.853599 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:58.852889 5052 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7f77c25-893f-40a2-a406-1891790e4cb5-istio-data\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:59:58.853599 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:58.852900 5052 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7f77c25-893f-40a2-a406-1891790e4cb5-workload-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:59:58.880479 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:58.880446 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7f77c25-893f-40a2-a406-1891790e4cb5-istio-envoy" pod "d7f77c25-893f-40a2-a406-1891790e4cb5" (UID: "d7f77c25-893f-40a2-a406-1891790e4cb5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:59:58.883558 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:58.883526 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7f77c25-893f-40a2-a406-1891790e4cb5-kube-api-access-d5cn6" pod "d7f77c25-893f-40a2-a406-1891790e4cb5" (UID: "d7f77c25-893f-40a2-a406-1891790e4cb5"). InnerVolumeSpecName "kube-api-access-d5cn6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:59:58.890058 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:58.890016 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7f77c25-893f-40a2-a406-1891790e4cb5-istio-token" pod "d7f77c25-893f-40a2-a406-1891790e4cb5" (UID: "d7f77c25-893f-40a2-a406-1891790e4cb5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:59:58.891840 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:58.891543 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d7f77c25-893f-40a2-a406-1891790e4cb5-istio-podinfo" pod "d7f77c25-893f-40a2-a406-1891790e4cb5" (UID: "d7f77c25-893f-40a2-a406-1891790e4cb5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:59:58.954494 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:58.954416 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5cn6\" (UniqueName: \"kubernetes.io/projected/d7f77c25-893f-40a2-a406-1891790e4cb5-kube-api-access-d5cn6\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:59:58.954494 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:58.954449 5052 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7f77c25-893f-40a2-a406-1891790e4cb5-istio-token\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:59:58.954494 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:58.954458 5052 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7f77c25-893f-40a2-a406-1891790e4cb5-istio-envoy\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:59:58.954494 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:58.954470 5052 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7f77c25-893f-40a2-a406-1891790e4cb5-istio-podinfo\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 16:59:59.712258 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:59.712218 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a59502-openshift-default-74889b7c5f-9m8g5" event={"ID":"d7f77c25-893f-40a2-a406-1891790e4cb5","Type":"ContainerDied","Data":"1de9440a9b246f56fd568ad961d5ca4ea5093d14d22e94fa5cdc24352a499c08"} Oct 30 16:59:59.712676 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:59.712281 5052 scope.go:122] "RemoveContainer" containerID="360a67d2a2b0835a961c6a60c20a1225eae3dddaf4be89423b1fa3345dad7cfa" Oct 30 16:59:59.712676 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:59.712238 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a59502-openshift-default-74889b7c5f-9m8g5" Oct 30 16:59:59.781776 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:59.781746 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a59502-openshift-default-74889b7c5f-9m8g5"] Oct 30 16:59:59.791007 ip-10-0-131-136 kubenswrapper[5052]: I1030 16:59:59.790983 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-a59502-openshift-default-74889b7c5f-9m8g5"] Oct 30 17:00:00.328192 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:00.328157 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9bfcac-29889660-vn8xn"] Oct 30 17:00:00.329517 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:00.329473 5052 cpu_manager.go:414] "containerMap: removing container" podUID="c2915da3-9830-4751-b58f-837063130fa1" containerName="main" Oct 30 17:00:00.329517 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:00.329496 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2915da3-9830-4751-b58f-837063130fa1" containerName="main" Oct 30 17:00:00.329517 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:00.329507 5052 cpu_manager.go:414] "containerMap: removing container" podUID="d7f77c25-893f-40a2-a406-1891790e4cb5" containerName="istio-proxy" Oct 30 17:00:00.329517 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:00.329513 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7f77c25-893f-40a2-a406-1891790e4cb5" containerName="istio-proxy" Oct 30 17:00:00.329850 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:00.329531 5052 cpu_manager.go:414] "containerMap: removing container" podUID="17327d7e-8d87-4461-be7f-1ed4c1d17701" containerName="main" Oct 30 17:00:00.329850 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:00.329536 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17327d7e-8d87-4461-be7f-1ed4c1d17701" containerName="main" Oct 30 17:00:00.329850 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:00.329761 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2915da3-9830-4751-b58f-837063130fa1" containerName="main" Oct 30 17:00:00.329850 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:00.329781 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7f77c25-893f-40a2-a406-1891790e4cb5" containerName="istio-proxy" Oct 30 17:00:00.329850 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:00.329797 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="17327d7e-8d87-4461-be7f-1ed4c1d17701" containerName="main" Oct 30 17:00:00.335760 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:00.335738 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9bfcac-29889660-vn8xn" Oct 30 17:00:00.353041 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:00.353010 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f92caf-29889660-44r98"] Oct 30 17:00:00.354275 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:00.354247 5052 cpu_manager.go:414] "containerMap: removing container" podUID="d7f77c25-893f-40a2-a406-1891790e4cb5" containerName="istio-proxy" Oct 30 17:00:00.354275 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:00.354274 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7f77c25-893f-40a2-a406-1891790e4cb5" containerName="istio-proxy" Oct 30 17:00:00.354518 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:00.354472 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7f77c25-893f-40a2-a406-1891790e4cb5" containerName="istio-proxy" Oct 30 17:00:00.360294 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:00.360212 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9bfcac-29889660-vn8xn"] Oct 30 17:00:00.360517 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:00.360473 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f92caf-29889660-44r98" Oct 30 17:00:00.360796 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:00.360519 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 17:00:00.373706 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:00.373678 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-17f60f-29889660-8tgc2"] Oct 30 17:00:00.379957 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:00.379931 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-17f60f-29889660-8tgc2" Oct 30 17:00:00.381622 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:00.381564 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jggrx\" (UniqueName: \"kubernetes.io/projected/ec93f0a0-6634-4719-8289-d936260d7973-kube-api-access-jggrx\") pod \"maas-api-key-cleanup-e2e-inf-b-9bfcac-29889660-vn8xn\" (UID: \"ec93f0a0-6634-4719-8289-d936260d7973\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9bfcac-29889660-vn8xn" Oct 30 17:00:00.381742 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:00.381656 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec93f0a0-6634-4719-8289-d936260d7973-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-9bfcac-29889660-vn8xn\" (UID: \"ec93f0a0-6634-4719-8289-d936260d7973\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9bfcac-29889660-vn8xn" Oct 30 17:00:00.381834 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:00.381816 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdcqq\" (UniqueName: \"kubernetes.io/projected/1760617a-b6d1-40e5-a3b2-ac46dd644dee-kube-api-access-fdcqq\") pod \"maas-api-key-cleanup-e2e-inf-b-f92caf-29889660-44r98\" (UID: \"1760617a-b6d1-40e5-a3b2-ac46dd644dee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f92caf-29889660-44r98" Oct 30 17:00:00.382072 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:00.382056 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1760617a-b6d1-40e5-a3b2-ac46dd644dee-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-f92caf-29889660-44r98\" (UID: \"1760617a-b6d1-40e5-a3b2-ac46dd644dee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f92caf-29889660-44r98" Oct 30 17:00:00.383800 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:00.383774 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f92caf-29889660-44r98"] Oct 30 17:00:00.404721 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:00.404696 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-17f60f-29889660-8tgc2"] Oct 30 17:00:00.483702 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:00.483665 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jggrx\" (UniqueName: \"kubernetes.io/projected/ec93f0a0-6634-4719-8289-d936260d7973-kube-api-access-jggrx\") pod \"maas-api-key-cleanup-e2e-inf-b-9bfcac-29889660-vn8xn\" (UID: \"ec93f0a0-6634-4719-8289-d936260d7973\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9bfcac-29889660-vn8xn" Oct 30 17:00:00.483895 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:00.483714 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e73618ad-d79c-4833-bb07-e22057d3f9f7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-17f60f-29889660-8tgc2\" (UID: \"e73618ad-d79c-4833-bb07-e22057d3f9f7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-17f60f-29889660-8tgc2" Oct 30 17:00:00.483895 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:00.483748 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec93f0a0-6634-4719-8289-d936260d7973-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-9bfcac-29889660-vn8xn\" (UID: \"ec93f0a0-6634-4719-8289-d936260d7973\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9bfcac-29889660-vn8xn" Oct 30 17:00:00.484029 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:00.483903 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zttfl\" (UniqueName: \"kubernetes.io/projected/e73618ad-d79c-4833-bb07-e22057d3f9f7-kube-api-access-zttfl\") pod \"maas-api-key-cleanup-e2e-inf-a-17f60f-29889660-8tgc2\" (UID: \"e73618ad-d79c-4833-bb07-e22057d3f9f7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-17f60f-29889660-8tgc2" Oct 30 17:00:00.484029 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:00.483960 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdcqq\" (UniqueName: \"kubernetes.io/projected/1760617a-b6d1-40e5-a3b2-ac46dd644dee-kube-api-access-fdcqq\") pod \"maas-api-key-cleanup-e2e-inf-b-f92caf-29889660-44r98\" (UID: \"1760617a-b6d1-40e5-a3b2-ac46dd644dee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f92caf-29889660-44r98" Oct 30 17:00:00.484256 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:00.484229 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1760617a-b6d1-40e5-a3b2-ac46dd644dee-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-f92caf-29889660-44r98\" (UID: \"1760617a-b6d1-40e5-a3b2-ac46dd644dee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f92caf-29889660-44r98" Oct 30 17:00:00.484520 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:00.484499 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec93f0a0-6634-4719-8289-d936260d7973-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-9bfcac-29889660-vn8xn\" (UID: \"ec93f0a0-6634-4719-8289-d936260d7973\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9bfcac-29889660-vn8xn" Oct 30 17:00:00.484866 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:00.484842 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1760617a-b6d1-40e5-a3b2-ac46dd644dee-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-f92caf-29889660-44r98\" (UID: \"1760617a-b6d1-40e5-a3b2-ac46dd644dee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f92caf-29889660-44r98" Oct 30 17:00:00.509700 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:00.509665 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdcqq\" (UniqueName: \"kubernetes.io/projected/1760617a-b6d1-40e5-a3b2-ac46dd644dee-kube-api-access-fdcqq\") pod \"maas-api-key-cleanup-e2e-inf-b-f92caf-29889660-44r98\" (UID: \"1760617a-b6d1-40e5-a3b2-ac46dd644dee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f92caf-29889660-44r98" Oct 30 17:00:00.509818 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:00.509673 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jggrx\" (UniqueName: \"kubernetes.io/projected/ec93f0a0-6634-4719-8289-d936260d7973-kube-api-access-jggrx\") pod \"maas-api-key-cleanup-e2e-inf-b-9bfcac-29889660-vn8xn\" (UID: \"ec93f0a0-6634-4719-8289-d936260d7973\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9bfcac-29889660-vn8xn" Oct 30 17:00:00.585134 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:00.585038 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zttfl\" (UniqueName: \"kubernetes.io/projected/e73618ad-d79c-4833-bb07-e22057d3f9f7-kube-api-access-zttfl\") pod \"maas-api-key-cleanup-e2e-inf-a-17f60f-29889660-8tgc2\" (UID: \"e73618ad-d79c-4833-bb07-e22057d3f9f7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-17f60f-29889660-8tgc2" Oct 30 17:00:00.585347 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:00.585329 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e73618ad-d79c-4833-bb07-e22057d3f9f7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-17f60f-29889660-8tgc2\" (UID: \"e73618ad-d79c-4833-bb07-e22057d3f9f7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-17f60f-29889660-8tgc2" Oct 30 17:00:00.586139 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:00.586115 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e73618ad-d79c-4833-bb07-e22057d3f9f7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-17f60f-29889660-8tgc2\" (UID: \"e73618ad-d79c-4833-bb07-e22057d3f9f7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-17f60f-29889660-8tgc2" Oct 30 17:00:00.601997 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:00.601964 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7f77c25-893f-40a2-a406-1891790e4cb5" path="/var/lib/kubelet/pods/d7f77c25-893f-40a2-a406-1891790e4cb5/volumes" Oct 30 17:00:00.610269 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:00.610243 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zttfl\" (UniqueName: \"kubernetes.io/projected/e73618ad-d79c-4833-bb07-e22057d3f9f7-kube-api-access-zttfl\") pod \"maas-api-key-cleanup-e2e-inf-a-17f60f-29889660-8tgc2\" (UID: \"e73618ad-d79c-4833-bb07-e22057d3f9f7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-17f60f-29889660-8tgc2" Oct 30 17:00:00.653040 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:00.653013 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9bfcac-29889660-vn8xn" Oct 30 17:00:00.697572 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:00.697543 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f92caf-29889660-44r98" Oct 30 17:00:00.704850 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:00.704822 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-17f60f-29889660-8tgc2" Oct 30 17:00:00.979442 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:00.979400 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-17f60f/test-model-17f60f-kserve-6d99c5d4f5-n2vss" podUID="06b94bcd-8c5a-4143-a936-34d62f4189b0" containerName="main" containerID="cri-o://a95ab4dfe0d08568ad11183f96f79b971e509ca877f188bee22411ab5c27614f" gracePeriod=45 Oct 30 17:00:00.985066 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:00.985037 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9bfcac-29889660-vn8xn"] Oct 30 17:00:00.985464 ip-10-0-131-136 kubenswrapper[5052]: W1030 17:00:00.985438 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec93f0a0_6634_4719_8289_d936260d7973.slice/crio-633197bcfaad8283e066fea1cc9c24b237acf42ad861cd7ba4db14e6b7541172 WatchSource:0}: Error finding container 633197bcfaad8283e066fea1cc9c24b237acf42ad861cd7ba4db14e6b7541172: Status 404 returned error can't find the container with id 633197bcfaad8283e066fea1cc9c24b237acf42ad861cd7ba4db14e6b7541172 Oct 30 17:00:00.988126 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:00.988107 5052 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 17:00:01.049128 ip-10-0-131-136 kubenswrapper[5052]: W1030 17:00:01.049099 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1760617a_b6d1_40e5_a3b2_ac46dd644dee.slice/crio-21d804a2a7b295d68724f7b592b5bd764cd8b046b6acd37a5e51b9670588d345 WatchSource:0}: Error finding container 21d804a2a7b295d68724f7b592b5bd764cd8b046b6acd37a5e51b9670588d345: Status 404 returned error can't find the container with id 21d804a2a7b295d68724f7b592b5bd764cd8b046b6acd37a5e51b9670588d345 Oct 30 17:00:01.049528 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:01.049504 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f92caf-29889660-44r98"] Oct 30 17:00:01.065991 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:01.065964 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-17f60f-29889660-8tgc2"] Oct 30 17:00:01.073342 ip-10-0-131-136 kubenswrapper[5052]: W1030 17:00:01.073312 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode73618ad_d79c_4833_bb07_e22057d3f9f7.slice/crio-f99554ae0cc259e4f16335f3078151a60b549a7cdd08fff59f0ce95b9e96233d WatchSource:0}: Error finding container f99554ae0cc259e4f16335f3078151a60b549a7cdd08fff59f0ce95b9e96233d: Status 404 returned error can't find the container with id f99554ae0cc259e4f16335f3078151a60b549a7cdd08fff59f0ce95b9e96233d Oct 30 17:00:01.262117 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:01.262090 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-17f60f/test-model-17f60f-kserve-6d99c5d4f5-n2vss" Oct 30 17:00:01.302798 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:01.302766 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06b94bcd-8c5a-4143-a936-34d62f4189b0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/06b94bcd-8c5a-4143-a936-34d62f4189b0-dshm\") pod \"06b94bcd-8c5a-4143-a936-34d62f4189b0\" (UID: \"06b94bcd-8c5a-4143-a936-34d62f4189b0\") " Oct 30 17:00:01.302991 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:01.302875 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06b94bcd-8c5a-4143-a936-34d62f4189b0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06b94bcd-8c5a-4143-a936-34d62f4189b0-model-cache\") pod \"06b94bcd-8c5a-4143-a936-34d62f4189b0\" (UID: \"06b94bcd-8c5a-4143-a936-34d62f4189b0\") " Oct 30 17:00:01.302991 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:01.302931 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06b94bcd-8c5a-4143-a936-34d62f4189b0-home\" (UniqueName: \"kubernetes.io/empty-dir/06b94bcd-8c5a-4143-a936-34d62f4189b0-home\") pod \"06b94bcd-8c5a-4143-a936-34d62f4189b0\" (UID: \"06b94bcd-8c5a-4143-a936-34d62f4189b0\") " Oct 30 17:00:01.302991 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:01.302962 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06b94bcd-8c5a-4143-a936-34d62f4189b0-tls-certs\" (UniqueName: \"kubernetes.io/secret/06b94bcd-8c5a-4143-a936-34d62f4189b0-tls-certs\") pod \"06b94bcd-8c5a-4143-a936-34d62f4189b0\" (UID: \"06b94bcd-8c5a-4143-a936-34d62f4189b0\") " Oct 30 17:00:01.303156 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:01.302998 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06b94bcd-8c5a-4143-a936-34d62f4189b0-kube-api-access-99569\" (UniqueName: \"kubernetes.io/projected/06b94bcd-8c5a-4143-a936-34d62f4189b0-kube-api-access-99569\") pod \"06b94bcd-8c5a-4143-a936-34d62f4189b0\" (UID: \"06b94bcd-8c5a-4143-a936-34d62f4189b0\") " Oct 30 17:00:01.303156 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:01.303072 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06b94bcd-8c5a-4143-a936-34d62f4189b0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06b94bcd-8c5a-4143-a936-34d62f4189b0-tmp-dir\") pod \"06b94bcd-8c5a-4143-a936-34d62f4189b0\" (UID: \"06b94bcd-8c5a-4143-a936-34d62f4189b0\") " Oct 30 17:00:01.304113 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:01.304089 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06b94bcd-8c5a-4143-a936-34d62f4189b0-home" pod "06b94bcd-8c5a-4143-a936-34d62f4189b0" (UID: "06b94bcd-8c5a-4143-a936-34d62f4189b0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:00:01.304636 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:01.304551 5052 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06b94bcd-8c5a-4143-a936-34d62f4189b0-home\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:00:01.305251 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:01.305218 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06b94bcd-8c5a-4143-a936-34d62f4189b0-model-cache" pod "06b94bcd-8c5a-4143-a936-34d62f4189b0" (UID: "06b94bcd-8c5a-4143-a936-34d62f4189b0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:00:01.305432 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:01.305398 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06b94bcd-8c5a-4143-a936-34d62f4189b0-tmp-dir" pod "06b94bcd-8c5a-4143-a936-34d62f4189b0" (UID: "06b94bcd-8c5a-4143-a936-34d62f4189b0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:00:01.307955 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:01.307930 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06b94bcd-8c5a-4143-a936-34d62f4189b0-tls-certs" pod "06b94bcd-8c5a-4143-a936-34d62f4189b0" (UID: "06b94bcd-8c5a-4143-a936-34d62f4189b0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:00:01.308208 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:01.308183 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06b94bcd-8c5a-4143-a936-34d62f4189b0-dshm" pod "06b94bcd-8c5a-4143-a936-34d62f4189b0" (UID: "06b94bcd-8c5a-4143-a936-34d62f4189b0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:00:01.313601 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:01.309899 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06b94bcd-8c5a-4143-a936-34d62f4189b0-kube-api-access-99569" pod "06b94bcd-8c5a-4143-a936-34d62f4189b0" (UID: "06b94bcd-8c5a-4143-a936-34d62f4189b0"). InnerVolumeSpecName "kube-api-access-99569". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:00:01.406721 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:01.406682 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99569\" (UniqueName: \"kubernetes.io/projected/06b94bcd-8c5a-4143-a936-34d62f4189b0-kube-api-access-99569\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:00:01.406721 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:01.406718 5052 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06b94bcd-8c5a-4143-a936-34d62f4189b0-tmp-dir\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:00:01.406934 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:01.406736 5052 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06b94bcd-8c5a-4143-a936-34d62f4189b0-dshm\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:00:01.406934 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:01.406749 5052 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06b94bcd-8c5a-4143-a936-34d62f4189b0-model-cache\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:00:01.406934 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:01.406762 5052 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06b94bcd-8c5a-4143-a936-34d62f4189b0-tls-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:00:01.797634 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:01.793148 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9bfcac-29889660-vn8xn" event={"ID":"ec93f0a0-6634-4719-8289-d936260d7973","Type":"ContainerStarted","Data":"633197bcfaad8283e066fea1cc9c24b237acf42ad861cd7ba4db14e6b7541172"} Oct 30 17:00:01.797634 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:01.796991 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f92caf-29889660-44r98" event={"ID":"1760617a-b6d1-40e5-a3b2-ac46dd644dee","Type":"ContainerStarted","Data":"21d804a2a7b295d68724f7b592b5bd764cd8b046b6acd37a5e51b9670588d345"} Oct 30 17:00:01.799335 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:01.799236 5052 generic.go:356] "Generic (PLEG): container finished" podID="06b94bcd-8c5a-4143-a936-34d62f4189b0" containerID="a95ab4dfe0d08568ad11183f96f79b971e509ca877f188bee22411ab5c27614f" exitCode=0 Oct 30 17:00:01.799678 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:01.799648 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-17f60f/test-model-17f60f-kserve-6d99c5d4f5-n2vss" Oct 30 17:00:01.800404 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:01.800362 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-17f60f/test-model-17f60f-kserve-6d99c5d4f5-n2vss" event={"ID":"06b94bcd-8c5a-4143-a936-34d62f4189b0","Type":"ContainerDied","Data":"a95ab4dfe0d08568ad11183f96f79b971e509ca877f188bee22411ab5c27614f"} Oct 30 17:00:01.800514 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:01.800406 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-17f60f/test-model-17f60f-kserve-6d99c5d4f5-n2vss" event={"ID":"06b94bcd-8c5a-4143-a936-34d62f4189b0","Type":"ContainerDied","Data":"3d9745094c61d86ccfe25185b494025ae1bf9686cdb14f759829b92e80ad36e6"} Oct 30 17:00:01.800514 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:01.800425 5052 scope.go:122] "RemoveContainer" containerID="a95ab4dfe0d08568ad11183f96f79b971e509ca877f188bee22411ab5c27614f" Oct 30 17:00:01.802841 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:01.802480 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-17f60f-29889660-8tgc2" event={"ID":"e73618ad-d79c-4833-bb07-e22057d3f9f7","Type":"ContainerStarted","Data":"f99554ae0cc259e4f16335f3078151a60b549a7cdd08fff59f0ce95b9e96233d"} Oct 30 17:00:01.818205 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:01.817601 5052 scope.go:122] "RemoveContainer" containerID="a95ab4dfe0d08568ad11183f96f79b971e509ca877f188bee22411ab5c27614f" Oct 30 17:00:01.818205 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:00:01.817951 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a95ab4dfe0d08568ad11183f96f79b971e509ca877f188bee22411ab5c27614f\": container with ID starting with a95ab4dfe0d08568ad11183f96f79b971e509ca877f188bee22411ab5c27614f not found: ID does not exist" containerID="a95ab4dfe0d08568ad11183f96f79b971e509ca877f188bee22411ab5c27614f" Oct 30 17:00:01.818205 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:01.817990 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a95ab4dfe0d08568ad11183f96f79b971e509ca877f188bee22411ab5c27614f"} err="failed to get container status \"a95ab4dfe0d08568ad11183f96f79b971e509ca877f188bee22411ab5c27614f\": rpc error: code = NotFound desc = could not find container \"a95ab4dfe0d08568ad11183f96f79b971e509ca877f188bee22411ab5c27614f\": container with ID starting with a95ab4dfe0d08568ad11183f96f79b971e509ca877f188bee22411ab5c27614f not found: ID does not exist" Oct 30 17:00:01.869497 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:01.869463 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-17f60f/test-model-17f60f-kserve-6d99c5d4f5-n2vss"] Oct 30 17:00:01.873758 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:01.873729 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-17f60f/test-model-17f60f-kserve-6d99c5d4f5-n2vss"] Oct 30 17:00:02.606146 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:02.606109 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06b94bcd-8c5a-4143-a936-34d62f4189b0" path="/var/lib/kubelet/pods/06b94bcd-8c5a-4143-a936-34d62f4189b0/volumes" Oct 30 17:00:03.794647 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:03.794549 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9c1e26-5f76c9f698-gvxqs"] Oct 30 17:00:05.857413 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:05.857372 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-17f60f-29889660-8tgc2" event={"ID":"e73618ad-d79c-4833-bb07-e22057d3f9f7","Type":"ContainerStarted","Data":"27400c44cba8f9e1ed5b99a44c24a5359d605bed386967598e94d801b6a42f6f"} Oct 30 17:00:05.859526 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:05.859491 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9bfcac-29889660-vn8xn" event={"ID":"ec93f0a0-6634-4719-8289-d936260d7973","Type":"ContainerStarted","Data":"f86ac208ecf89d74e7419735da126613d56df1fc165fed5db444d0291f15f133"} Oct 30 17:00:05.861130 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:05.861108 5052 generic.go:356] "Generic (PLEG): container finished" podID="1760617a-b6d1-40e5-a3b2-ac46dd644dee" containerID="863681b897568d63c6f81a87980e5bfe38dbcada1345bba48b70de83be75ff96" exitCode=60 Oct 30 17:00:05.861213 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:05.861180 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f92caf-29889660-44r98" event={"ID":"1760617a-b6d1-40e5-a3b2-ac46dd644dee","Type":"ContainerDied","Data":"863681b897568d63c6f81a87980e5bfe38dbcada1345bba48b70de83be75ff96"} Oct 30 17:00:05.861414 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:05.861399 5052 scope.go:122] "RemoveContainer" containerID="863681b897568d63c6f81a87980e5bfe38dbcada1345bba48b70de83be75ff96" Oct 30 17:00:05.908030 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:05.907986 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-17f60f-29889660-8tgc2" podStartSLOduration=1.89200038 podStartE2EDuration="5.907973173s" totalImagesPullingTime="4.015972793s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:00:05.904088316 +0000 UTC m=+2361.431423536" watchObservedRunningTime="2026-10-30 17:00:05.907973173 +0000 UTC m=+2361.435308392" Oct 30 17:00:05.997955 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:05.997869 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9bfcac-29889660-vn8xn" podStartSLOduration=1.889963168 podStartE2EDuration="5.997856519s" totalImagesPullingTime="4.107893351s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:00:05.993283899 +0000 UTC m=+2361.520619113" watchObservedRunningTime="2026-10-30 17:00:05.997856519 +0000 UTC m=+2361.525191737" Oct 30 17:00:06.872108 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:06.872080 5052 generic.go:356] "Generic (PLEG): container finished" podID="e73618ad-d79c-4833-bb07-e22057d3f9f7" containerID="27400c44cba8f9e1ed5b99a44c24a5359d605bed386967598e94d801b6a42f6f" exitCode=60 Oct 30 17:00:06.872473 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:06.872128 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-17f60f-29889660-8tgc2" event={"ID":"e73618ad-d79c-4833-bb07-e22057d3f9f7","Type":"ContainerDied","Data":"27400c44cba8f9e1ed5b99a44c24a5359d605bed386967598e94d801b6a42f6f"} Oct 30 17:00:06.872537 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:06.872504 5052 scope.go:122] "RemoveContainer" containerID="27400c44cba8f9e1ed5b99a44c24a5359d605bed386967598e94d801b6a42f6f" Oct 30 17:00:06.874059 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:06.874032 5052 generic.go:356] "Generic (PLEG): container finished" podID="ec93f0a0-6634-4719-8289-d936260d7973" containerID="f86ac208ecf89d74e7419735da126613d56df1fc165fed5db444d0291f15f133" exitCode=60 Oct 30 17:00:06.874159 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:06.874107 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9bfcac-29889660-vn8xn" event={"ID":"ec93f0a0-6634-4719-8289-d936260d7973","Type":"ContainerDied","Data":"f86ac208ecf89d74e7419735da126613d56df1fc165fed5db444d0291f15f133"} Oct 30 17:00:06.874436 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:06.874413 5052 scope.go:122] "RemoveContainer" containerID="f86ac208ecf89d74e7419735da126613d56df1fc165fed5db444d0291f15f133" Oct 30 17:00:06.876344 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:06.876319 5052 generic.go:356] "Generic (PLEG): container finished" podID="1760617a-b6d1-40e5-a3b2-ac46dd644dee" containerID="373412f835cae64fc1a61b2e5d3fc06de691da6a3e945df9ddde8a5b7a264b39" exitCode=60 Oct 30 17:00:06.876433 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:06.876364 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f92caf-29889660-44r98" event={"ID":"1760617a-b6d1-40e5-a3b2-ac46dd644dee","Type":"ContainerDied","Data":"373412f835cae64fc1a61b2e5d3fc06de691da6a3e945df9ddde8a5b7a264b39"} Oct 30 17:00:06.876433 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:06.876415 5052 scope.go:122] "RemoveContainer" containerID="863681b897568d63c6f81a87980e5bfe38dbcada1345bba48b70de83be75ff96" Oct 30 17:00:06.876711 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:06.876692 5052 scope.go:122] "RemoveContainer" containerID="373412f835cae64fc1a61b2e5d3fc06de691da6a3e945df9ddde8a5b7a264b39" Oct 30 17:00:06.877890 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:00:06.877867 5052 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-f92caf-29889660-44r98_odh-ai-gateway-infra(1760617a-b6d1-40e5-a3b2-ac46dd644dee)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f92caf-29889660-44r98" podUID="1760617a-b6d1-40e5-a3b2-ac46dd644dee" Oct 30 17:00:07.047218 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:07.047185 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9bfcac-86969d48d7-j9dgf"] Oct 30 17:00:07.898370 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:07.897193 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-17f60f-29889660-8tgc2" event={"ID":"e73618ad-d79c-4833-bb07-e22057d3f9f7","Type":"ContainerStarted","Data":"68eef90bd81dc4cc2a5c432828df371b8adc2361d0b756d83f3c04c5cdf12ba2"} Oct 30 17:00:07.900179 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:07.900154 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9bfcac-29889660-vn8xn" event={"ID":"ec93f0a0-6634-4719-8289-d936260d7973","Type":"ContainerStarted","Data":"917f6146bdbff9b93e30df4efaec361f54df903f449fa897ada111d83721cb07"} Oct 30 17:00:07.902655 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:07.902623 5052 scope.go:122] "RemoveContainer" containerID="373412f835cae64fc1a61b2e5d3fc06de691da6a3e945df9ddde8a5b7a264b39" Oct 30 17:00:07.903179 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:00:07.903143 5052 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-f92caf-29889660-44r98_odh-ai-gateway-infra(1760617a-b6d1-40e5-a3b2-ac46dd644dee)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f92caf-29889660-44r98" podUID="1760617a-b6d1-40e5-a3b2-ac46dd644dee" Oct 30 17:00:08.795729 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:08.795680 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-9c1e26-5f76c9f698-gvxqs" podUID="9e7b9aff-3301-4350-b511-262ff5088c3b" containerName="payload-processing" containerID="cri-o://3077a6727c59171224522b4dd4c2b03c017d1dd8ce9611468dad0b30654fb136" gracePeriod=25 Oct 30 17:00:08.914039 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:08.914007 5052 generic.go:356] "Generic (PLEG): container finished" podID="e73618ad-d79c-4833-bb07-e22057d3f9f7" containerID="68eef90bd81dc4cc2a5c432828df371b8adc2361d0b756d83f3c04c5cdf12ba2" exitCode=60 Oct 30 17:00:08.914570 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:08.914092 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-17f60f-29889660-8tgc2" event={"ID":"e73618ad-d79c-4833-bb07-e22057d3f9f7","Type":"ContainerDied","Data":"68eef90bd81dc4cc2a5c432828df371b8adc2361d0b756d83f3c04c5cdf12ba2"} Oct 30 17:00:08.914570 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:08.914157 5052 scope.go:122] "RemoveContainer" containerID="27400c44cba8f9e1ed5b99a44c24a5359d605bed386967598e94d801b6a42f6f" Oct 30 17:00:08.915216 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:08.915190 5052 scope.go:122] "RemoveContainer" containerID="68eef90bd81dc4cc2a5c432828df371b8adc2361d0b756d83f3c04c5cdf12ba2" Oct 30 17:00:08.916102 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:00:08.916070 5052 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-17f60f-29889660-8tgc2_odh-ai-gateway-infra(e73618ad-d79c-4833-bb07-e22057d3f9f7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-17f60f-29889660-8tgc2" podUID="e73618ad-d79c-4833-bb07-e22057d3f9f7" Oct 30 17:00:08.926613 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:08.923336 5052 generic.go:356] "Generic (PLEG): container finished" podID="ec93f0a0-6634-4719-8289-d936260d7973" containerID="917f6146bdbff9b93e30df4efaec361f54df903f449fa897ada111d83721cb07" exitCode=60 Oct 30 17:00:08.926613 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:08.923457 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9bfcac-29889660-vn8xn" event={"ID":"ec93f0a0-6634-4719-8289-d936260d7973","Type":"ContainerDied","Data":"917f6146bdbff9b93e30df4efaec361f54df903f449fa897ada111d83721cb07"} Oct 30 17:00:08.926613 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:08.923952 5052 scope.go:122] "RemoveContainer" containerID="917f6146bdbff9b93e30df4efaec361f54df903f449fa897ada111d83721cb07" Oct 30 17:00:08.926613 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:00:08.924450 5052 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-9bfcac-29889660-vn8xn_odh-ai-gateway-infra(ec93f0a0-6634-4719-8289-d936260d7973)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9bfcac-29889660-vn8xn" podUID="ec93f0a0-6634-4719-8289-d936260d7973" Oct 30 17:00:09.021653 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:09.020324 5052 scope.go:122] "RemoveContainer" containerID="f86ac208ecf89d74e7419735da126613d56df1fc165fed5db444d0291f15f133" Oct 30 17:00:09.178200 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:09.178165 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-9c1e26-5f76c9f698-gvxqs" Oct 30 17:00:09.244450 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:09.244418 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e7b9aff-3301-4350-b511-262ff5088c3b-kube-api-access-4sl5s\" (UniqueName: \"kubernetes.io/projected/9e7b9aff-3301-4350-b511-262ff5088c3b-kube-api-access-4sl5s\") pod \"9e7b9aff-3301-4350-b511-262ff5088c3b\" (UID: \"9e7b9aff-3301-4350-b511-262ff5088c3b\") " Oct 30 17:00:09.244648 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:09.244614 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e7b9aff-3301-4350-b511-262ff5088c3b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e7b9aff-3301-4350-b511-262ff5088c3b-plugins-config-volume\") pod \"9e7b9aff-3301-4350-b511-262ff5088c3b\" (UID: \"9e7b9aff-3301-4350-b511-262ff5088c3b\") " Oct 30 17:00:09.245342 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:09.245315 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e7b9aff-3301-4350-b511-262ff5088c3b-plugins-config-volume" pod "9e7b9aff-3301-4350-b511-262ff5088c3b" (UID: "9e7b9aff-3301-4350-b511-262ff5088c3b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:00:09.245708 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:09.245684 5052 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e7b9aff-3301-4350-b511-262ff5088c3b-plugins-config-volume\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:00:09.246976 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:09.246941 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e7b9aff-3301-4350-b511-262ff5088c3b-kube-api-access-4sl5s" pod "9e7b9aff-3301-4350-b511-262ff5088c3b" (UID: "9e7b9aff-3301-4350-b511-262ff5088c3b"). InnerVolumeSpecName "kube-api-access-4sl5s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:00:09.347863 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:09.347789 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4sl5s\" (UniqueName: \"kubernetes.io/projected/9e7b9aff-3301-4350-b511-262ff5088c3b-kube-api-access-4sl5s\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:00:09.932909 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:09.932875 5052 scope.go:122] "RemoveContainer" containerID="917f6146bdbff9b93e30df4efaec361f54df903f449fa897ada111d83721cb07" Oct 30 17:00:09.933329 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:00:09.933266 5052 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-9bfcac-29889660-vn8xn_odh-ai-gateway-infra(ec93f0a0-6634-4719-8289-d936260d7973)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9bfcac-29889660-vn8xn" podUID="ec93f0a0-6634-4719-8289-d936260d7973" Oct 30 17:00:09.933905 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:09.933879 5052 generic.go:356] "Generic (PLEG): container finished" podID="9e7b9aff-3301-4350-b511-262ff5088c3b" containerID="3077a6727c59171224522b4dd4c2b03c017d1dd8ce9611468dad0b30654fb136" exitCode=0 Oct 30 17:00:09.934013 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:09.933914 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-9c1e26-5f76c9f698-gvxqs" Oct 30 17:00:09.934013 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:09.933967 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-9c1e26-5f76c9f698-gvxqs" event={"ID":"9e7b9aff-3301-4350-b511-262ff5088c3b","Type":"ContainerDied","Data":"3077a6727c59171224522b4dd4c2b03c017d1dd8ce9611468dad0b30654fb136"} Oct 30 17:00:09.934104 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:09.934019 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-9c1e26-5f76c9f698-gvxqs" event={"ID":"9e7b9aff-3301-4350-b511-262ff5088c3b","Type":"ContainerDied","Data":"7eef620115790e7f43c12a59f2a63f0feed9fc3c66d9296257c8f6ca1ae4296f"} Oct 30 17:00:09.934104 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:09.934045 5052 scope.go:122] "RemoveContainer" containerID="3077a6727c59171224522b4dd4c2b03c017d1dd8ce9611468dad0b30654fb136" Oct 30 17:00:09.937112 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:09.937092 5052 scope.go:122] "RemoveContainer" containerID="68eef90bd81dc4cc2a5c432828df371b8adc2361d0b756d83f3c04c5cdf12ba2" Oct 30 17:00:09.937406 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:00:09.937383 5052 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-17f60f-29889660-8tgc2_odh-ai-gateway-infra(e73618ad-d79c-4833-bb07-e22057d3f9f7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-17f60f-29889660-8tgc2" podUID="e73618ad-d79c-4833-bb07-e22057d3f9f7" Oct 30 17:00:09.946027 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:09.946006 5052 scope.go:122] "RemoveContainer" containerID="3077a6727c59171224522b4dd4c2b03c017d1dd8ce9611468dad0b30654fb136" Oct 30 17:00:09.946282 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:00:09.946263 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3077a6727c59171224522b4dd4c2b03c017d1dd8ce9611468dad0b30654fb136\": container with ID starting with 3077a6727c59171224522b4dd4c2b03c017d1dd8ce9611468dad0b30654fb136 not found: ID does not exist" containerID="3077a6727c59171224522b4dd4c2b03c017d1dd8ce9611468dad0b30654fb136" Oct 30 17:00:09.946360 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:09.946288 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3077a6727c59171224522b4dd4c2b03c017d1dd8ce9611468dad0b30654fb136"} err="failed to get container status \"3077a6727c59171224522b4dd4c2b03c017d1dd8ce9611468dad0b30654fb136\": rpc error: code = NotFound desc = could not find container \"3077a6727c59171224522b4dd4c2b03c017d1dd8ce9611468dad0b30654fb136\": container with ID starting with 3077a6727c59171224522b4dd4c2b03c017d1dd8ce9611468dad0b30654fb136 not found: ID does not exist" Oct 30 17:00:10.060915 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:10.060888 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9c1e26-5f76c9f698-gvxqs"] Oct 30 17:00:10.064795 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:10.064770 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9c1e26-5f76c9f698-gvxqs"] Oct 30 17:00:10.601140 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:10.601106 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e7b9aff-3301-4350-b511-262ff5088c3b" path="/var/lib/kubelet/pods/9e7b9aff-3301-4350-b511-262ff5088c3b/volumes" Oct 30 17:00:12.048086 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:12.048001 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9bfcac-86969d48d7-j9dgf" podUID="c979b538-a34c-49dc-8269-98ef5063e17c" containerName="payload-pre-processing" containerID="cri-o://e3c481be9b5f6ee677c001e9f8fb8243806843fd8b61536d9b70617f8175315d" gracePeriod=25 Oct 30 17:00:12.368306 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:12.368230 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9bfcac-86969d48d7-j9dgf" podUID="c979b538-a34c-49dc-8269-98ef5063e17c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:00:14.980279 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:14.978977 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-689899/test-model-689899-kserve-5f8945cf6f-925q2"] Oct 30 17:00:18.590461 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:18.590430 5052 scope.go:122] "RemoveContainer" containerID="68eef90bd81dc4cc2a5c432828df371b8adc2361d0b756d83f3c04c5cdf12ba2" Oct 30 17:00:18.853468 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:00:18.853422 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 17:00:19.590416 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:19.590376 5052 scope.go:122] "RemoveContainer" containerID="373412f835cae64fc1a61b2e5d3fc06de691da6a3e945df9ddde8a5b7a264b39" Oct 30 17:00:19.590632 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:19.590606 5052 scope.go:122] "RemoveContainer" containerID="917f6146bdbff9b93e30df4efaec361f54df903f449fa897ada111d83721cb07" Oct 30 17:00:20.022773 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:20.022738 5052 generic.go:356] "Generic (PLEG): container finished" podID="e73618ad-d79c-4833-bb07-e22057d3f9f7" containerID="5efc3efa40c6a8e8d4018ee4ce52184e3427346dc6007ec3477256d77efd52ba" exitCode=60 Oct 30 17:00:20.022977 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:20.022812 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-17f60f-29889660-8tgc2" event={"ID":"e73618ad-d79c-4833-bb07-e22057d3f9f7","Type":"ContainerDied","Data":"5efc3efa40c6a8e8d4018ee4ce52184e3427346dc6007ec3477256d77efd52ba"} Oct 30 17:00:20.022977 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:20.022887 5052 scope.go:122] "RemoveContainer" containerID="68eef90bd81dc4cc2a5c432828df371b8adc2361d0b756d83f3c04c5cdf12ba2" Oct 30 17:00:20.023706 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:20.023686 5052 scope.go:122] "RemoveContainer" containerID="5efc3efa40c6a8e8d4018ee4ce52184e3427346dc6007ec3477256d77efd52ba" Oct 30 17:00:20.024071 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:00:20.024050 5052 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-17f60f-29889660-8tgc2_odh-ai-gateway-infra(e73618ad-d79c-4833-bb07-e22057d3f9f7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-17f60f-29889660-8tgc2" podUID="e73618ad-d79c-4833-bb07-e22057d3f9f7" Oct 30 17:00:21.033315 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:21.033279 5052 generic.go:356] "Generic (PLEG): container finished" podID="ec93f0a0-6634-4719-8289-d936260d7973" containerID="f40e01db402a8642ecd94d04e97f82374dd0e7998662a4753c87b229ef3854aa" exitCode=60 Oct 30 17:00:21.033798 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:21.033352 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9bfcac-29889660-vn8xn" event={"ID":"ec93f0a0-6634-4719-8289-d936260d7973","Type":"ContainerDied","Data":"f40e01db402a8642ecd94d04e97f82374dd0e7998662a4753c87b229ef3854aa"} Oct 30 17:00:21.033798 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:21.033479 5052 scope.go:122] "RemoveContainer" containerID="917f6146bdbff9b93e30df4efaec361f54df903f449fa897ada111d83721cb07" Oct 30 17:00:21.033982 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:21.033966 5052 scope.go:122] "RemoveContainer" containerID="f40e01db402a8642ecd94d04e97f82374dd0e7998662a4753c87b229ef3854aa" Oct 30 17:00:21.034412 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:00:21.034383 5052 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-9bfcac-29889660-vn8xn_odh-ai-gateway-infra(ec93f0a0-6634-4719-8289-d936260d7973)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9bfcac-29889660-vn8xn" podUID="ec93f0a0-6634-4719-8289-d936260d7973" Oct 30 17:00:21.036013 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:21.035760 5052 generic.go:356] "Generic (PLEG): container finished" podID="1760617a-b6d1-40e5-a3b2-ac46dd644dee" containerID="058eaf9d22b4a2789a3995d5abb591bec868c78d6458a71408da4899e8d64114" exitCode=60 Oct 30 17:00:21.036013 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:21.035797 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f92caf-29889660-44r98" event={"ID":"1760617a-b6d1-40e5-a3b2-ac46dd644dee","Type":"ContainerDied","Data":"058eaf9d22b4a2789a3995d5abb591bec868c78d6458a71408da4899e8d64114"} Oct 30 17:00:21.036302 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:21.036282 5052 scope.go:122] "RemoveContainer" containerID="058eaf9d22b4a2789a3995d5abb591bec868c78d6458a71408da4899e8d64114" Oct 30 17:00:21.036915 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:00:21.036884 5052 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-f92caf-29889660-44r98_odh-ai-gateway-infra(1760617a-b6d1-40e5-a3b2-ac46dd644dee)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f92caf-29889660-44r98" podUID="1760617a-b6d1-40e5-a3b2-ac46dd644dee" Oct 30 17:00:21.048952 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:21.048936 5052 scope.go:122] "RemoveContainer" containerID="373412f835cae64fc1a61b2e5d3fc06de691da6a3e945df9ddde8a5b7a264b39" Oct 30 17:00:21.098133 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:21.098060 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-17f60f-29889660-8tgc2"] Oct 30 17:00:21.459964 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:21.459937 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-17f60f-29889660-8tgc2" Oct 30 17:00:21.610238 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:21.610199 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e73618ad-d79c-4833-bb07-e22057d3f9f7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e73618ad-d79c-4833-bb07-e22057d3f9f7-maas-api-service-ca\") pod \"e73618ad-d79c-4833-bb07-e22057d3f9f7\" (UID: \"e73618ad-d79c-4833-bb07-e22057d3f9f7\") " Oct 30 17:00:21.610410 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:21.610362 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e73618ad-d79c-4833-bb07-e22057d3f9f7-kube-api-access-zttfl\" (UniqueName: \"kubernetes.io/projected/e73618ad-d79c-4833-bb07-e22057d3f9f7-kube-api-access-zttfl\") pod \"e73618ad-d79c-4833-bb07-e22057d3f9f7\" (UID: \"e73618ad-d79c-4833-bb07-e22057d3f9f7\") " Oct 30 17:00:21.610624 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:21.610565 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e73618ad-d79c-4833-bb07-e22057d3f9f7-maas-api-service-ca" pod "e73618ad-d79c-4833-bb07-e22057d3f9f7" (UID: "e73618ad-d79c-4833-bb07-e22057d3f9f7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:00:21.611301 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:21.611270 5052 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e73618ad-d79c-4833-bb07-e22057d3f9f7-maas-api-service-ca\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:00:21.613097 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:21.613075 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e73618ad-d79c-4833-bb07-e22057d3f9f7-kube-api-access-zttfl" pod "e73618ad-d79c-4833-bb07-e22057d3f9f7" (UID: "e73618ad-d79c-4833-bb07-e22057d3f9f7"). InnerVolumeSpecName "kube-api-access-zttfl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:00:21.713181 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:21.713113 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zttfl\" (UniqueName: \"kubernetes.io/projected/e73618ad-d79c-4833-bb07-e22057d3f9f7-kube-api-access-zttfl\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:00:22.050434 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:22.050360 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-17f60f-29889660-8tgc2" Oct 30 17:00:22.050434 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:22.050380 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-17f60f-29889660-8tgc2" event={"ID":"e73618ad-d79c-4833-bb07-e22057d3f9f7","Type":"ContainerDied","Data":"f99554ae0cc259e4f16335f3078151a60b549a7cdd08fff59f0ce95b9e96233d"} Oct 30 17:00:22.050929 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:22.050444 5052 scope.go:122] "RemoveContainer" containerID="5efc3efa40c6a8e8d4018ee4ce52184e3427346dc6007ec3477256d77efd52ba" Oct 30 17:00:22.114358 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:22.114328 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9bfcac-29889660-vn8xn"] Oct 30 17:00:22.122237 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:22.122213 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-17f60f-29889660-8tgc2"] Oct 30 17:00:22.129655 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:22.129625 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-17f60f-29889660-8tgc2"] Oct 30 17:00:22.152480 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:22.152446 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f92caf-29889660-44r98"] Oct 30 17:00:22.368122 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:22.368076 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9bfcac-86969d48d7-j9dgf" podUID="c979b538-a34c-49dc-8269-98ef5063e17c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:00:22.392269 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:22.392249 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9bfcac-29889660-vn8xn" Oct 30 17:00:22.394456 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:22.394434 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f92caf-29889660-44r98" Oct 30 17:00:22.527953 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:22.527907 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1760617a-b6d1-40e5-a3b2-ac46dd644dee-kube-api-access-fdcqq\" (UniqueName: \"kubernetes.io/projected/1760617a-b6d1-40e5-a3b2-ac46dd644dee-kube-api-access-fdcqq\") pod \"1760617a-b6d1-40e5-a3b2-ac46dd644dee\" (UID: \"1760617a-b6d1-40e5-a3b2-ac46dd644dee\") " Oct 30 17:00:22.528158 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:22.528078 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec93f0a0-6634-4719-8289-d936260d7973-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec93f0a0-6634-4719-8289-d936260d7973-maas-api-service-ca\") pod \"ec93f0a0-6634-4719-8289-d936260d7973\" (UID: \"ec93f0a0-6634-4719-8289-d936260d7973\") " Oct 30 17:00:22.528158 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:22.528153 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1760617a-b6d1-40e5-a3b2-ac46dd644dee-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1760617a-b6d1-40e5-a3b2-ac46dd644dee-maas-api-service-ca\") pod \"1760617a-b6d1-40e5-a3b2-ac46dd644dee\" (UID: \"1760617a-b6d1-40e5-a3b2-ac46dd644dee\") " Oct 30 17:00:22.528787 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:22.528235 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec93f0a0-6634-4719-8289-d936260d7973-kube-api-access-jggrx\" (UniqueName: \"kubernetes.io/projected/ec93f0a0-6634-4719-8289-d936260d7973-kube-api-access-jggrx\") pod \"ec93f0a0-6634-4719-8289-d936260d7973\" (UID: \"ec93f0a0-6634-4719-8289-d936260d7973\") " Oct 30 17:00:22.528787 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:22.528490 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec93f0a0-6634-4719-8289-d936260d7973-maas-api-service-ca" pod "ec93f0a0-6634-4719-8289-d936260d7973" (UID: "ec93f0a0-6634-4719-8289-d936260d7973"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:00:22.528944 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:22.528828 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1760617a-b6d1-40e5-a3b2-ac46dd644dee-maas-api-service-ca" pod "1760617a-b6d1-40e5-a3b2-ac46dd644dee" (UID: "1760617a-b6d1-40e5-a3b2-ac46dd644dee"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:00:22.529722 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:22.529695 5052 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec93f0a0-6634-4719-8289-d936260d7973-maas-api-service-ca\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:00:22.529824 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:22.529725 5052 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1760617a-b6d1-40e5-a3b2-ac46dd644dee-maas-api-service-ca\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:00:22.531356 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:22.531294 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1760617a-b6d1-40e5-a3b2-ac46dd644dee-kube-api-access-fdcqq" pod "1760617a-b6d1-40e5-a3b2-ac46dd644dee" (UID: "1760617a-b6d1-40e5-a3b2-ac46dd644dee"). InnerVolumeSpecName "kube-api-access-fdcqq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:00:22.532121 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:22.532063 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec93f0a0-6634-4719-8289-d936260d7973-kube-api-access-jggrx" pod "ec93f0a0-6634-4719-8289-d936260d7973" (UID: "ec93f0a0-6634-4719-8289-d936260d7973"). InnerVolumeSpecName "kube-api-access-jggrx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:00:22.600901 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:22.600842 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e73618ad-d79c-4833-bb07-e22057d3f9f7" path="/var/lib/kubelet/pods/e73618ad-d79c-4833-bb07-e22057d3f9f7/volumes" Oct 30 17:00:22.631164 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:22.631142 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdcqq\" (UniqueName: \"kubernetes.io/projected/1760617a-b6d1-40e5-a3b2-ac46dd644dee-kube-api-access-fdcqq\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:00:22.631164 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:22.631164 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jggrx\" (UniqueName: \"kubernetes.io/projected/ec93f0a0-6634-4719-8289-d936260d7973-kube-api-access-jggrx\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:00:23.065745 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:23.065698 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9bfcac-29889660-vn8xn" event={"ID":"ec93f0a0-6634-4719-8289-d936260d7973","Type":"ContainerDied","Data":"633197bcfaad8283e066fea1cc9c24b237acf42ad861cd7ba4db14e6b7541172"} Oct 30 17:00:23.066222 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:23.065786 5052 scope.go:122] "RemoveContainer" containerID="f40e01db402a8642ecd94d04e97f82374dd0e7998662a4753c87b229ef3854aa" Oct 30 17:00:23.066222 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:23.065708 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9bfcac-29889660-vn8xn" Oct 30 17:00:23.067886 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:23.067867 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f92caf-29889660-44r98" Oct 30 17:00:23.068029 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:23.067912 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f92caf-29889660-44r98" event={"ID":"1760617a-b6d1-40e5-a3b2-ac46dd644dee","Type":"ContainerDied","Data":"21d804a2a7b295d68724f7b592b5bd764cd8b046b6acd37a5e51b9670588d345"} Oct 30 17:00:23.077912 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:23.077892 5052 scope.go:122] "RemoveContainer" containerID="058eaf9d22b4a2789a3995d5abb591bec868c78d6458a71408da4899e8d64114" Oct 30 17:00:23.131145 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:23.131115 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9bfcac-29889660-vn8xn"] Oct 30 17:00:23.143057 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:23.143032 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9bfcac-29889660-vn8xn"] Oct 30 17:00:23.181386 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:23.181356 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f92caf-29889660-44r98"] Oct 30 17:00:23.186910 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:23.186886 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f92caf-29889660-44r98"] Oct 30 17:00:24.532602 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:24.527178 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-661509-openshift-default-b5d9994f-xlcqt"] Oct 30 17:00:24.532602 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:24.528456 5052 cpu_manager.go:414] "containerMap: removing container" podUID="ec93f0a0-6634-4719-8289-d936260d7973" containerName="cleanup" Oct 30 17:00:24.532602 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:24.528479 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec93f0a0-6634-4719-8289-d936260d7973" containerName="cleanup" Oct 30 17:00:24.532602 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:24.528500 5052 cpu_manager.go:414] "containerMap: removing container" podUID="9e7b9aff-3301-4350-b511-262ff5088c3b" containerName="payload-processing" Oct 30 17:00:24.532602 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:24.528511 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e7b9aff-3301-4350-b511-262ff5088c3b" containerName="payload-processing" Oct 30 17:00:24.532602 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:24.528531 5052 cpu_manager.go:414] "containerMap: removing container" podUID="1760617a-b6d1-40e5-a3b2-ac46dd644dee" containerName="cleanup" Oct 30 17:00:24.532602 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:24.528539 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1760617a-b6d1-40e5-a3b2-ac46dd644dee" containerName="cleanup" Oct 30 17:00:24.532602 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:24.528553 5052 cpu_manager.go:414] "containerMap: removing container" podUID="e73618ad-d79c-4833-bb07-e22057d3f9f7" containerName="cleanup" Oct 30 17:00:24.532602 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:24.528560 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e73618ad-d79c-4833-bb07-e22057d3f9f7" containerName="cleanup" Oct 30 17:00:24.532602 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:24.528633 5052 cpu_manager.go:414] "containerMap: removing container" podUID="ec93f0a0-6634-4719-8289-d936260d7973" containerName="cleanup" Oct 30 17:00:24.532602 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:24.528642 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec93f0a0-6634-4719-8289-d936260d7973" containerName="cleanup" Oct 30 17:00:24.532602 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:24.528678 5052 cpu_manager.go:414] "containerMap: removing container" podUID="e73618ad-d79c-4833-bb07-e22057d3f9f7" containerName="cleanup" Oct 30 17:00:24.532602 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:24.528687 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e73618ad-d79c-4833-bb07-e22057d3f9f7" containerName="cleanup" Oct 30 17:00:24.532602 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:24.528707 5052 cpu_manager.go:414] "containerMap: removing container" podUID="e73618ad-d79c-4833-bb07-e22057d3f9f7" containerName="cleanup" Oct 30 17:00:24.532602 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:24.528715 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e73618ad-d79c-4833-bb07-e22057d3f9f7" containerName="cleanup" Oct 30 17:00:24.532602 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:24.528729 5052 cpu_manager.go:414] "containerMap: removing container" podUID="1760617a-b6d1-40e5-a3b2-ac46dd644dee" containerName="cleanup" Oct 30 17:00:24.532602 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:24.528736 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1760617a-b6d1-40e5-a3b2-ac46dd644dee" containerName="cleanup" Oct 30 17:00:24.532602 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:24.528747 5052 cpu_manager.go:414] "containerMap: removing container" podUID="06b94bcd-8c5a-4143-a936-34d62f4189b0" containerName="main" Oct 30 17:00:24.532602 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:24.528755 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06b94bcd-8c5a-4143-a936-34d62f4189b0" containerName="main" Oct 30 17:00:24.532602 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:24.528998 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec93f0a0-6634-4719-8289-d936260d7973" containerName="cleanup" Oct 30 17:00:24.532602 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:24.529017 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e7b9aff-3301-4350-b511-262ff5088c3b" containerName="payload-processing" Oct 30 17:00:24.532602 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:24.529040 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="1760617a-b6d1-40e5-a3b2-ac46dd644dee" containerName="cleanup" Oct 30 17:00:24.532602 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:24.529051 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="e73618ad-d79c-4833-bb07-e22057d3f9f7" containerName="cleanup" Oct 30 17:00:24.532602 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:24.529067 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec93f0a0-6634-4719-8289-d936260d7973" containerName="cleanup" Oct 30 17:00:24.532602 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:24.529084 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec93f0a0-6634-4719-8289-d936260d7973" containerName="cleanup" Oct 30 17:00:24.532602 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:24.529099 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="e73618ad-d79c-4833-bb07-e22057d3f9f7" containerName="cleanup" Oct 30 17:00:24.532602 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:24.529113 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="1760617a-b6d1-40e5-a3b2-ac46dd644dee" containerName="cleanup" Oct 30 17:00:24.532602 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:24.529122 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="06b94bcd-8c5a-4143-a936-34d62f4189b0" containerName="main" Oct 30 17:00:24.537877 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:24.537481 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-661509-openshift-default-b5d9994f-xlcqt" Oct 30 17:00:24.553508 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:24.552083 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-661509-openshift-default-b5d9994f-xlcqt"] Oct 30 17:00:24.558109 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:24.558086 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-661509-openshift-default-dockercfg-fzxw6\"" Oct 30 17:00:24.613668 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:24.611356 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1760617a-b6d1-40e5-a3b2-ac46dd644dee" path="/var/lib/kubelet/pods/1760617a-b6d1-40e5-a3b2-ac46dd644dee/volumes" Oct 30 17:00:24.613668 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:24.612223 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec93f0a0-6634-4719-8289-d936260d7973" path="/var/lib/kubelet/pods/ec93f0a0-6634-4719-8289-d936260d7973/volumes" Oct 30 17:00:24.667241 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:24.667213 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6rrm\" (UniqueName: \"kubernetes.io/projected/24bfdf71-2df9-478f-a6af-eca34fc212ca-kube-api-access-s6rrm\") pod \"e2e-shared-b-w3-661509-openshift-default-b5d9994f-xlcqt\" (UID: \"24bfdf71-2df9-478f-a6af-eca34fc212ca\") " pod="openshift-ingress/e2e-shared-b-w3-661509-openshift-default-b5d9994f-xlcqt" Oct 30 17:00:24.667389 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:24.667349 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24bfdf71-2df9-478f-a6af-eca34fc212ca-istio-envoy\") pod \"e2e-shared-b-w3-661509-openshift-default-b5d9994f-xlcqt\" (UID: \"24bfdf71-2df9-478f-a6af-eca34fc212ca\") " pod="openshift-ingress/e2e-shared-b-w3-661509-openshift-default-b5d9994f-xlcqt" Oct 30 17:00:24.667389 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:24.667384 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24bfdf71-2df9-478f-a6af-eca34fc212ca-credential-socket\") pod \"e2e-shared-b-w3-661509-openshift-default-b5d9994f-xlcqt\" (UID: \"24bfdf71-2df9-478f-a6af-eca34fc212ca\") " pod="openshift-ingress/e2e-shared-b-w3-661509-openshift-default-b5d9994f-xlcqt" Oct 30 17:00:24.667482 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:24.667431 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24bfdf71-2df9-478f-a6af-eca34fc212ca-istio-data\") pod \"e2e-shared-b-w3-661509-openshift-default-b5d9994f-xlcqt\" (UID: \"24bfdf71-2df9-478f-a6af-eca34fc212ca\") " pod="openshift-ingress/e2e-shared-b-w3-661509-openshift-default-b5d9994f-xlcqt" Oct 30 17:00:24.667592 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:24.667568 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24bfdf71-2df9-478f-a6af-eca34fc212ca-workload-certs\") pod \"e2e-shared-b-w3-661509-openshift-default-b5d9994f-xlcqt\" (UID: \"24bfdf71-2df9-478f-a6af-eca34fc212ca\") " pod="openshift-ingress/e2e-shared-b-w3-661509-openshift-default-b5d9994f-xlcqt" Oct 30 17:00:24.667638 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:24.667600 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24bfdf71-2df9-478f-a6af-eca34fc212ca-istiod-ca-cert\") pod \"e2e-shared-b-w3-661509-openshift-default-b5d9994f-xlcqt\" (UID: \"24bfdf71-2df9-478f-a6af-eca34fc212ca\") " pod="openshift-ingress/e2e-shared-b-w3-661509-openshift-default-b5d9994f-xlcqt" Oct 30 17:00:24.667638 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:24.667620 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24bfdf71-2df9-478f-a6af-eca34fc212ca-workload-socket\") pod \"e2e-shared-b-w3-661509-openshift-default-b5d9994f-xlcqt\" (UID: \"24bfdf71-2df9-478f-a6af-eca34fc212ca\") " pod="openshift-ingress/e2e-shared-b-w3-661509-openshift-default-b5d9994f-xlcqt" Oct 30 17:00:24.667728 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:24.667710 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24bfdf71-2df9-478f-a6af-eca34fc212ca-istio-podinfo\") pod \"e2e-shared-b-w3-661509-openshift-default-b5d9994f-xlcqt\" (UID: \"24bfdf71-2df9-478f-a6af-eca34fc212ca\") " pod="openshift-ingress/e2e-shared-b-w3-661509-openshift-default-b5d9994f-xlcqt" Oct 30 17:00:24.667853 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:24.667838 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24bfdf71-2df9-478f-a6af-eca34fc212ca-istio-token\") pod \"e2e-shared-b-w3-661509-openshift-default-b5d9994f-xlcqt\" (UID: \"24bfdf71-2df9-478f-a6af-eca34fc212ca\") " pod="openshift-ingress/e2e-shared-b-w3-661509-openshift-default-b5d9994f-xlcqt" Oct 30 17:00:24.768651 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:24.768619 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24bfdf71-2df9-478f-a6af-eca34fc212ca-workload-certs\") pod \"e2e-shared-b-w3-661509-openshift-default-b5d9994f-xlcqt\" (UID: \"24bfdf71-2df9-478f-a6af-eca34fc212ca\") " pod="openshift-ingress/e2e-shared-b-w3-661509-openshift-default-b5d9994f-xlcqt" Oct 30 17:00:24.768651 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:24.768654 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24bfdf71-2df9-478f-a6af-eca34fc212ca-istiod-ca-cert\") pod \"e2e-shared-b-w3-661509-openshift-default-b5d9994f-xlcqt\" (UID: \"24bfdf71-2df9-478f-a6af-eca34fc212ca\") " pod="openshift-ingress/e2e-shared-b-w3-661509-openshift-default-b5d9994f-xlcqt" Oct 30 17:00:24.768885 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:24.768675 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24bfdf71-2df9-478f-a6af-eca34fc212ca-workload-socket\") pod \"e2e-shared-b-w3-661509-openshift-default-b5d9994f-xlcqt\" (UID: \"24bfdf71-2df9-478f-a6af-eca34fc212ca\") " pod="openshift-ingress/e2e-shared-b-w3-661509-openshift-default-b5d9994f-xlcqt" Oct 30 17:00:24.768885 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:24.768713 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24bfdf71-2df9-478f-a6af-eca34fc212ca-istio-podinfo\") pod \"e2e-shared-b-w3-661509-openshift-default-b5d9994f-xlcqt\" (UID: \"24bfdf71-2df9-478f-a6af-eca34fc212ca\") " pod="openshift-ingress/e2e-shared-b-w3-661509-openshift-default-b5d9994f-xlcqt" Oct 30 17:00:24.768885 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:24.768838 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24bfdf71-2df9-478f-a6af-eca34fc212ca-istio-token\") pod \"e2e-shared-b-w3-661509-openshift-default-b5d9994f-xlcqt\" (UID: \"24bfdf71-2df9-478f-a6af-eca34fc212ca\") " pod="openshift-ingress/e2e-shared-b-w3-661509-openshift-default-b5d9994f-xlcqt" Oct 30 17:00:24.768885 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:24.768880 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6rrm\" (UniqueName: \"kubernetes.io/projected/24bfdf71-2df9-478f-a6af-eca34fc212ca-kube-api-access-s6rrm\") pod \"e2e-shared-b-w3-661509-openshift-default-b5d9994f-xlcqt\" (UID: \"24bfdf71-2df9-478f-a6af-eca34fc212ca\") " pod="openshift-ingress/e2e-shared-b-w3-661509-openshift-default-b5d9994f-xlcqt" Oct 30 17:00:24.769114 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:24.769025 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24bfdf71-2df9-478f-a6af-eca34fc212ca-istio-envoy\") pod \"e2e-shared-b-w3-661509-openshift-default-b5d9994f-xlcqt\" (UID: \"24bfdf71-2df9-478f-a6af-eca34fc212ca\") " pod="openshift-ingress/e2e-shared-b-w3-661509-openshift-default-b5d9994f-xlcqt" Oct 30 17:00:24.769114 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:24.769062 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24bfdf71-2df9-478f-a6af-eca34fc212ca-workload-certs\") pod \"e2e-shared-b-w3-661509-openshift-default-b5d9994f-xlcqt\" (UID: \"24bfdf71-2df9-478f-a6af-eca34fc212ca\") " pod="openshift-ingress/e2e-shared-b-w3-661509-openshift-default-b5d9994f-xlcqt" Oct 30 17:00:24.769225 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:24.769126 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24bfdf71-2df9-478f-a6af-eca34fc212ca-workload-socket\") pod \"e2e-shared-b-w3-661509-openshift-default-b5d9994f-xlcqt\" (UID: \"24bfdf71-2df9-478f-a6af-eca34fc212ca\") " pod="openshift-ingress/e2e-shared-b-w3-661509-openshift-default-b5d9994f-xlcqt" Oct 30 17:00:24.769225 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:24.769064 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24bfdf71-2df9-478f-a6af-eca34fc212ca-credential-socket\") pod \"e2e-shared-b-w3-661509-openshift-default-b5d9994f-xlcqt\" (UID: \"24bfdf71-2df9-478f-a6af-eca34fc212ca\") " pod="openshift-ingress/e2e-shared-b-w3-661509-openshift-default-b5d9994f-xlcqt" Oct 30 17:00:24.769329 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:24.769229 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24bfdf71-2df9-478f-a6af-eca34fc212ca-istio-data\") pod \"e2e-shared-b-w3-661509-openshift-default-b5d9994f-xlcqt\" (UID: \"24bfdf71-2df9-478f-a6af-eca34fc212ca\") " pod="openshift-ingress/e2e-shared-b-w3-661509-openshift-default-b5d9994f-xlcqt" Oct 30 17:00:24.769329 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:24.769295 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24bfdf71-2df9-478f-a6af-eca34fc212ca-credential-socket\") pod \"e2e-shared-b-w3-661509-openshift-default-b5d9994f-xlcqt\" (UID: \"24bfdf71-2df9-478f-a6af-eca34fc212ca\") " pod="openshift-ingress/e2e-shared-b-w3-661509-openshift-default-b5d9994f-xlcqt" Oct 30 17:00:24.769436 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:24.769406 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24bfdf71-2df9-478f-a6af-eca34fc212ca-istiod-ca-cert\") pod \"e2e-shared-b-w3-661509-openshift-default-b5d9994f-xlcqt\" (UID: \"24bfdf71-2df9-478f-a6af-eca34fc212ca\") " pod="openshift-ingress/e2e-shared-b-w3-661509-openshift-default-b5d9994f-xlcqt" Oct 30 17:00:24.769594 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:24.769562 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24bfdf71-2df9-478f-a6af-eca34fc212ca-istio-data\") pod \"e2e-shared-b-w3-661509-openshift-default-b5d9994f-xlcqt\" (UID: \"24bfdf71-2df9-478f-a6af-eca34fc212ca\") " pod="openshift-ingress/e2e-shared-b-w3-661509-openshift-default-b5d9994f-xlcqt" Oct 30 17:00:24.771482 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:24.771461 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24bfdf71-2df9-478f-a6af-eca34fc212ca-istio-envoy\") pod \"e2e-shared-b-w3-661509-openshift-default-b5d9994f-xlcqt\" (UID: \"24bfdf71-2df9-478f-a6af-eca34fc212ca\") " pod="openshift-ingress/e2e-shared-b-w3-661509-openshift-default-b5d9994f-xlcqt" Oct 30 17:00:24.771796 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:24.771770 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24bfdf71-2df9-478f-a6af-eca34fc212ca-istio-podinfo\") pod \"e2e-shared-b-w3-661509-openshift-default-b5d9994f-xlcqt\" (UID: \"24bfdf71-2df9-478f-a6af-eca34fc212ca\") " pod="openshift-ingress/e2e-shared-b-w3-661509-openshift-default-b5d9994f-xlcqt" Oct 30 17:00:24.796802 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:24.796750 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24bfdf71-2df9-478f-a6af-eca34fc212ca-istio-token\") pod \"e2e-shared-b-w3-661509-openshift-default-b5d9994f-xlcqt\" (UID: \"24bfdf71-2df9-478f-a6af-eca34fc212ca\") " pod="openshift-ingress/e2e-shared-b-w3-661509-openshift-default-b5d9994f-xlcqt" Oct 30 17:00:24.798502 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:24.798387 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6rrm\" (UniqueName: \"kubernetes.io/projected/24bfdf71-2df9-478f-a6af-eca34fc212ca-kube-api-access-s6rrm\") pod \"e2e-shared-b-w3-661509-openshift-default-b5d9994f-xlcqt\" (UID: \"24bfdf71-2df9-478f-a6af-eca34fc212ca\") " pod="openshift-ingress/e2e-shared-b-w3-661509-openshift-default-b5d9994f-xlcqt" Oct 30 17:00:24.860344 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:24.860311 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-661509-openshift-default-b5d9994f-xlcqt" Oct 30 17:00:25.189700 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:25.189667 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-661509-openshift-default-b5d9994f-xlcqt"] Oct 30 17:00:25.190202 ip-10-0-131-136 kubenswrapper[5052]: W1030 17:00:25.190175 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24bfdf71_2df9_478f_a6af_eca34fc212ca.slice/crio-1b3b8638e44dc25bbbdf1205f0a29d87edc5427971ae4764fae74b4f4b1c3843 WatchSource:0}: Error finding container 1b3b8638e44dc25bbbdf1205f0a29d87edc5427971ae4764fae74b4f4b1c3843: Status 404 returned error can't find the container with id 1b3b8638e44dc25bbbdf1205f0a29d87edc5427971ae4764fae74b4f4b1c3843 Oct 30 17:00:25.192953 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:25.192917 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:00:25.193041 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:25.192983 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:00:25.193041 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:25.193029 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:00:25.677960 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:25.677924 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-17f60f-5fbf6b59c9-vvk5s"] Oct 30 17:00:26.096170 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:26.096089 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-661509-openshift-default-b5d9994f-xlcqt" event={"ID":"24bfdf71-2df9-478f-a6af-eca34fc212ca","Type":"ContainerStarted","Data":"3795d7d00d009b87c7b9d83bb82a861bfd1cce311005cc3e5dde058a74e4c64a"} Oct 30 17:00:26.096170 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:26.096149 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-661509-openshift-default-b5d9994f-xlcqt" event={"ID":"24bfdf71-2df9-478f-a6af-eca34fc212ca","Type":"ContainerStarted","Data":"1b3b8638e44dc25bbbdf1205f0a29d87edc5427971ae4764fae74b4f4b1c3843"} Oct 30 17:00:26.145608 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:26.144596 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-661509-openshift-default-b5d9994f-xlcqt" podStartSLOduration=2.144558325 podStartE2EDuration="2.144558325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:00:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:00:26.144109133 +0000 UTC m=+2381.671444358" watchObservedRunningTime="2026-10-30 17:00:26.144558325 +0000 UTC m=+2381.671893547" Oct 30 17:00:26.860919 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:26.860879 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-661509-openshift-default-b5d9994f-xlcqt" Oct 30 17:00:26.862216 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:26.862192 5052 patch_prober.go:29] interesting pod/e2e-shared-b-w3-661509-openshift-default-b5d9994f-xlcqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Oct 30 17:00:26.862287 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:26.862241 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-661509-openshift-default-b5d9994f-xlcqt" podUID="24bfdf71-2df9-478f-a6af-eca34fc212ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Oct 30 17:00:27.861177 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:27.861141 5052 patch_prober.go:29] interesting pod/e2e-shared-b-w3-661509-openshift-default-b5d9994f-xlcqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Oct 30 17:00:27.861566 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:27.861193 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-661509-openshift-default-b5d9994f-xlcqt" podUID="24bfdf71-2df9-478f-a6af-eca34fc212ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Oct 30 17:00:28.017656 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:28.017625 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-fxsm2"] Oct 30 17:00:28.018463 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:28.018446 5052 cpu_manager.go:414] "containerMap: removing container" podUID="ec93f0a0-6634-4719-8289-d936260d7973" containerName="cleanup" Oct 30 17:00:28.018521 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:28.018464 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec93f0a0-6634-4719-8289-d936260d7973" containerName="cleanup" Oct 30 17:00:28.018521 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:28.018488 5052 cpu_manager.go:414] "containerMap: removing container" podUID="1760617a-b6d1-40e5-a3b2-ac46dd644dee" containerName="cleanup" Oct 30 17:00:28.018521 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:28.018493 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1760617a-b6d1-40e5-a3b2-ac46dd644dee" containerName="cleanup" Oct 30 17:00:28.018681 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:28.018658 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="1760617a-b6d1-40e5-a3b2-ac46dd644dee" containerName="cleanup" Oct 30 17:00:28.018717 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:28.018687 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="e73618ad-d79c-4833-bb07-e22057d3f9f7" containerName="cleanup" Oct 30 17:00:28.022650 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:28.022614 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-fxsm2" Oct 30 17:00:28.056465 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:28.056378 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-7c4b84-openshift-default-dockercfg-5sg2r\"" Oct 30 17:00:28.060295 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:28.060270 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-fxsm2"] Oct 30 17:00:28.075899 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:28.075869 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0607bb9-cd62-4149-a429-4051390ad2a1-istio-token\") pod \"e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-fxsm2\" (UID: \"f0607bb9-cd62-4149-a429-4051390ad2a1\") " pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-fxsm2" Oct 30 17:00:28.076092 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:28.076071 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0607bb9-cd62-4149-a429-4051390ad2a1-istio-data\") pod \"e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-fxsm2\" (UID: \"f0607bb9-cd62-4149-a429-4051390ad2a1\") " pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-fxsm2" Oct 30 17:00:28.076292 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:28.076269 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0607bb9-cd62-4149-a429-4051390ad2a1-credential-socket\") pod \"e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-fxsm2\" (UID: \"f0607bb9-cd62-4149-a429-4051390ad2a1\") " pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-fxsm2" Oct 30 17:00:28.076480 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:28.076458 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0607bb9-cd62-4149-a429-4051390ad2a1-istiod-ca-cert\") pod \"e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-fxsm2\" (UID: \"f0607bb9-cd62-4149-a429-4051390ad2a1\") " pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-fxsm2" Oct 30 17:00:28.076746 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:28.076725 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0607bb9-cd62-4149-a429-4051390ad2a1-istio-podinfo\") pod \"e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-fxsm2\" (UID: \"f0607bb9-cd62-4149-a429-4051390ad2a1\") " pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-fxsm2" Oct 30 17:00:28.076883 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:28.076867 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0607bb9-cd62-4149-a429-4051390ad2a1-istio-envoy\") pod \"e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-fxsm2\" (UID: \"f0607bb9-cd62-4149-a429-4051390ad2a1\") " pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-fxsm2" Oct 30 17:00:28.076996 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:28.076980 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0607bb9-cd62-4149-a429-4051390ad2a1-workload-certs\") pod \"e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-fxsm2\" (UID: \"f0607bb9-cd62-4149-a429-4051390ad2a1\") " pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-fxsm2" Oct 30 17:00:28.077254 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:28.077235 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0607bb9-cd62-4149-a429-4051390ad2a1-workload-socket\") pod \"e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-fxsm2\" (UID: \"f0607bb9-cd62-4149-a429-4051390ad2a1\") " pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-fxsm2" Oct 30 17:00:28.077388 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:28.077371 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tghhx\" (UniqueName: \"kubernetes.io/projected/f0607bb9-cd62-4149-a429-4051390ad2a1-kube-api-access-tghhx\") pod \"e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-fxsm2\" (UID: \"f0607bb9-cd62-4149-a429-4051390ad2a1\") " pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-fxsm2" Oct 30 17:00:28.179153 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:28.179117 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0607bb9-cd62-4149-a429-4051390ad2a1-istio-podinfo\") pod \"e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-fxsm2\" (UID: \"f0607bb9-cd62-4149-a429-4051390ad2a1\") " pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-fxsm2" Oct 30 17:00:28.179153 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:28.179157 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0607bb9-cd62-4149-a429-4051390ad2a1-istio-envoy\") pod \"e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-fxsm2\" (UID: \"f0607bb9-cd62-4149-a429-4051390ad2a1\") " pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-fxsm2" Oct 30 17:00:28.179411 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:28.179176 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0607bb9-cd62-4149-a429-4051390ad2a1-workload-certs\") pod \"e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-fxsm2\" (UID: \"f0607bb9-cd62-4149-a429-4051390ad2a1\") " pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-fxsm2" Oct 30 17:00:28.179411 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:28.179271 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0607bb9-cd62-4149-a429-4051390ad2a1-workload-socket\") pod \"e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-fxsm2\" (UID: \"f0607bb9-cd62-4149-a429-4051390ad2a1\") " pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-fxsm2" Oct 30 17:00:28.179411 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:28.179305 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tghhx\" (UniqueName: \"kubernetes.io/projected/f0607bb9-cd62-4149-a429-4051390ad2a1-kube-api-access-tghhx\") pod \"e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-fxsm2\" (UID: \"f0607bb9-cd62-4149-a429-4051390ad2a1\") " pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-fxsm2" Oct 30 17:00:28.179632 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:28.179532 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0607bb9-cd62-4149-a429-4051390ad2a1-istio-token\") pod \"e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-fxsm2\" (UID: \"f0607bb9-cd62-4149-a429-4051390ad2a1\") " pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-fxsm2" Oct 30 17:00:28.179632 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:28.179595 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0607bb9-cd62-4149-a429-4051390ad2a1-istio-data\") pod \"e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-fxsm2\" (UID: \"f0607bb9-cd62-4149-a429-4051390ad2a1\") " pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-fxsm2" Oct 30 17:00:28.179769 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:28.179668 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0607bb9-cd62-4149-a429-4051390ad2a1-workload-certs\") pod \"e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-fxsm2\" (UID: \"f0607bb9-cd62-4149-a429-4051390ad2a1\") " pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-fxsm2" Oct 30 17:00:28.179769 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:28.179678 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0607bb9-cd62-4149-a429-4051390ad2a1-workload-socket\") pod \"e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-fxsm2\" (UID: \"f0607bb9-cd62-4149-a429-4051390ad2a1\") " pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-fxsm2" Oct 30 17:00:28.179888 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:28.179833 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0607bb9-cd62-4149-a429-4051390ad2a1-credential-socket\") pod \"e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-fxsm2\" (UID: \"f0607bb9-cd62-4149-a429-4051390ad2a1\") " pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-fxsm2" Oct 30 17:00:28.179888 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:28.179876 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0607bb9-cd62-4149-a429-4051390ad2a1-istiod-ca-cert\") pod \"e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-fxsm2\" (UID: \"f0607bb9-cd62-4149-a429-4051390ad2a1\") " pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-fxsm2" Oct 30 17:00:28.180109 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:28.180084 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0607bb9-cd62-4149-a429-4051390ad2a1-credential-socket\") pod \"e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-fxsm2\" (UID: \"f0607bb9-cd62-4149-a429-4051390ad2a1\") " pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-fxsm2" Oct 30 17:00:28.180257 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:28.180150 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0607bb9-cd62-4149-a429-4051390ad2a1-istio-data\") pod \"e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-fxsm2\" (UID: \"f0607bb9-cd62-4149-a429-4051390ad2a1\") " pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-fxsm2" Oct 30 17:00:28.180452 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:28.180428 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0607bb9-cd62-4149-a429-4051390ad2a1-istiod-ca-cert\") pod \"e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-fxsm2\" (UID: \"f0607bb9-cd62-4149-a429-4051390ad2a1\") " pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-fxsm2" Oct 30 17:00:28.182116 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:28.182095 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0607bb9-cd62-4149-a429-4051390ad2a1-istio-envoy\") pod \"e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-fxsm2\" (UID: \"f0607bb9-cd62-4149-a429-4051390ad2a1\") " pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-fxsm2" Oct 30 17:00:28.182434 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:28.182413 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0607bb9-cd62-4149-a429-4051390ad2a1-istio-podinfo\") pod \"e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-fxsm2\" (UID: \"f0607bb9-cd62-4149-a429-4051390ad2a1\") " pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-fxsm2" Oct 30 17:00:28.205118 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:28.205083 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tghhx\" (UniqueName: \"kubernetes.io/projected/f0607bb9-cd62-4149-a429-4051390ad2a1-kube-api-access-tghhx\") pod \"e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-fxsm2\" (UID: \"f0607bb9-cd62-4149-a429-4051390ad2a1\") " pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-fxsm2" Oct 30 17:00:28.207924 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:28.207897 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0607bb9-cd62-4149-a429-4051390ad2a1-istio-token\") pod \"e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-fxsm2\" (UID: \"f0607bb9-cd62-4149-a429-4051390ad2a1\") " pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-fxsm2" Oct 30 17:00:28.365022 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:28.364989 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-fxsm2" Oct 30 17:00:28.712463 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:28.712432 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-fxsm2"] Oct 30 17:00:28.712875 ip-10-0-131-136 kubenswrapper[5052]: W1030 17:00:28.712844 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0607bb9_cd62_4149_a429_4051390ad2a1.slice/crio-b3bd99bdb6981a31749ea961acce1cf46dd8b641345a978e92484df179ae739e WatchSource:0}: Error finding container b3bd99bdb6981a31749ea961acce1cf46dd8b641345a978e92484df179ae739e: Status 404 returned error can't find the container with id b3bd99bdb6981a31749ea961acce1cf46dd8b641345a978e92484df179ae739e Oct 30 17:00:28.720030 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:28.719991 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:00:28.720153 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:28.720079 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:00:28.720153 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:28.720127 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:00:28.861445 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:28.861411 5052 patch_prober.go:29] interesting pod/e2e-shared-b-w3-661509-openshift-default-b5d9994f-xlcqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Oct 30 17:00:28.861853 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:28.861468 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-661509-openshift-default-b5d9994f-xlcqt" podUID="24bfdf71-2df9-478f-a6af-eca34fc212ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Oct 30 17:00:29.124050 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:29.123962 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-fxsm2" event={"ID":"f0607bb9-cd62-4149-a429-4051390ad2a1","Type":"ContainerStarted","Data":"fefb376a529947de9ff314f3583738dc55d221e956a1f3663fbce2a5ccc42760"} Oct 30 17:00:29.124050 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:29.124017 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-fxsm2" event={"ID":"f0607bb9-cd62-4149-a429-4051390ad2a1","Type":"ContainerStarted","Data":"b3bd99bdb6981a31749ea961acce1cf46dd8b641345a978e92484df179ae739e"} Oct 30 17:00:29.175278 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:29.175221 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-fxsm2" podStartSLOduration=2.175206247 podStartE2EDuration="2.175206247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:00:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:00:29.171260722 +0000 UTC m=+2384.698595941" watchObservedRunningTime="2026-10-30 17:00:29.175206247 +0000 UTC m=+2384.702541466" Oct 30 17:00:29.365703 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:29.365674 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-fxsm2" Oct 30 17:00:29.367360 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:29.367332 5052 patch_prober.go:29] interesting pod/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-fxsm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 30 17:00:29.367526 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:29.367507 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-fxsm2" podUID="f0607bb9-cd62-4149-a429-4051390ad2a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 30 17:00:29.865609 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:29.864125 5052 patch_prober.go:29] interesting pod/e2e-shared-b-w3-661509-openshift-default-b5d9994f-xlcqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Oct 30 17:00:29.865609 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:29.864193 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-661509-openshift-default-b5d9994f-xlcqt" podUID="24bfdf71-2df9-478f-a6af-eca34fc212ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Oct 30 17:00:30.003141 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:30.003088 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-689899/test-model-689899-kserve-5f8945cf6f-925q2" podUID="cbc3abfb-2bcf-4133-9a49-65b29f06f12a" containerName="main" containerID="cri-o://d14238fc4fd13d9b07f20e44075b2921d50f02e6db0c76c7c95d7796177f42be" gracePeriod=45 Oct 30 17:00:30.081755 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:30.075026 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-689899/test-model-689899-kserve-5f8945cf6f-925q2" podUID="cbc3abfb-2bcf-4133-9a49-65b29f06f12a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.170:8000/ready\": dial tcp 10.132.0.170:8000: connect: connection refused" Oct 30 17:00:30.193181 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:30.193136 5052 generic.go:356] "Generic (PLEG): container finished" podID="cbc3abfb-2bcf-4133-9a49-65b29f06f12a" containerID="d14238fc4fd13d9b07f20e44075b2921d50f02e6db0c76c7c95d7796177f42be" exitCode=0 Oct 30 17:00:30.193595 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:30.193550 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-689899/test-model-689899-kserve-5f8945cf6f-925q2" event={"ID":"cbc3abfb-2bcf-4133-9a49-65b29f06f12a","Type":"ContainerDied","Data":"d14238fc4fd13d9b07f20e44075b2921d50f02e6db0c76c7c95d7796177f42be"} Oct 30 17:00:30.384420 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:30.384325 5052 patch_prober.go:29] interesting pod/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-fxsm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 30 17:00:30.384731 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:30.384698 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-fxsm2" podUID="f0607bb9-cd62-4149-a429-4051390ad2a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 30 17:00:30.501390 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:30.501361 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-689899/test-model-689899-kserve-5f8945cf6f-925q2" Oct 30 17:00:30.549606 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:30.548650 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbc3abfb-2bcf-4133-9a49-65b29f06f12a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbc3abfb-2bcf-4133-9a49-65b29f06f12a-dshm\") pod \"cbc3abfb-2bcf-4133-9a49-65b29f06f12a\" (UID: \"cbc3abfb-2bcf-4133-9a49-65b29f06f12a\") " Oct 30 17:00:30.587675 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:30.572690 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cbc3abfb-2bcf-4133-9a49-65b29f06f12a-tls-certs\" (UniqueName: \"kubernetes.io/secret/cbc3abfb-2bcf-4133-9a49-65b29f06f12a-tls-certs\") pod \"cbc3abfb-2bcf-4133-9a49-65b29f06f12a\" (UID: \"cbc3abfb-2bcf-4133-9a49-65b29f06f12a\") " Oct 30 17:00:30.587675 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:30.572751 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbc3abfb-2bcf-4133-9a49-65b29f06f12a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbc3abfb-2bcf-4133-9a49-65b29f06f12a-tmp-dir\") pod \"cbc3abfb-2bcf-4133-9a49-65b29f06f12a\" (UID: \"cbc3abfb-2bcf-4133-9a49-65b29f06f12a\") " Oct 30 17:00:30.587675 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:30.572892 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbc3abfb-2bcf-4133-9a49-65b29f06f12a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbc3abfb-2bcf-4133-9a49-65b29f06f12a-model-cache\") pod \"cbc3abfb-2bcf-4133-9a49-65b29f06f12a\" (UID: \"cbc3abfb-2bcf-4133-9a49-65b29f06f12a\") " Oct 30 17:00:30.587675 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:30.572935 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbc3abfb-2bcf-4133-9a49-65b29f06f12a-home\" (UniqueName: \"kubernetes.io/empty-dir/cbc3abfb-2bcf-4133-9a49-65b29f06f12a-home\") pod \"cbc3abfb-2bcf-4133-9a49-65b29f06f12a\" (UID: \"cbc3abfb-2bcf-4133-9a49-65b29f06f12a\") " Oct 30 17:00:30.587675 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:30.573142 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbc3abfb-2bcf-4133-9a49-65b29f06f12a-kube-api-access-gzgv2\" (UniqueName: \"kubernetes.io/projected/cbc3abfb-2bcf-4133-9a49-65b29f06f12a-kube-api-access-gzgv2\") pod \"cbc3abfb-2bcf-4133-9a49-65b29f06f12a\" (UID: \"cbc3abfb-2bcf-4133-9a49-65b29f06f12a\") " Oct 30 17:00:30.587675 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:30.586012 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbc3abfb-2bcf-4133-9a49-65b29f06f12a-tmp-dir" pod "cbc3abfb-2bcf-4133-9a49-65b29f06f12a" (UID: "cbc3abfb-2bcf-4133-9a49-65b29f06f12a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:00:30.587675 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:30.586619 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbc3abfb-2bcf-4133-9a49-65b29f06f12a-home" pod "cbc3abfb-2bcf-4133-9a49-65b29f06f12a" (UID: "cbc3abfb-2bcf-4133-9a49-65b29f06f12a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:00:30.587675 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:30.586719 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbc3abfb-2bcf-4133-9a49-65b29f06f12a-model-cache" pod "cbc3abfb-2bcf-4133-9a49-65b29f06f12a" (UID: "cbc3abfb-2bcf-4133-9a49-65b29f06f12a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:00:30.603539 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:30.602934 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbc3abfb-2bcf-4133-9a49-65b29f06f12a-kube-api-access-gzgv2" pod "cbc3abfb-2bcf-4133-9a49-65b29f06f12a" (UID: "cbc3abfb-2bcf-4133-9a49-65b29f06f12a"). InnerVolumeSpecName "kube-api-access-gzgv2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:00:30.604197 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:30.604160 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbc3abfb-2bcf-4133-9a49-65b29f06f12a-tls-certs" pod "cbc3abfb-2bcf-4133-9a49-65b29f06f12a" (UID: "cbc3abfb-2bcf-4133-9a49-65b29f06f12a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:00:30.605342 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:30.605319 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbc3abfb-2bcf-4133-9a49-65b29f06f12a-dshm" pod "cbc3abfb-2bcf-4133-9a49-65b29f06f12a" (UID: "cbc3abfb-2bcf-4133-9a49-65b29f06f12a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:00:30.678410 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:30.678347 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-17f60f-5fbf6b59c9-vvk5s" podUID="141c2e28-1616-44fb-8e0b-275678b894b0" containerName="payload-pre-processing" containerID="cri-o://a3802794e114812ee6b30885078114d3cc867ea9cf7d613aa2c08b2c678c5155" gracePeriod=25 Oct 30 17:00:30.682223 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:30.682148 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzgv2\" (UniqueName: \"kubernetes.io/projected/cbc3abfb-2bcf-4133-9a49-65b29f06f12a-kube-api-access-gzgv2\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:00:30.682411 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:30.682382 5052 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbc3abfb-2bcf-4133-9a49-65b29f06f12a-dshm\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:00:30.682542 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:30.682524 5052 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbc3abfb-2bcf-4133-9a49-65b29f06f12a-tls-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:00:30.682658 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:30.682646 5052 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbc3abfb-2bcf-4133-9a49-65b29f06f12a-tmp-dir\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:00:30.682756 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:30.682745 5052 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbc3abfb-2bcf-4133-9a49-65b29f06f12a-model-cache\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:00:30.682849 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:30.682839 5052 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cbc3abfb-2bcf-4133-9a49-65b29f06f12a-home\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:00:30.862615 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:30.860978 5052 patch_prober.go:29] interesting pod/e2e-shared-b-w3-661509-openshift-default-b5d9994f-xlcqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Oct 30 17:00:30.862615 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:30.861036 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-661509-openshift-default-b5d9994f-xlcqt" podUID="24bfdf71-2df9-478f-a6af-eca34fc212ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Oct 30 17:00:31.211818 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:31.211782 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-689899/test-model-689899-kserve-5f8945cf6f-925q2" Oct 30 17:00:31.212289 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:31.211810 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-689899/test-model-689899-kserve-5f8945cf6f-925q2" event={"ID":"cbc3abfb-2bcf-4133-9a49-65b29f06f12a","Type":"ContainerDied","Data":"644c5299ef2c48cab18c5d998f498f5a5426e7455875d9fcc28c7474ea079183"} Oct 30 17:00:31.212289 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:31.211999 5052 scope.go:122] "RemoveContainer" containerID="d14238fc4fd13d9b07f20e44075b2921d50f02e6db0c76c7c95d7796177f42be" Oct 30 17:00:31.291543 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:31.291510 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-689899/test-model-689899-kserve-5f8945cf6f-925q2"] Oct 30 17:00:31.303655 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:31.303627 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-689899/test-model-689899-kserve-5f8945cf6f-925q2"] Oct 30 17:00:31.370613 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:31.369989 5052 patch_prober.go:29] interesting pod/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-fxsm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 30 17:00:31.370613 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:31.370048 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-fxsm2" podUID="f0607bb9-cd62-4149-a429-4051390ad2a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 30 17:00:31.860966 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:31.860928 5052 patch_prober.go:29] interesting pod/e2e-shared-b-w3-661509-openshift-default-b5d9994f-xlcqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Oct 30 17:00:31.861156 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:31.860992 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-661509-openshift-default-b5d9994f-xlcqt" podUID="24bfdf71-2df9-478f-a6af-eca34fc212ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Oct 30 17:00:32.285770 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:32.285728 5052 generic.go:356] "Generic (PLEG): container finished" podID="c979b538-a34c-49dc-8269-98ef5063e17c" containerID="e3c481be9b5f6ee677c001e9f8fb8243806843fd8b61536d9b70617f8175315d" exitCode=0 Oct 30 17:00:32.286352 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:32.285890 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9bfcac-86969d48d7-j9dgf" event={"ID":"c979b538-a34c-49dc-8269-98ef5063e17c","Type":"ContainerDied","Data":"e3c481be9b5f6ee677c001e9f8fb8243806843fd8b61536d9b70617f8175315d"} Oct 30 17:00:32.379104 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:32.378042 5052 patch_prober.go:29] interesting pod/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-fxsm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 30 17:00:32.379104 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:32.378102 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-fxsm2" podUID="f0607bb9-cd62-4149-a429-4051390ad2a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 30 17:00:32.530044 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:32.528761 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9bfcac-86969d48d7-j9dgf" Oct 30 17:00:32.608003 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:32.607957 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbc3abfb-2bcf-4133-9a49-65b29f06f12a" path="/var/lib/kubelet/pods/cbc3abfb-2bcf-4133-9a49-65b29f06f12a/volumes" Oct 30 17:00:32.674060 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:32.674026 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c979b538-a34c-49dc-8269-98ef5063e17c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c979b538-a34c-49dc-8269-98ef5063e17c-plugins-config-volume\") pod \"c979b538-a34c-49dc-8269-98ef5063e17c\" (UID: \"c979b538-a34c-49dc-8269-98ef5063e17c\") " Oct 30 17:00:32.674278 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:32.674123 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c979b538-a34c-49dc-8269-98ef5063e17c-kube-api-access-8vpjq\" (UniqueName: \"kubernetes.io/projected/c979b538-a34c-49dc-8269-98ef5063e17c-kube-api-access-8vpjq\") pod \"c979b538-a34c-49dc-8269-98ef5063e17c\" (UID: \"c979b538-a34c-49dc-8269-98ef5063e17c\") " Oct 30 17:00:32.674466 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:32.674437 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c979b538-a34c-49dc-8269-98ef5063e17c-plugins-config-volume" pod "c979b538-a34c-49dc-8269-98ef5063e17c" (UID: "c979b538-a34c-49dc-8269-98ef5063e17c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:00:32.675051 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:32.675030 5052 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c979b538-a34c-49dc-8269-98ef5063e17c-plugins-config-volume\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:00:32.687811 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:32.687785 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c979b538-a34c-49dc-8269-98ef5063e17c-kube-api-access-8vpjq" pod "c979b538-a34c-49dc-8269-98ef5063e17c" (UID: "c979b538-a34c-49dc-8269-98ef5063e17c"). InnerVolumeSpecName "kube-api-access-8vpjq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:00:32.777066 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:32.777028 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vpjq\" (UniqueName: \"kubernetes.io/projected/c979b538-a34c-49dc-8269-98ef5063e17c-kube-api-access-8vpjq\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:00:32.865980 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:32.865947 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-661509-openshift-default-b5d9994f-xlcqt" Oct 30 17:00:32.866425 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:32.866402 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-661509-openshift-default-b5d9994f-xlcqt" Oct 30 17:00:32.867287 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:32.867262 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-661509-openshift-default-b5d9994f-xlcqt" Oct 30 17:00:33.360656 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:33.352671 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9bfcac-86969d48d7-j9dgf" event={"ID":"c979b538-a34c-49dc-8269-98ef5063e17c","Type":"ContainerDied","Data":"9efaee2db2339f8ed315537a2b1144a0f398eb7ce8be4a16aab93ded06e5688f"} Oct 30 17:00:33.360656 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:33.352761 5052 scope.go:122] "RemoveContainer" containerID="e3c481be9b5f6ee677c001e9f8fb8243806843fd8b61536d9b70617f8175315d" Oct 30 17:00:33.360656 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:33.353199 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9bfcac-86969d48d7-j9dgf" Oct 30 17:00:33.388201 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:33.378144 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9bfcac-86969d48d7-j9dgf" podUID="c979b538-a34c-49dc-8269-98ef5063e17c" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.150:50052\" within 1s: context deadline exceeded" Oct 30 17:00:33.388201 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:33.378373 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9bfcac-86969d48d7-j9dgf" Oct 30 17:00:33.388201 ip-10-0-131-136 kubenswrapper[5052]: W1030 17:00:33.378741 5052 logging.go:55] [core] [Channel #4406 SubChannel #4407]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.150:50052", ServerName: "10.132.0.150:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.150:50052: operation was canceled" Oct 30 17:00:33.411980 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:33.401039 5052 patch_prober.go:29] interesting pod/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-fxsm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 30 17:00:33.411980 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:33.401092 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-fxsm2" podUID="f0607bb9-cd62-4149-a429-4051390ad2a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 30 17:00:33.541607 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:33.530317 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9bfcac-86969d48d7-j9dgf"] Oct 30 17:00:33.547246 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:33.547214 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9bfcac-86969d48d7-j9dgf"] Oct 30 17:00:34.381771 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:34.380702 5052 patch_prober.go:29] interesting pod/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-fxsm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 30 17:00:34.381771 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:34.380770 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-fxsm2" podUID="f0607bb9-cd62-4149-a429-4051390ad2a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 30 17:00:34.413303 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:34.413219 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-17f60f-5fbf6b59c9-vvk5s" podUID="141c2e28-1616-44fb-8e0b-275678b894b0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:00:35.092512 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:35.092468 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c979b538-a34c-49dc-8269-98ef5063e17c" path="/var/lib/kubelet/pods/c979b538-a34c-49dc-8269-98ef5063e17c/volumes" Oct 30 17:00:35.365561 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:35.365470 5052 patch_prober.go:29] interesting pod/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-fxsm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 30 17:00:35.365867 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:35.365837 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-fxsm2" podUID="f0607bb9-cd62-4149-a429-4051390ad2a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 30 17:00:36.366356 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:36.366316 5052 patch_prober.go:29] interesting pod/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-fxsm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 30 17:00:36.366843 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:36.366379 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-fxsm2" podUID="f0607bb9-cd62-4149-a429-4051390ad2a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 30 17:00:37.410178 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:37.408914 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-fxsm2" Oct 30 17:00:37.410178 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:37.409875 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-fxsm2" Oct 30 17:00:37.411974 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:37.411933 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-fxsm2" Oct 30 17:00:39.722700 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:39.707244 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9bfcac-6bb4c44994-95lp4"] Oct 30 17:00:39.722700 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:39.707495 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9bfcac-6bb4c44994-95lp4" podUID="c5898d00-6f84-40ef-890f-a476895b9690" containerName="maas-api" containerID="cri-o://2f92983ee0985e683975e6113285cb593685a38d154642a58581a572efc9c673" gracePeriod=30 Oct 30 17:00:40.386606 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:40.386471 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9bfcac-6bb4c44994-95lp4" Oct 30 17:00:40.487157 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:40.487124 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5898d00-6f84-40ef-890f-a476895b9690-kube-api-access-7q468\" (UniqueName: \"kubernetes.io/projected/c5898d00-6f84-40ef-890f-a476895b9690-kube-api-access-7q468\") pod \"c5898d00-6f84-40ef-890f-a476895b9690\" (UID: \"c5898d00-6f84-40ef-890f-a476895b9690\") " Oct 30 17:00:40.487320 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:40.487178 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5898d00-6f84-40ef-890f-a476895b9690-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5898d00-6f84-40ef-890f-a476895b9690-metrics-certs\") pod \"c5898d00-6f84-40ef-890f-a476895b9690\" (UID: \"c5898d00-6f84-40ef-890f-a476895b9690\") " Oct 30 17:00:40.487453 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:40.487435 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5898d00-6f84-40ef-890f-a476895b9690-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5898d00-6f84-40ef-890f-a476895b9690-maas-api-tls\") pod \"c5898d00-6f84-40ef-890f-a476895b9690\" (UID: \"c5898d00-6f84-40ef-890f-a476895b9690\") " Oct 30 17:00:40.487498 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:40.487482 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5898d00-6f84-40ef-890f-a476895b9690-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5898d00-6f84-40ef-890f-a476895b9690-openshift-service-ca\") pod \"c5898d00-6f84-40ef-890f-a476895b9690\" (UID: \"c5898d00-6f84-40ef-890f-a476895b9690\") " Oct 30 17:00:40.487922 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:40.487896 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5898d00-6f84-40ef-890f-a476895b9690-openshift-service-ca" pod "c5898d00-6f84-40ef-890f-a476895b9690" (UID: "c5898d00-6f84-40ef-890f-a476895b9690"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:00:40.488232 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:40.488209 5052 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5898d00-6f84-40ef-890f-a476895b9690-openshift-service-ca\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:00:40.489564 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:40.489541 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5898d00-6f84-40ef-890f-a476895b9690-metrics-certs" pod "c5898d00-6f84-40ef-890f-a476895b9690" (UID: "c5898d00-6f84-40ef-890f-a476895b9690"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:00:40.489696 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:40.489593 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5898d00-6f84-40ef-890f-a476895b9690-kube-api-access-7q468" pod "c5898d00-6f84-40ef-890f-a476895b9690" (UID: "c5898d00-6f84-40ef-890f-a476895b9690"). InnerVolumeSpecName "kube-api-access-7q468". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:00:40.489951 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:40.489923 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5898d00-6f84-40ef-890f-a476895b9690-maas-api-tls" pod "c5898d00-6f84-40ef-890f-a476895b9690" (UID: "c5898d00-6f84-40ef-890f-a476895b9690"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:00:40.504954 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:40.504921 5052 generic.go:356] "Generic (PLEG): container finished" podID="c5898d00-6f84-40ef-890f-a476895b9690" containerID="2f92983ee0985e683975e6113285cb593685a38d154642a58581a572efc9c673" exitCode=0 Oct 30 17:00:40.505159 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:40.504964 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9bfcac-6bb4c44994-95lp4" Oct 30 17:00:40.505159 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:40.504994 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9bfcac-6bb4c44994-95lp4" event={"ID":"c5898d00-6f84-40ef-890f-a476895b9690","Type":"ContainerDied","Data":"2f92983ee0985e683975e6113285cb593685a38d154642a58581a572efc9c673"} Oct 30 17:00:40.505159 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:40.505087 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9bfcac-6bb4c44994-95lp4" event={"ID":"c5898d00-6f84-40ef-890f-a476895b9690","Type":"ContainerDied","Data":"3e21c21e037e4a1f26036b1a37b21d3bee2ee0960b4fc94e36f83a0b30388851"} Oct 30 17:00:40.505159 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:40.505115 5052 scope.go:122] "RemoveContainer" containerID="2f92983ee0985e683975e6113285cb593685a38d154642a58581a572efc9c673" Oct 30 17:00:40.522748 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:40.522645 5052 scope.go:122] "RemoveContainer" containerID="2f92983ee0985e683975e6113285cb593685a38d154642a58581a572efc9c673" Oct 30 17:00:40.523030 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:00:40.522964 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f92983ee0985e683975e6113285cb593685a38d154642a58581a572efc9c673\": container with ID starting with 2f92983ee0985e683975e6113285cb593685a38d154642a58581a572efc9c673 not found: ID does not exist" containerID="2f92983ee0985e683975e6113285cb593685a38d154642a58581a572efc9c673" Oct 30 17:00:40.523030 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:40.523001 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f92983ee0985e683975e6113285cb593685a38d154642a58581a572efc9c673"} err="failed to get container status \"2f92983ee0985e683975e6113285cb593685a38d154642a58581a572efc9c673\": rpc error: code = NotFound desc = could not find container \"2f92983ee0985e683975e6113285cb593685a38d154642a58581a572efc9c673\": container with ID starting with 2f92983ee0985e683975e6113285cb593685a38d154642a58581a572efc9c673 not found: ID does not exist" Oct 30 17:00:40.580680 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:40.580602 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9bfcac-6bb4c44994-95lp4"] Oct 30 17:00:40.589711 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:40.589680 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7q468\" (UniqueName: \"kubernetes.io/projected/c5898d00-6f84-40ef-890f-a476895b9690-kube-api-access-7q468\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:00:40.589711 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:40.589707 5052 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5898d00-6f84-40ef-890f-a476895b9690-metrics-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:00:40.589927 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:40.589737 5052 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5898d00-6f84-40ef-890f-a476895b9690-maas-api-tls\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:00:40.602414 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:40.602384 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9bfcac-6bb4c44994-95lp4"] Oct 30 17:00:41.915481 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:41.915449 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-ea77d8/test-model-ea77d8-kserve-7f798c774c-tbxxl"] Oct 30 17:00:42.674650 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:42.666087 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5898d00-6f84-40ef-890f-a476895b9690" path="/var/lib/kubelet/pods/c5898d00-6f84-40ef-890f-a476895b9690/volumes" Oct 30 17:00:44.329752 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:44.329711 5052 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-h5v86 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:00:44.330242 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:44.329780 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-h5v86" podUID="029cd866-9c51-45c8-9ca4-4c16578e12c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:00:44.400810 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:44.400769 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-17f60f-5fbf6b59c9-vvk5s" podUID="141c2e28-1616-44fb-8e0b-275678b894b0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:00:45.533453 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:45.533424 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4wrf8_4f0f46f4-5cda-4d67-b6f3-47df06eb844c/console-operator/2.log" Oct 30 17:00:45.534555 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:45.534534 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4wrf8_4f0f46f4-5cda-4d67-b6f3-47df06eb844c/console-operator/2.log" Oct 30 17:00:46.645516 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:46.645433 5052 generic.go:356] "Generic (PLEG): container finished" podID="029cd866-9c51-45c8-9ca4-4c16578e12c9" containerID="e3c6f7aeea864977164007842e03e5ab7d8c32ea1a7105943fd2a500d7d7fafd" exitCode=0 Oct 30 17:00:46.645516 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:46.645503 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-h5v86" event={"ID":"029cd866-9c51-45c8-9ca4-4c16578e12c9","Type":"ContainerDied","Data":"e3c6f7aeea864977164007842e03e5ab7d8c32ea1a7105943fd2a500d7d7fafd"} Oct 30 17:00:46.645997 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:46.645944 5052 scope.go:122] "RemoveContainer" containerID="e3c6f7aeea864977164007842e03e5ab7d8c32ea1a7105943fd2a500d7d7fafd" Oct 30 17:00:46.647735 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:46.647709 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:00:46.647834 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:46.647778 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:00:46.647834 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:46.647812 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:00:47.657957 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:47.657911 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-h5v86" event={"ID":"029cd866-9c51-45c8-9ca4-4c16578e12c9","Type":"ContainerStarted","Data":"73c8b70b6d202f94562b22334a53f566f4884cd9d971ef55d243aed2ffc55b36"} Oct 30 17:00:48.491127 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:48.491088 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-h5v86" Oct 30 17:00:48.491916 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:48.491896 5052 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-h5v86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Oct 30 17:00:48.491985 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:48.491940 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-h5v86" podUID="029cd866-9c51-45c8-9ca4-4c16578e12c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Oct 30 17:00:48.869016 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:00:48.868915 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 17:00:49.487920 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:49.487815 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-9bfcac-openshift-default-676454d6b5-wh8mx"] Oct 30 17:00:49.489876 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:49.489819 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-9bfcac-openshift-default-676454d6b5-wh8mx" podUID="500d3721-818e-4ad7-b087-6dfd1c1d6368" containerName="istio-proxy" containerID="cri-o://099c1797818bf395a22a04383d5563beb9fff9169289d825621339785aa8ceec" gracePeriod=30 Oct 30 17:00:49.493174 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:49.493149 5052 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-h5v86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Oct 30 17:00:49.493383 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:49.493343 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-h5v86" podUID="029cd866-9c51-45c8-9ca4-4c16578e12c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Oct 30 17:00:50.491119 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:50.491072 5052 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-h5v86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Oct 30 17:00:50.491689 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:50.491159 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-h5v86" podUID="029cd866-9c51-45c8-9ca4-4c16578e12c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Oct 30 17:00:51.056879 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:51.056807 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-17f60f-5fbf6b59c9-vvk5s" Oct 30 17:00:51.167421 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:51.167378 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/141c2e28-1616-44fb-8e0b-275678b894b0-kube-api-access-4snk8\" (UniqueName: \"kubernetes.io/projected/141c2e28-1616-44fb-8e0b-275678b894b0-kube-api-access-4snk8\") pod \"141c2e28-1616-44fb-8e0b-275678b894b0\" (UID: \"141c2e28-1616-44fb-8e0b-275678b894b0\") " Oct 30 17:00:51.167655 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:51.167560 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/141c2e28-1616-44fb-8e0b-275678b894b0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/141c2e28-1616-44fb-8e0b-275678b894b0-plugins-config-volume\") pod \"141c2e28-1616-44fb-8e0b-275678b894b0\" (UID: \"141c2e28-1616-44fb-8e0b-275678b894b0\") " Oct 30 17:00:51.168034 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:51.168005 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/141c2e28-1616-44fb-8e0b-275678b894b0-plugins-config-volume" pod "141c2e28-1616-44fb-8e0b-275678b894b0" (UID: "141c2e28-1616-44fb-8e0b-275678b894b0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:00:51.168272 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:51.168253 5052 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/141c2e28-1616-44fb-8e0b-275678b894b0-plugins-config-volume\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:00:51.169760 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:51.169732 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/141c2e28-1616-44fb-8e0b-275678b894b0-kube-api-access-4snk8" pod "141c2e28-1616-44fb-8e0b-275678b894b0" (UID: "141c2e28-1616-44fb-8e0b-275678b894b0"). InnerVolumeSpecName "kube-api-access-4snk8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:00:51.269612 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:51.269561 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4snk8\" (UniqueName: \"kubernetes.io/projected/141c2e28-1616-44fb-8e0b-275678b894b0-kube-api-access-4snk8\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:00:51.498612 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:51.496957 5052 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-h5v86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Oct 30 17:00:51.498612 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:51.497018 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-h5v86" podUID="029cd866-9c51-45c8-9ca4-4c16578e12c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Oct 30 17:00:51.733610 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:51.728243 5052 generic.go:356] "Generic (PLEG): container finished" podID="141c2e28-1616-44fb-8e0b-275678b894b0" containerID="a3802794e114812ee6b30885078114d3cc867ea9cf7d613aa2c08b2c678c5155" exitCode=0 Oct 30 17:00:51.733909 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:51.733880 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-17f60f-5fbf6b59c9-vvk5s" event={"ID":"141c2e28-1616-44fb-8e0b-275678b894b0","Type":"ContainerDied","Data":"a3802794e114812ee6b30885078114d3cc867ea9cf7d613aa2c08b2c678c5155"} Oct 30 17:00:51.734059 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:51.734039 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-17f60f-5fbf6b59c9-vvk5s" event={"ID":"141c2e28-1616-44fb-8e0b-275678b894b0","Type":"ContainerDied","Data":"d348ccc842cf3206d8ce42f6283a6d19759a460cf0c4736e43c9a65b24705190"} Oct 30 17:00:51.734187 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:51.734165 5052 scope.go:122] "RemoveContainer" containerID="a3802794e114812ee6b30885078114d3cc867ea9cf7d613aa2c08b2c678c5155" Oct 30 17:00:51.734437 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:51.734421 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-17f60f-5fbf6b59c9-vvk5s" Oct 30 17:00:51.779805 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:51.775757 5052 scope.go:122] "RemoveContainer" containerID="a3802794e114812ee6b30885078114d3cc867ea9cf7d613aa2c08b2c678c5155" Oct 30 17:00:51.779805 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:00:51.776018 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3802794e114812ee6b30885078114d3cc867ea9cf7d613aa2c08b2c678c5155\": container with ID starting with a3802794e114812ee6b30885078114d3cc867ea9cf7d613aa2c08b2c678c5155 not found: ID does not exist" containerID="a3802794e114812ee6b30885078114d3cc867ea9cf7d613aa2c08b2c678c5155" Oct 30 17:00:51.779805 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:51.776044 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3802794e114812ee6b30885078114d3cc867ea9cf7d613aa2c08b2c678c5155"} err="failed to get container status \"a3802794e114812ee6b30885078114d3cc867ea9cf7d613aa2c08b2c678c5155\": rpc error: code = NotFound desc = could not find container \"a3802794e114812ee6b30885078114d3cc867ea9cf7d613aa2c08b2c678c5155\": container with ID starting with a3802794e114812ee6b30885078114d3cc867ea9cf7d613aa2c08b2c678c5155 not found: ID does not exist" Oct 30 17:00:51.810940 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:51.810910 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-17f60f-5fbf6b59c9-vvk5s"] Oct 30 17:00:51.817876 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:51.817852 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-17f60f-5fbf6b59c9-vvk5s"] Oct 30 17:00:51.878060 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:51.878020 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-c4z7h"] Oct 30 17:00:51.878432 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:51.878367 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-c4z7h" podUID="dd7dff09-720a-411c-9c69-97bc38ac2178" containerName="istio-proxy" containerID="cri-o://bd59317436f748983cc0bd1aaa1c88e2b5ae8b0d58b5b4797864cbb0566fa3ce" gracePeriod=30 Oct 30 17:00:52.492163 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:52.491904 5052 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-h5v86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Oct 30 17:00:52.492163 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:52.491961 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-h5v86" podUID="029cd866-9c51-45c8-9ca4-4c16578e12c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Oct 30 17:00:52.600855 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:52.600810 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="141c2e28-1616-44fb-8e0b-275678b894b0" path="/var/lib/kubelet/pods/141c2e28-1616-44fb-8e0b-275678b894b0/volumes" Oct 30 17:00:53.491055 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:53.491021 5052 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-h5v86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Oct 30 17:00:53.491258 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:53.491078 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-h5v86" podUID="029cd866-9c51-45c8-9ca4-4c16578e12c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Oct 30 17:00:54.495956 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:54.495924 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-h5v86" Oct 30 17:00:54.496444 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:54.496430 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-h5v86" Oct 30 17:00:54.497202 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:54.497182 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-h5v86" Oct 30 17:00:54.511015 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:54.510984 5052 patch_prober.go:29] interesting pod/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-c4z7h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 17:00:54.511115 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:54.511036 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-c4z7h" podUID="dd7dff09-720a-411c-9c69-97bc38ac2178" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 17:00:54.783393 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:54.783296 5052 generic.go:356] "Generic (PLEG): container finished" podID="500d3721-818e-4ad7-b087-6dfd1c1d6368" containerID="099c1797818bf395a22a04383d5563beb9fff9169289d825621339785aa8ceec" exitCode=0 Oct 30 17:00:54.783393 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:54.783372 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9bfcac-openshift-default-676454d6b5-wh8mx" event={"ID":"500d3721-818e-4ad7-b087-6dfd1c1d6368","Type":"ContainerDied","Data":"099c1797818bf395a22a04383d5563beb9fff9169289d825621339785aa8ceec"} Oct 30 17:00:54.832066 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:54.832031 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9bfcac-openshift-default-676454d6b5-wh8mx" Oct 30 17:00:54.993216 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:54.993168 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/500d3721-818e-4ad7-b087-6dfd1c1d6368-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/500d3721-818e-4ad7-b087-6dfd1c1d6368-istio-data\") pod \"500d3721-818e-4ad7-b087-6dfd1c1d6368\" (UID: \"500d3721-818e-4ad7-b087-6dfd1c1d6368\") " Oct 30 17:00:54.993430 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:54.993231 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/500d3721-818e-4ad7-b087-6dfd1c1d6368-kube-api-access-p6hj4\" (UniqueName: \"kubernetes.io/projected/500d3721-818e-4ad7-b087-6dfd1c1d6368-kube-api-access-p6hj4\") pod \"500d3721-818e-4ad7-b087-6dfd1c1d6368\" (UID: \"500d3721-818e-4ad7-b087-6dfd1c1d6368\") " Oct 30 17:00:54.993430 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:54.993270 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/500d3721-818e-4ad7-b087-6dfd1c1d6368-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/500d3721-818e-4ad7-b087-6dfd1c1d6368-istio-envoy\") pod \"500d3721-818e-4ad7-b087-6dfd1c1d6368\" (UID: \"500d3721-818e-4ad7-b087-6dfd1c1d6368\") " Oct 30 17:00:54.993430 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:54.993389 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/500d3721-818e-4ad7-b087-6dfd1c1d6368-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/500d3721-818e-4ad7-b087-6dfd1c1d6368-istiod-ca-cert\") pod \"500d3721-818e-4ad7-b087-6dfd1c1d6368\" (UID: \"500d3721-818e-4ad7-b087-6dfd1c1d6368\") " Oct 30 17:00:54.993666 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:54.993438 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/500d3721-818e-4ad7-b087-6dfd1c1d6368-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/500d3721-818e-4ad7-b087-6dfd1c1d6368-istio-podinfo\") pod \"500d3721-818e-4ad7-b087-6dfd1c1d6368\" (UID: \"500d3721-818e-4ad7-b087-6dfd1c1d6368\") " Oct 30 17:00:54.993666 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:54.993443 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/500d3721-818e-4ad7-b087-6dfd1c1d6368-istio-data" pod "500d3721-818e-4ad7-b087-6dfd1c1d6368" (UID: "500d3721-818e-4ad7-b087-6dfd1c1d6368"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:00:54.993666 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:54.993455 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/500d3721-818e-4ad7-b087-6dfd1c1d6368-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/500d3721-818e-4ad7-b087-6dfd1c1d6368-credential-socket\") pod \"500d3721-818e-4ad7-b087-6dfd1c1d6368\" (UID: \"500d3721-818e-4ad7-b087-6dfd1c1d6368\") " Oct 30 17:00:54.993666 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:54.993523 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/500d3721-818e-4ad7-b087-6dfd1c1d6368-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/500d3721-818e-4ad7-b087-6dfd1c1d6368-workload-certs\") pod \"500d3721-818e-4ad7-b087-6dfd1c1d6368\" (UID: \"500d3721-818e-4ad7-b087-6dfd1c1d6368\") " Oct 30 17:00:54.993894 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:54.993725 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/500d3721-818e-4ad7-b087-6dfd1c1d6368-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/500d3721-818e-4ad7-b087-6dfd1c1d6368-workload-socket\") pod \"500d3721-818e-4ad7-b087-6dfd1c1d6368\" (UID: \"500d3721-818e-4ad7-b087-6dfd1c1d6368\") " Oct 30 17:00:54.993894 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:54.993751 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/500d3721-818e-4ad7-b087-6dfd1c1d6368-credential-socket" pod "500d3721-818e-4ad7-b087-6dfd1c1d6368" (UID: "500d3721-818e-4ad7-b087-6dfd1c1d6368"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:00:54.993894 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:54.993833 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/500d3721-818e-4ad7-b087-6dfd1c1d6368-workload-certs" pod "500d3721-818e-4ad7-b087-6dfd1c1d6368" (UID: "500d3721-818e-4ad7-b087-6dfd1c1d6368"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:00:54.994052 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:54.993925 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/500d3721-818e-4ad7-b087-6dfd1c1d6368-istio-token\" (UniqueName: \"kubernetes.io/projected/500d3721-818e-4ad7-b087-6dfd1c1d6368-istio-token\") pod \"500d3721-818e-4ad7-b087-6dfd1c1d6368\" (UID: \"500d3721-818e-4ad7-b087-6dfd1c1d6368\") " Oct 30 17:00:54.994052 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:54.993924 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/500d3721-818e-4ad7-b087-6dfd1c1d6368-istiod-ca-cert" pod "500d3721-818e-4ad7-b087-6dfd1c1d6368" (UID: "500d3721-818e-4ad7-b087-6dfd1c1d6368"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:00:54.994172 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:54.994016 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/500d3721-818e-4ad7-b087-6dfd1c1d6368-workload-socket" pod "500d3721-818e-4ad7-b087-6dfd1c1d6368" (UID: "500d3721-818e-4ad7-b087-6dfd1c1d6368"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:00:54.994658 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:54.994635 5052 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/500d3721-818e-4ad7-b087-6dfd1c1d6368-credential-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:00:54.994658 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:54.994655 5052 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/500d3721-818e-4ad7-b087-6dfd1c1d6368-workload-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:00:54.994850 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:54.994664 5052 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/500d3721-818e-4ad7-b087-6dfd1c1d6368-workload-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:00:54.994850 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:54.994672 5052 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/500d3721-818e-4ad7-b087-6dfd1c1d6368-istio-data\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:00:54.994850 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:54.994681 5052 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/500d3721-818e-4ad7-b087-6dfd1c1d6368-istiod-ca-cert\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:00:54.995741 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:54.995717 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/500d3721-818e-4ad7-b087-6dfd1c1d6368-istio-podinfo" pod "500d3721-818e-4ad7-b087-6dfd1c1d6368" (UID: "500d3721-818e-4ad7-b087-6dfd1c1d6368"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:00:54.995828 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:54.995779 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/500d3721-818e-4ad7-b087-6dfd1c1d6368-istio-envoy" pod "500d3721-818e-4ad7-b087-6dfd1c1d6368" (UID: "500d3721-818e-4ad7-b087-6dfd1c1d6368"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:00:54.996054 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:54.996039 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/500d3721-818e-4ad7-b087-6dfd1c1d6368-kube-api-access-p6hj4" pod "500d3721-818e-4ad7-b087-6dfd1c1d6368" (UID: "500d3721-818e-4ad7-b087-6dfd1c1d6368"). InnerVolumeSpecName "kube-api-access-p6hj4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:00:54.996139 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:54.996125 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/500d3721-818e-4ad7-b087-6dfd1c1d6368-istio-token" pod "500d3721-818e-4ad7-b087-6dfd1c1d6368" (UID: "500d3721-818e-4ad7-b087-6dfd1c1d6368"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:00:55.096431 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:55.096326 5052 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/500d3721-818e-4ad7-b087-6dfd1c1d6368-istio-podinfo\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:00:55.096431 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:55.096370 5052 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/500d3721-818e-4ad7-b087-6dfd1c1d6368-istio-token\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:00:55.096431 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:55.096386 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6hj4\" (UniqueName: \"kubernetes.io/projected/500d3721-818e-4ad7-b087-6dfd1c1d6368-kube-api-access-p6hj4\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:00:55.096431 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:55.096399 5052 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/500d3721-818e-4ad7-b087-6dfd1c1d6368-istio-envoy\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:00:55.830603 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:55.828506 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9bfcac-openshift-default-676454d6b5-wh8mx" Oct 30 17:00:55.836534 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:55.834767 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9bfcac-openshift-default-676454d6b5-wh8mx" event={"ID":"500d3721-818e-4ad7-b087-6dfd1c1d6368","Type":"ContainerDied","Data":"6cbba401ecfe585bbcd13ab20b150469c626ae20c3c0d6b8090b507e2c05200b"} Oct 30 17:00:55.836534 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:55.834841 5052 scope.go:122] "RemoveContainer" containerID="099c1797818bf395a22a04383d5563beb9fff9169289d825621339785aa8ceec" Oct 30 17:00:55.906602 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:55.906557 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-9bfcac-openshift-default-676454d6b5-wh8mx"] Oct 30 17:00:55.916818 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:55.916791 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-9bfcac-openshift-default-676454d6b5-wh8mx"] Oct 30 17:00:56.600482 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:56.600452 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="500d3721-818e-4ad7-b087-6dfd1c1d6368" path="/var/lib/kubelet/pods/500d3721-818e-4ad7-b087-6dfd1c1d6368/volumes" Oct 30 17:00:56.944783 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:56.944740 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-ea77d8/test-model-ea77d8-kserve-7f798c774c-tbxxl" podUID="67ba1b17-147c-4b27-b2bd-9660840a9778" containerName="main" containerID="cri-o://ded388af59dacf976347a1cdb510b8d63807af86594aea32f40a2e10be67f2e0" gracePeriod=45 Oct 30 17:00:57.584979 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:57.584948 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-c4z7h" Oct 30 17:00:57.614616 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:57.614575 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ea77d8/test-model-ea77d8-kserve-7f798c774c-tbxxl" Oct 30 17:00:57.734971 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:57.734895 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd7dff09-720a-411c-9c69-97bc38ac2178-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd7dff09-720a-411c-9c69-97bc38ac2178-workload-certs\") pod \"dd7dff09-720a-411c-9c69-97bc38ac2178\" (UID: \"dd7dff09-720a-411c-9c69-97bc38ac2178\") " Oct 30 17:00:57.734971 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:57.734957 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67ba1b17-147c-4b27-b2bd-9660840a9778-kube-api-access-vsb74\" (UniqueName: \"kubernetes.io/projected/67ba1b17-147c-4b27-b2bd-9660840a9778-kube-api-access-vsb74\") pod \"67ba1b17-147c-4b27-b2bd-9660840a9778\" (UID: \"67ba1b17-147c-4b27-b2bd-9660840a9778\") " Oct 30 17:00:57.735202 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:57.734998 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67ba1b17-147c-4b27-b2bd-9660840a9778-home\" (UniqueName: \"kubernetes.io/empty-dir/67ba1b17-147c-4b27-b2bd-9660840a9778-home\") pod \"67ba1b17-147c-4b27-b2bd-9660840a9778\" (UID: \"67ba1b17-147c-4b27-b2bd-9660840a9778\") " Oct 30 17:00:57.735202 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:57.735058 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd7dff09-720a-411c-9c69-97bc38ac2178-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd7dff09-720a-411c-9c69-97bc38ac2178-istio-envoy\") pod \"dd7dff09-720a-411c-9c69-97bc38ac2178\" (UID: \"dd7dff09-720a-411c-9c69-97bc38ac2178\") " Oct 30 17:00:57.735202 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:57.735115 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd7dff09-720a-411c-9c69-97bc38ac2178-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd7dff09-720a-411c-9c69-97bc38ac2178-istiod-ca-cert\") pod \"dd7dff09-720a-411c-9c69-97bc38ac2178\" (UID: \"dd7dff09-720a-411c-9c69-97bc38ac2178\") " Oct 30 17:00:57.735202 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:57.735186 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd7dff09-720a-411c-9c69-97bc38ac2178-istio-token\" (UniqueName: \"kubernetes.io/projected/dd7dff09-720a-411c-9c69-97bc38ac2178-istio-token\") pod \"dd7dff09-720a-411c-9c69-97bc38ac2178\" (UID: \"dd7dff09-720a-411c-9c69-97bc38ac2178\") " Oct 30 17:00:57.735439 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:57.735248 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd7dff09-720a-411c-9c69-97bc38ac2178-workload-certs" pod "dd7dff09-720a-411c-9c69-97bc38ac2178" (UID: "dd7dff09-720a-411c-9c69-97bc38ac2178"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:00:57.735439 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:57.735260 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd7dff09-720a-411c-9c69-97bc38ac2178-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd7dff09-720a-411c-9c69-97bc38ac2178-istio-data\") pod \"dd7dff09-720a-411c-9c69-97bc38ac2178\" (UID: \"dd7dff09-720a-411c-9c69-97bc38ac2178\") " Oct 30 17:00:57.735439 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:57.735294 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd7dff09-720a-411c-9c69-97bc38ac2178-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd7dff09-720a-411c-9c69-97bc38ac2178-workload-socket\") pod \"dd7dff09-720a-411c-9c69-97bc38ac2178\" (UID: \"dd7dff09-720a-411c-9c69-97bc38ac2178\") " Oct 30 17:00:57.735439 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:57.735363 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67ba1b17-147c-4b27-b2bd-9660840a9778-tls-certs\" (UniqueName: \"kubernetes.io/secret/67ba1b17-147c-4b27-b2bd-9660840a9778-tls-certs\") pod \"67ba1b17-147c-4b27-b2bd-9660840a9778\" (UID: \"67ba1b17-147c-4b27-b2bd-9660840a9778\") " Oct 30 17:00:57.735439 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:57.735400 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67ba1b17-147c-4b27-b2bd-9660840a9778-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67ba1b17-147c-4b27-b2bd-9660840a9778-model-cache\") pod \"67ba1b17-147c-4b27-b2bd-9660840a9778\" (UID: \"67ba1b17-147c-4b27-b2bd-9660840a9778\") " Oct 30 17:00:57.735747 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:57.735507 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67ba1b17-147c-4b27-b2bd-9660840a9778-dshm\" (UniqueName: \"kubernetes.io/empty-dir/67ba1b17-147c-4b27-b2bd-9660840a9778-dshm\") pod \"67ba1b17-147c-4b27-b2bd-9660840a9778\" (UID: \"67ba1b17-147c-4b27-b2bd-9660840a9778\") " Oct 30 17:00:57.735747 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:57.735567 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dd7dff09-720a-411c-9c69-97bc38ac2178-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd7dff09-720a-411c-9c69-97bc38ac2178-istio-podinfo\") pod \"dd7dff09-720a-411c-9c69-97bc38ac2178\" (UID: \"dd7dff09-720a-411c-9c69-97bc38ac2178\") " Oct 30 17:00:57.735747 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:57.735624 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd7dff09-720a-411c-9c69-97bc38ac2178-istio-data" pod "dd7dff09-720a-411c-9c69-97bc38ac2178" (UID: "dd7dff09-720a-411c-9c69-97bc38ac2178"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:00:57.735747 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:57.735644 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67ba1b17-147c-4b27-b2bd-9660840a9778-home" pod "67ba1b17-147c-4b27-b2bd-9660840a9778" (UID: "67ba1b17-147c-4b27-b2bd-9660840a9778"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:00:57.735747 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:57.735683 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd7dff09-720a-411c-9c69-97bc38ac2178-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd7dff09-720a-411c-9c69-97bc38ac2178-credential-socket\") pod \"dd7dff09-720a-411c-9c69-97bc38ac2178\" (UID: \"dd7dff09-720a-411c-9c69-97bc38ac2178\") " Oct 30 17:00:57.735747 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:57.735742 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67ba1b17-147c-4b27-b2bd-9660840a9778-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67ba1b17-147c-4b27-b2bd-9660840a9778-tmp-dir\") pod \"67ba1b17-147c-4b27-b2bd-9660840a9778\" (UID: \"67ba1b17-147c-4b27-b2bd-9660840a9778\") " Oct 30 17:00:57.736076 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:57.735797 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd7dff09-720a-411c-9c69-97bc38ac2178-kube-api-access-8z7cn\" (UniqueName: \"kubernetes.io/projected/dd7dff09-720a-411c-9c69-97bc38ac2178-kube-api-access-8z7cn\") pod \"dd7dff09-720a-411c-9c69-97bc38ac2178\" (UID: \"dd7dff09-720a-411c-9c69-97bc38ac2178\") " Oct 30 17:00:57.736076 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:57.735840 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67ba1b17-147c-4b27-b2bd-9660840a9778-model-cache" pod "67ba1b17-147c-4b27-b2bd-9660840a9778" (UID: "67ba1b17-147c-4b27-b2bd-9660840a9778"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:00:57.736076 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:57.736029 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd7dff09-720a-411c-9c69-97bc38ac2178-workload-socket" pod "dd7dff09-720a-411c-9c69-97bc38ac2178" (UID: "dd7dff09-720a-411c-9c69-97bc38ac2178"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:00:57.736614 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:57.736437 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd7dff09-720a-411c-9c69-97bc38ac2178-istiod-ca-cert" pod "dd7dff09-720a-411c-9c69-97bc38ac2178" (UID: "dd7dff09-720a-411c-9c69-97bc38ac2178"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:00:57.736915 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:57.736892 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd7dff09-720a-411c-9c69-97bc38ac2178-credential-socket" pod "dd7dff09-720a-411c-9c69-97bc38ac2178" (UID: "dd7dff09-720a-411c-9c69-97bc38ac2178"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:00:57.737051 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:57.737028 5052 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd7dff09-720a-411c-9c69-97bc38ac2178-workload-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:00:57.737138 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:57.737057 5052 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67ba1b17-147c-4b27-b2bd-9660840a9778-home\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:00:57.737138 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:57.737070 5052 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd7dff09-720a-411c-9c69-97bc38ac2178-istiod-ca-cert\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:00:57.737138 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:57.737084 5052 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd7dff09-720a-411c-9c69-97bc38ac2178-istio-data\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:00:57.737138 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:57.737098 5052 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd7dff09-720a-411c-9c69-97bc38ac2178-workload-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:00:57.737138 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:57.737112 5052 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67ba1b17-147c-4b27-b2bd-9660840a9778-model-cache\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:00:57.737138 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:57.737068 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67ba1b17-147c-4b27-b2bd-9660840a9778-tmp-dir" pod "67ba1b17-147c-4b27-b2bd-9660840a9778" (UID: "67ba1b17-147c-4b27-b2bd-9660840a9778"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:00:57.738370 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:57.738192 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd7dff09-720a-411c-9c69-97bc38ac2178-istio-envoy" pod "dd7dff09-720a-411c-9c69-97bc38ac2178" (UID: "dd7dff09-720a-411c-9c69-97bc38ac2178"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:00:57.738502 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:57.738417 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd7dff09-720a-411c-9c69-97bc38ac2178-istio-token" pod "dd7dff09-720a-411c-9c69-97bc38ac2178" (UID: "dd7dff09-720a-411c-9c69-97bc38ac2178"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:00:57.738687 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:57.738663 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67ba1b17-147c-4b27-b2bd-9660840a9778-tls-certs" pod "67ba1b17-147c-4b27-b2bd-9660840a9778" (UID: "67ba1b17-147c-4b27-b2bd-9660840a9778"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:00:57.738949 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:57.738927 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67ba1b17-147c-4b27-b2bd-9660840a9778-kube-api-access-vsb74" pod "67ba1b17-147c-4b27-b2bd-9660840a9778" (UID: "67ba1b17-147c-4b27-b2bd-9660840a9778"). InnerVolumeSpecName "kube-api-access-vsb74". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:00:57.739922 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:57.739897 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dd7dff09-720a-411c-9c69-97bc38ac2178-istio-podinfo" pod "dd7dff09-720a-411c-9c69-97bc38ac2178" (UID: "dd7dff09-720a-411c-9c69-97bc38ac2178"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:00:57.740039 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:57.740020 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67ba1b17-147c-4b27-b2bd-9660840a9778-dshm" pod "67ba1b17-147c-4b27-b2bd-9660840a9778" (UID: "67ba1b17-147c-4b27-b2bd-9660840a9778"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:00:57.740501 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:57.740480 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd7dff09-720a-411c-9c69-97bc38ac2178-kube-api-access-8z7cn" pod "dd7dff09-720a-411c-9c69-97bc38ac2178" (UID: "dd7dff09-720a-411c-9c69-97bc38ac2178"). InnerVolumeSpecName "kube-api-access-8z7cn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:00:57.838943 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:57.838917 5052 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd7dff09-720a-411c-9c69-97bc38ac2178-istio-token\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:00:57.838943 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:57.838941 5052 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67ba1b17-147c-4b27-b2bd-9660840a9778-tls-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:00:57.838943 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:57.838949 5052 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67ba1b17-147c-4b27-b2bd-9660840a9778-dshm\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:00:57.839174 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:57.838960 5052 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd7dff09-720a-411c-9c69-97bc38ac2178-istio-podinfo\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:00:57.839174 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:57.838971 5052 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd7dff09-720a-411c-9c69-97bc38ac2178-credential-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:00:57.839174 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:57.838982 5052 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67ba1b17-147c-4b27-b2bd-9660840a9778-tmp-dir\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:00:57.839174 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:57.838991 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8z7cn\" (UniqueName: \"kubernetes.io/projected/dd7dff09-720a-411c-9c69-97bc38ac2178-kube-api-access-8z7cn\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:00:57.839174 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:57.839002 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsb74\" (UniqueName: \"kubernetes.io/projected/67ba1b17-147c-4b27-b2bd-9660840a9778-kube-api-access-vsb74\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:00:57.839174 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:57.839011 5052 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd7dff09-720a-411c-9c69-97bc38ac2178-istio-envoy\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:00:57.844858 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:57.844832 5052 generic.go:356] "Generic (PLEG): container finished" podID="dd7dff09-720a-411c-9c69-97bc38ac2178" containerID="bd59317436f748983cc0bd1aaa1c88e2b5ae8b0d58b5b4797864cbb0566fa3ce" exitCode=0 Oct 30 17:00:57.844989 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:57.844884 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-c4z7h" Oct 30 17:00:57.844989 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:57.844917 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-c4z7h" event={"ID":"dd7dff09-720a-411c-9c69-97bc38ac2178","Type":"ContainerDied","Data":"bd59317436f748983cc0bd1aaa1c88e2b5ae8b0d58b5b4797864cbb0566fa3ce"} Oct 30 17:00:57.844989 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:57.844979 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-c4z7h" event={"ID":"dd7dff09-720a-411c-9c69-97bc38ac2178","Type":"ContainerDied","Data":"0169dd9ab829153b6613bad338316816820b7cb6c99b3612b15d72892eebf4b4"} Oct 30 17:00:57.845132 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:57.845009 5052 scope.go:122] "RemoveContainer" containerID="bd59317436f748983cc0bd1aaa1c88e2b5ae8b0d58b5b4797864cbb0566fa3ce" Oct 30 17:00:57.846527 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:57.846501 5052 generic.go:356] "Generic (PLEG): container finished" podID="67ba1b17-147c-4b27-b2bd-9660840a9778" containerID="ded388af59dacf976347a1cdb510b8d63807af86594aea32f40a2e10be67f2e0" exitCode=0 Oct 30 17:00:57.846671 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:57.846546 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ea77d8/test-model-ea77d8-kserve-7f798c774c-tbxxl" Oct 30 17:00:57.846671 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:57.846626 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ea77d8/test-model-ea77d8-kserve-7f798c774c-tbxxl" event={"ID":"67ba1b17-147c-4b27-b2bd-9660840a9778","Type":"ContainerDied","Data":"ded388af59dacf976347a1cdb510b8d63807af86594aea32f40a2e10be67f2e0"} Oct 30 17:00:57.846671 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:57.846664 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ea77d8/test-model-ea77d8-kserve-7f798c774c-tbxxl" event={"ID":"67ba1b17-147c-4b27-b2bd-9660840a9778","Type":"ContainerDied","Data":"eb9672e795462d24c9aa077aad92f4e9eb3b79eceda0f9b85f54165c35a07f8a"} Oct 30 17:00:57.858366 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:57.858349 5052 scope.go:122] "RemoveContainer" containerID="d320e735460998219605a75ccf5c0b2679ce5f7943727b561b6bacccaf698dab" Oct 30 17:00:57.871461 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:57.871444 5052 scope.go:122] "RemoveContainer" containerID="bd59317436f748983cc0bd1aaa1c88e2b5ae8b0d58b5b4797864cbb0566fa3ce" Oct 30 17:00:57.871742 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:00:57.871724 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd59317436f748983cc0bd1aaa1c88e2b5ae8b0d58b5b4797864cbb0566fa3ce\": container with ID starting with bd59317436f748983cc0bd1aaa1c88e2b5ae8b0d58b5b4797864cbb0566fa3ce not found: ID does not exist" containerID="bd59317436f748983cc0bd1aaa1c88e2b5ae8b0d58b5b4797864cbb0566fa3ce" Oct 30 17:00:57.871796 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:57.871749 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd59317436f748983cc0bd1aaa1c88e2b5ae8b0d58b5b4797864cbb0566fa3ce"} err="failed to get container status \"bd59317436f748983cc0bd1aaa1c88e2b5ae8b0d58b5b4797864cbb0566fa3ce\": rpc error: code = NotFound desc = could not find container \"bd59317436f748983cc0bd1aaa1c88e2b5ae8b0d58b5b4797864cbb0566fa3ce\": container with ID starting with bd59317436f748983cc0bd1aaa1c88e2b5ae8b0d58b5b4797864cbb0566fa3ce not found: ID does not exist" Oct 30 17:00:57.871796 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:57.871766 5052 scope.go:122] "RemoveContainer" containerID="d320e735460998219605a75ccf5c0b2679ce5f7943727b561b6bacccaf698dab" Oct 30 17:00:57.872048 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:00:57.872021 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d320e735460998219605a75ccf5c0b2679ce5f7943727b561b6bacccaf698dab\": container with ID starting with d320e735460998219605a75ccf5c0b2679ce5f7943727b561b6bacccaf698dab not found: ID does not exist" containerID="d320e735460998219605a75ccf5c0b2679ce5f7943727b561b6bacccaf698dab" Oct 30 17:00:57.872142 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:57.872056 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d320e735460998219605a75ccf5c0b2679ce5f7943727b561b6bacccaf698dab"} err="failed to get container status \"d320e735460998219605a75ccf5c0b2679ce5f7943727b561b6bacccaf698dab\": rpc error: code = NotFound desc = could not find container \"d320e735460998219605a75ccf5c0b2679ce5f7943727b561b6bacccaf698dab\": container with ID starting with d320e735460998219605a75ccf5c0b2679ce5f7943727b561b6bacccaf698dab not found: ID does not exist" Oct 30 17:00:57.872142 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:57.872075 5052 scope.go:122] "RemoveContainer" containerID="ded388af59dacf976347a1cdb510b8d63807af86594aea32f40a2e10be67f2e0" Oct 30 17:00:57.881262 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:57.881238 5052 scope.go:122] "RemoveContainer" containerID="ded388af59dacf976347a1cdb510b8d63807af86594aea32f40a2e10be67f2e0" Oct 30 17:00:57.885114 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:00:57.884487 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ded388af59dacf976347a1cdb510b8d63807af86594aea32f40a2e10be67f2e0\": container with ID starting with ded388af59dacf976347a1cdb510b8d63807af86594aea32f40a2e10be67f2e0 not found: ID does not exist" containerID="ded388af59dacf976347a1cdb510b8d63807af86594aea32f40a2e10be67f2e0" Oct 30 17:00:57.885114 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:57.884529 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ded388af59dacf976347a1cdb510b8d63807af86594aea32f40a2e10be67f2e0"} err="failed to get container status \"ded388af59dacf976347a1cdb510b8d63807af86594aea32f40a2e10be67f2e0\": rpc error: code = NotFound desc = could not find container \"ded388af59dacf976347a1cdb510b8d63807af86594aea32f40a2e10be67f2e0\": container with ID starting with ded388af59dacf976347a1cdb510b8d63807af86594aea32f40a2e10be67f2e0 not found: ID does not exist" Oct 30 17:00:57.907176 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:57.907140 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-ea77d8/test-model-ea77d8-kserve-7f798c774c-tbxxl"] Oct 30 17:00:57.913395 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:57.913373 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-ea77d8/test-model-ea77d8-kserve-7f798c774c-tbxxl"] Oct 30 17:00:57.954910 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:57.954887 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-c4z7h"] Oct 30 17:00:57.961707 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:57.961681 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-c4z7h"] Oct 30 17:00:58.599607 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:58.599562 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67ba1b17-147c-4b27-b2bd-9660840a9778" path="/var/lib/kubelet/pods/67ba1b17-147c-4b27-b2bd-9660840a9778/volumes" Oct 30 17:00:58.600188 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:00:58.600169 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd7dff09-720a-411c-9c69-97bc38ac2178" path="/var/lib/kubelet/pods/dd7dff09-720a-411c-9c69-97bc38ac2178/volumes" Oct 30 17:01:03.032603 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:03.026102 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-689899-84d4f48f6d-48std"] Oct 30 17:01:03.032603 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:03.031047 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-689899-7c7b96964c-ltxh4"] Oct 30 17:01:08.026642 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:08.026600 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-689899-84d4f48f6d-48std" podUID="7301ced5-39dd-4e54-bff6-5262a04a0ebf" containerName="payload-processing" containerID="cri-o://933da0766f70f368f236f5e5eca0f5329edb418a88ce653b777215f1ade554db" gracePeriod=25 Oct 30 17:01:08.032106 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:08.032066 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-689899-7c7b96964c-ltxh4" podUID="50a076fb-ca0f-4c64-8c69-38b4ab22b088" containerName="payload-pre-processing" containerID="cri-o://8091e31f66ae35e784eab02725393affca71048ef30bacb31dcf2c92749538ab" gracePeriod=25 Oct 30 17:01:08.206322 ip-10-0-131-136 kubenswrapper[5052]: W1030 17:01:08.206294 5052 logging.go:55] [core] [Channel #4579 SubChannel #4580]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.159:50052", ServerName: "10.132.0.159:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.159:50052: connect: connection refused" Oct 30 17:01:08.318953 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:08.318927 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-689899-84d4f48f6d-48std" Oct 30 17:01:08.374301 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:08.374053 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7301ced5-39dd-4e54-bff6-5262a04a0ebf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7301ced5-39dd-4e54-bff6-5262a04a0ebf-plugins-config-volume\") pod \"7301ced5-39dd-4e54-bff6-5262a04a0ebf\" (UID: \"7301ced5-39dd-4e54-bff6-5262a04a0ebf\") " Oct 30 17:01:08.374301 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:08.374183 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7301ced5-39dd-4e54-bff6-5262a04a0ebf-kube-api-access-bv9qz\" (UniqueName: \"kubernetes.io/projected/7301ced5-39dd-4e54-bff6-5262a04a0ebf-kube-api-access-bv9qz\") pod \"7301ced5-39dd-4e54-bff6-5262a04a0ebf\" (UID: \"7301ced5-39dd-4e54-bff6-5262a04a0ebf\") " Oct 30 17:01:08.374716 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:08.374686 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7301ced5-39dd-4e54-bff6-5262a04a0ebf-plugins-config-volume" pod "7301ced5-39dd-4e54-bff6-5262a04a0ebf" (UID: "7301ced5-39dd-4e54-bff6-5262a04a0ebf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:01:08.374954 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:08.374933 5052 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7301ced5-39dd-4e54-bff6-5262a04a0ebf-plugins-config-volume\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:01:08.377245 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:08.377203 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7301ced5-39dd-4e54-bff6-5262a04a0ebf-kube-api-access-bv9qz" pod "7301ced5-39dd-4e54-bff6-5262a04a0ebf" (UID: "7301ced5-39dd-4e54-bff6-5262a04a0ebf"). InnerVolumeSpecName "kube-api-access-bv9qz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:01:08.487955 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:08.487904 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bv9qz\" (UniqueName: \"kubernetes.io/projected/7301ced5-39dd-4e54-bff6-5262a04a0ebf-kube-api-access-bv9qz\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:01:08.854692 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:08.854652 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-f92caf/test-model-f92caf-kserve-6c694d4757-z2nrr"] Oct 30 17:01:09.089636 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:09.089560 5052 generic.go:356] "Generic (PLEG): container finished" podID="7301ced5-39dd-4e54-bff6-5262a04a0ebf" containerID="933da0766f70f368f236f5e5eca0f5329edb418a88ce653b777215f1ade554db" exitCode=0 Oct 30 17:01:09.089636 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:09.089639 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-689899-84d4f48f6d-48std" Oct 30 17:01:09.089636 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:09.089643 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-689899-84d4f48f6d-48std" event={"ID":"7301ced5-39dd-4e54-bff6-5262a04a0ebf","Type":"ContainerDied","Data":"933da0766f70f368f236f5e5eca0f5329edb418a88ce653b777215f1ade554db"} Oct 30 17:01:09.090264 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:09.089707 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-689899-84d4f48f6d-48std" event={"ID":"7301ced5-39dd-4e54-bff6-5262a04a0ebf","Type":"ContainerDied","Data":"4738ddd107b1aa04e293c47ca8ce9955336648ec3ed8dfc2ca85109bcee9bb2b"} Oct 30 17:01:09.090264 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:09.089742 5052 scope.go:122] "RemoveContainer" containerID="933da0766f70f368f236f5e5eca0f5329edb418a88ce653b777215f1ade554db" Oct 30 17:01:09.101060 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:09.101034 5052 scope.go:122] "RemoveContainer" containerID="933da0766f70f368f236f5e5eca0f5329edb418a88ce653b777215f1ade554db" Oct 30 17:01:09.101386 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:01:09.101361 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"933da0766f70f368f236f5e5eca0f5329edb418a88ce653b777215f1ade554db\": container with ID starting with 933da0766f70f368f236f5e5eca0f5329edb418a88ce653b777215f1ade554db not found: ID does not exist" containerID="933da0766f70f368f236f5e5eca0f5329edb418a88ce653b777215f1ade554db" Oct 30 17:01:09.101442 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:09.101399 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"933da0766f70f368f236f5e5eca0f5329edb418a88ce653b777215f1ade554db"} err="failed to get container status \"933da0766f70f368f236f5e5eca0f5329edb418a88ce653b777215f1ade554db\": rpc error: code = NotFound desc = could not find container \"933da0766f70f368f236f5e5eca0f5329edb418a88ce653b777215f1ade554db\": container with ID starting with 933da0766f70f368f236f5e5eca0f5329edb418a88ce653b777215f1ade554db not found: ID does not exist" Oct 30 17:01:09.150707 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:09.150669 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-689899-84d4f48f6d-48std"] Oct 30 17:01:09.161087 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:09.161047 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-689899-84d4f48f6d-48std"] Oct 30 17:01:09.206233 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:09.206186 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-689899-84d4f48f6d-48std" podUID="7301ced5-39dd-4e54-bff6-5262a04a0ebf" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.159:50052\" within 1s: context deadline exceeded" Oct 30 17:01:09.220435 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:09.220398 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-689899-7c7b96964c-ltxh4" podUID="50a076fb-ca0f-4c64-8c69-38b4ab22b088" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:01:10.601075 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:10.601043 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7301ced5-39dd-4e54-bff6-5262a04a0ebf" path="/var/lib/kubelet/pods/7301ced5-39dd-4e54-bff6-5262a04a0ebf/volumes" Oct 30 17:01:18.842065 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:01:18.842030 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 17:01:19.246783 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:19.246737 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-689899-7c7b96964c-ltxh4" podUID="50a076fb-ca0f-4c64-8c69-38b4ab22b088" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:01:19.595917 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:19.595830 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ea77d8-6d444d798d-29glb"] Oct 30 17:01:19.633637 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:19.633601 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ea77d8-584f58fcf-k88cs"] Oct 30 17:01:19.666476 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:19.666444 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cdwcj"] Oct 30 17:01:23.878759 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:23.878718 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-f92caf/test-model-f92caf-kserve-6c694d4757-z2nrr" podUID="9551ea7f-15f8-4907-aad5-f9160f7b604e" containerName="main" containerID="cri-o://445476025c22533ea9898f34c5ed7d4ce87b3a94fe65563f418173b1076a6ceb" gracePeriod=45 Oct 30 17:01:24.188305 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:24.188278 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f92caf/test-model-f92caf-kserve-6c694d4757-z2nrr" Oct 30 17:01:24.251565 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:24.251535 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9551ea7f-15f8-4907-aad5-f9160f7b604e-home\" (UniqueName: \"kubernetes.io/empty-dir/9551ea7f-15f8-4907-aad5-f9160f7b604e-home\") pod \"9551ea7f-15f8-4907-aad5-f9160f7b604e\" (UID: \"9551ea7f-15f8-4907-aad5-f9160f7b604e\") " Oct 30 17:01:24.251762 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:24.251612 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9551ea7f-15f8-4907-aad5-f9160f7b604e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9551ea7f-15f8-4907-aad5-f9160f7b604e-dshm\") pod \"9551ea7f-15f8-4907-aad5-f9160f7b604e\" (UID: \"9551ea7f-15f8-4907-aad5-f9160f7b604e\") " Oct 30 17:01:24.251762 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:24.251738 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9551ea7f-15f8-4907-aad5-f9160f7b604e-kube-api-access-xhws4\" (UniqueName: \"kubernetes.io/projected/9551ea7f-15f8-4907-aad5-f9160f7b604e-kube-api-access-xhws4\") pod \"9551ea7f-15f8-4907-aad5-f9160f7b604e\" (UID: \"9551ea7f-15f8-4907-aad5-f9160f7b604e\") " Oct 30 17:01:24.251885 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:24.251777 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9551ea7f-15f8-4907-aad5-f9160f7b604e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9551ea7f-15f8-4907-aad5-f9160f7b604e-model-cache\") pod \"9551ea7f-15f8-4907-aad5-f9160f7b604e\" (UID: \"9551ea7f-15f8-4907-aad5-f9160f7b604e\") " Oct 30 17:01:24.251885 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:24.251838 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9551ea7f-15f8-4907-aad5-f9160f7b604e-tls-certs\" (UniqueName: \"kubernetes.io/secret/9551ea7f-15f8-4907-aad5-f9160f7b604e-tls-certs\") pod \"9551ea7f-15f8-4907-aad5-f9160f7b604e\" (UID: \"9551ea7f-15f8-4907-aad5-f9160f7b604e\") " Oct 30 17:01:24.251885 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:24.251849 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9551ea7f-15f8-4907-aad5-f9160f7b604e-home" pod "9551ea7f-15f8-4907-aad5-f9160f7b604e" (UID: "9551ea7f-15f8-4907-aad5-f9160f7b604e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:01:24.252046 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:24.251907 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9551ea7f-15f8-4907-aad5-f9160f7b604e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9551ea7f-15f8-4907-aad5-f9160f7b604e-tmp-dir\") pod \"9551ea7f-15f8-4907-aad5-f9160f7b604e\" (UID: \"9551ea7f-15f8-4907-aad5-f9160f7b604e\") " Oct 30 17:01:24.252107 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:24.252067 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9551ea7f-15f8-4907-aad5-f9160f7b604e-model-cache" pod "9551ea7f-15f8-4907-aad5-f9160f7b604e" (UID: "9551ea7f-15f8-4907-aad5-f9160f7b604e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:01:24.252348 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:24.252330 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9551ea7f-15f8-4907-aad5-f9160f7b604e-tmp-dir" pod "9551ea7f-15f8-4907-aad5-f9160f7b604e" (UID: "9551ea7f-15f8-4907-aad5-f9160f7b604e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:01:24.252769 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:24.252744 5052 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9551ea7f-15f8-4907-aad5-f9160f7b604e-model-cache\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:01:24.252883 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:24.252773 5052 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9551ea7f-15f8-4907-aad5-f9160f7b604e-tmp-dir\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:01:24.252883 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:24.252786 5052 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9551ea7f-15f8-4907-aad5-f9160f7b604e-home\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:01:24.254980 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:24.254942 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9551ea7f-15f8-4907-aad5-f9160f7b604e-tls-certs" pod "9551ea7f-15f8-4907-aad5-f9160f7b604e" (UID: "9551ea7f-15f8-4907-aad5-f9160f7b604e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:01:24.255079 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:24.254997 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9551ea7f-15f8-4907-aad5-f9160f7b604e-dshm" pod "9551ea7f-15f8-4907-aad5-f9160f7b604e" (UID: "9551ea7f-15f8-4907-aad5-f9160f7b604e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:01:24.255156 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:24.255132 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9551ea7f-15f8-4907-aad5-f9160f7b604e-kube-api-access-xhws4" pod "9551ea7f-15f8-4907-aad5-f9160f7b604e" (UID: "9551ea7f-15f8-4907-aad5-f9160f7b604e"). InnerVolumeSpecName "kube-api-access-xhws4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:01:24.317731 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:24.317702 5052 generic.go:356] "Generic (PLEG): container finished" podID="9551ea7f-15f8-4907-aad5-f9160f7b604e" containerID="445476025c22533ea9898f34c5ed7d4ce87b3a94fe65563f418173b1076a6ceb" exitCode=0 Oct 30 17:01:24.317877 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:24.317761 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f92caf/test-model-f92caf-kserve-6c694d4757-z2nrr" Oct 30 17:01:24.317877 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:24.317758 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f92caf/test-model-f92caf-kserve-6c694d4757-z2nrr" event={"ID":"9551ea7f-15f8-4907-aad5-f9160f7b604e","Type":"ContainerDied","Data":"445476025c22533ea9898f34c5ed7d4ce87b3a94fe65563f418173b1076a6ceb"} Oct 30 17:01:24.317877 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:24.317870 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f92caf/test-model-f92caf-kserve-6c694d4757-z2nrr" event={"ID":"9551ea7f-15f8-4907-aad5-f9160f7b604e","Type":"ContainerDied","Data":"1e7114eed1c959cbac01f10408fb917d7935d75d6992b4a7649fe58de9d42d9e"} Oct 30 17:01:24.318028 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:24.317889 5052 scope.go:122] "RemoveContainer" containerID="445476025c22533ea9898f34c5ed7d4ce87b3a94fe65563f418173b1076a6ceb" Oct 30 17:01:24.330379 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:24.330359 5052 scope.go:122] "RemoveContainer" containerID="445476025c22533ea9898f34c5ed7d4ce87b3a94fe65563f418173b1076a6ceb" Oct 30 17:01:24.330669 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:01:24.330649 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"445476025c22533ea9898f34c5ed7d4ce87b3a94fe65563f418173b1076a6ceb\": container with ID starting with 445476025c22533ea9898f34c5ed7d4ce87b3a94fe65563f418173b1076a6ceb not found: ID does not exist" containerID="445476025c22533ea9898f34c5ed7d4ce87b3a94fe65563f418173b1076a6ceb" Oct 30 17:01:24.330745 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:24.330675 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"445476025c22533ea9898f34c5ed7d4ce87b3a94fe65563f418173b1076a6ceb"} err="failed to get container status \"445476025c22533ea9898f34c5ed7d4ce87b3a94fe65563f418173b1076a6ceb\": rpc error: code = NotFound desc = could not find container \"445476025c22533ea9898f34c5ed7d4ce87b3a94fe65563f418173b1076a6ceb\": container with ID starting with 445476025c22533ea9898f34c5ed7d4ce87b3a94fe65563f418173b1076a6ceb not found: ID does not exist" Oct 30 17:01:24.354839 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:24.354811 5052 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9551ea7f-15f8-4907-aad5-f9160f7b604e-dshm\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:01:24.354839 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:24.354840 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhws4\" (UniqueName: \"kubernetes.io/projected/9551ea7f-15f8-4907-aad5-f9160f7b604e-kube-api-access-xhws4\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:01:24.354974 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:24.354850 5052 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9551ea7f-15f8-4907-aad5-f9160f7b604e-tls-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:01:24.376666 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:24.376644 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-f92caf/test-model-f92caf-kserve-6c694d4757-z2nrr"] Oct 30 17:01:24.384460 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:24.384439 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-f92caf/test-model-f92caf-kserve-6c694d4757-z2nrr"] Oct 30 17:01:24.596330 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:24.596246 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-ea77d8-6d444d798d-29glb" podUID="a2715b96-2ae8-4862-a787-5df13822387c" containerName="payload-processing" containerID="cri-o://fe01afa52c0e2d1956ceaa330da7faf621e1036310ddfce36615623ad17e840a" gracePeriod=25 Oct 30 17:01:24.600105 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:24.600078 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9551ea7f-15f8-4907-aad5-f9160f7b604e" path="/var/lib/kubelet/pods/9551ea7f-15f8-4907-aad5-f9160f7b604e/volumes" Oct 30 17:01:24.655642 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:24.655597 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ea77d8-584f58fcf-k88cs" podUID="2b967416-660d-4e1e-be9a-963007619616" containerName="payload-pre-processing" containerID="cri-o://a638ce3ad204fcc6ef0208a71e2f98b04b7a9ff9de546fa1d7867f0297fa296e" gracePeriod=25 Oct 30 17:01:24.883205 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:24.883179 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ea77d8-6d444d798d-29glb" Oct 30 17:01:24.965844 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:24.965811 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2715b96-2ae8-4862-a787-5df13822387c-kube-api-access-t2tg2\" (UniqueName: \"kubernetes.io/projected/a2715b96-2ae8-4862-a787-5df13822387c-kube-api-access-t2tg2\") pod \"a2715b96-2ae8-4862-a787-5df13822387c\" (UID: \"a2715b96-2ae8-4862-a787-5df13822387c\") " Oct 30 17:01:24.966435 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:24.966079 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2715b96-2ae8-4862-a787-5df13822387c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2715b96-2ae8-4862-a787-5df13822387c-plugins-config-volume\") pod \"a2715b96-2ae8-4862-a787-5df13822387c\" (UID: \"a2715b96-2ae8-4862-a787-5df13822387c\") " Oct 30 17:01:24.966533 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:24.966509 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2715b96-2ae8-4862-a787-5df13822387c-plugins-config-volume" pod "a2715b96-2ae8-4862-a787-5df13822387c" (UID: "a2715b96-2ae8-4862-a787-5df13822387c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:01:24.966954 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:24.966930 5052 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2715b96-2ae8-4862-a787-5df13822387c-plugins-config-volume\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:01:24.968134 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:24.968101 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2715b96-2ae8-4862-a787-5df13822387c-kube-api-access-t2tg2" pod "a2715b96-2ae8-4862-a787-5df13822387c" (UID: "a2715b96-2ae8-4862-a787-5df13822387c"). InnerVolumeSpecName "kube-api-access-t2tg2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:01:25.068787 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:25.068758 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2tg2\" (UniqueName: \"kubernetes.io/projected/a2715b96-2ae8-4862-a787-5df13822387c-kube-api-access-t2tg2\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:01:25.150151 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:25.150104 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-f92caf/test-model-f92caf-kserve-6c694d4757-z2nrr" podUID="9551ea7f-15f8-4907-aad5-f9160f7b604e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.172:8000/ready\": context deadline exceeded" Oct 30 17:01:25.329516 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:25.329481 5052 generic.go:356] "Generic (PLEG): container finished" podID="a2715b96-2ae8-4862-a787-5df13822387c" containerID="fe01afa52c0e2d1956ceaa330da7faf621e1036310ddfce36615623ad17e840a" exitCode=0 Oct 30 17:01:25.329692 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:25.329526 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ea77d8-6d444d798d-29glb" Oct 30 17:01:25.329692 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:25.329552 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ea77d8-6d444d798d-29glb" event={"ID":"a2715b96-2ae8-4862-a787-5df13822387c","Type":"ContainerDied","Data":"fe01afa52c0e2d1956ceaa330da7faf621e1036310ddfce36615623ad17e840a"} Oct 30 17:01:25.329692 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:25.329637 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ea77d8-6d444d798d-29glb" event={"ID":"a2715b96-2ae8-4862-a787-5df13822387c","Type":"ContainerDied","Data":"4acfeb0297631c1e95a0d0389cd96430ee26f37c4189ae4cf05037d4f08d0c58"} Oct 30 17:01:25.329692 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:25.329662 5052 scope.go:122] "RemoveContainer" containerID="fe01afa52c0e2d1956ceaa330da7faf621e1036310ddfce36615623ad17e840a" Oct 30 17:01:25.340792 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:25.340772 5052 scope.go:122] "RemoveContainer" containerID="fe01afa52c0e2d1956ceaa330da7faf621e1036310ddfce36615623ad17e840a" Oct 30 17:01:25.341045 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:01:25.341022 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe01afa52c0e2d1956ceaa330da7faf621e1036310ddfce36615623ad17e840a\": container with ID starting with fe01afa52c0e2d1956ceaa330da7faf621e1036310ddfce36615623ad17e840a not found: ID does not exist" containerID="fe01afa52c0e2d1956ceaa330da7faf621e1036310ddfce36615623ad17e840a" Oct 30 17:01:25.341095 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:25.341054 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe01afa52c0e2d1956ceaa330da7faf621e1036310ddfce36615623ad17e840a"} err="failed to get container status \"fe01afa52c0e2d1956ceaa330da7faf621e1036310ddfce36615623ad17e840a\": rpc error: code = NotFound desc = could not find container \"fe01afa52c0e2d1956ceaa330da7faf621e1036310ddfce36615623ad17e840a\": container with ID starting with fe01afa52c0e2d1956ceaa330da7faf621e1036310ddfce36615623ad17e840a not found: ID does not exist" Oct 30 17:01:25.385847 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:25.385826 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ea77d8-6d444d798d-29glb"] Oct 30 17:01:25.392552 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:25.392533 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ea77d8-6d444d798d-29glb"] Oct 30 17:01:26.599842 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:26.599812 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2715b96-2ae8-4862-a787-5df13822387c" path="/var/lib/kubelet/pods/a2715b96-2ae8-4862-a787-5df13822387c/volumes" Oct 30 17:01:27.705845 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:27.705812 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ea77d8-64c6656497-w87sb"] Oct 30 17:01:27.706517 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:27.706038 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ea77d8-64c6656497-w87sb" podUID="0867eae6-ab5d-495b-8822-5e64be91d7f1" containerName="maas-api" containerID="cri-o://3b377054dfa0fc597c707de84836e193a2ab2f5eab1ae2b77c6bc2de89ce11cd" gracePeriod=30 Oct 30 17:01:28.288605 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:28.286173 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ea77d8-64c6656497-w87sb" Oct 30 17:01:28.370655 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:28.358517 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0867eae6-ab5d-495b-8822-5e64be91d7f1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0867eae6-ab5d-495b-8822-5e64be91d7f1-metrics-certs\") pod \"0867eae6-ab5d-495b-8822-5e64be91d7f1\" (UID: \"0867eae6-ab5d-495b-8822-5e64be91d7f1\") " Oct 30 17:01:28.370655 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:28.358636 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0867eae6-ab5d-495b-8822-5e64be91d7f1-kube-api-access-vx54p\" (UniqueName: \"kubernetes.io/projected/0867eae6-ab5d-495b-8822-5e64be91d7f1-kube-api-access-vx54p\") pod \"0867eae6-ab5d-495b-8822-5e64be91d7f1\" (UID: \"0867eae6-ab5d-495b-8822-5e64be91d7f1\") " Oct 30 17:01:28.370655 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:28.358749 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0867eae6-ab5d-495b-8822-5e64be91d7f1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0867eae6-ab5d-495b-8822-5e64be91d7f1-maas-api-tls\") pod \"0867eae6-ab5d-495b-8822-5e64be91d7f1\" (UID: \"0867eae6-ab5d-495b-8822-5e64be91d7f1\") " Oct 30 17:01:28.370655 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:28.358941 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0867eae6-ab5d-495b-8822-5e64be91d7f1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0867eae6-ab5d-495b-8822-5e64be91d7f1-openshift-service-ca\") pod \"0867eae6-ab5d-495b-8822-5e64be91d7f1\" (UID: \"0867eae6-ab5d-495b-8822-5e64be91d7f1\") " Oct 30 17:01:28.370655 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:28.360001 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0867eae6-ab5d-495b-8822-5e64be91d7f1-openshift-service-ca" pod "0867eae6-ab5d-495b-8822-5e64be91d7f1" (UID: "0867eae6-ab5d-495b-8822-5e64be91d7f1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:01:28.370655 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:28.367959 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0867eae6-ab5d-495b-8822-5e64be91d7f1-kube-api-access-vx54p" pod "0867eae6-ab5d-495b-8822-5e64be91d7f1" (UID: "0867eae6-ab5d-495b-8822-5e64be91d7f1"). InnerVolumeSpecName "kube-api-access-vx54p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:01:28.376604 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:28.372389 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0867eae6-ab5d-495b-8822-5e64be91d7f1-maas-api-tls" pod "0867eae6-ab5d-495b-8822-5e64be91d7f1" (UID: "0867eae6-ab5d-495b-8822-5e64be91d7f1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:01:28.376604 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:28.375710 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0867eae6-ab5d-495b-8822-5e64be91d7f1-metrics-certs" pod "0867eae6-ab5d-495b-8822-5e64be91d7f1" (UID: "0867eae6-ab5d-495b-8822-5e64be91d7f1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:01:28.389558 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:28.389527 5052 generic.go:356] "Generic (PLEG): container finished" podID="50a076fb-ca0f-4c64-8c69-38b4ab22b088" containerID="8091e31f66ae35e784eab02725393affca71048ef30bacb31dcf2c92749538ab" exitCode=0 Oct 30 17:01:28.389724 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:28.389694 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-689899-7c7b96964c-ltxh4" event={"ID":"50a076fb-ca0f-4c64-8c69-38b4ab22b088","Type":"ContainerDied","Data":"8091e31f66ae35e784eab02725393affca71048ef30bacb31dcf2c92749538ab"} Oct 30 17:01:28.391418 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:28.391393 5052 generic.go:356] "Generic (PLEG): container finished" podID="0867eae6-ab5d-495b-8822-5e64be91d7f1" containerID="3b377054dfa0fc597c707de84836e193a2ab2f5eab1ae2b77c6bc2de89ce11cd" exitCode=0 Oct 30 17:01:28.391555 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:28.391540 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ea77d8-64c6656497-w87sb" event={"ID":"0867eae6-ab5d-495b-8822-5e64be91d7f1","Type":"ContainerDied","Data":"3b377054dfa0fc597c707de84836e193a2ab2f5eab1ae2b77c6bc2de89ce11cd"} Oct 30 17:01:28.391665 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:28.391570 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ea77d8-64c6656497-w87sb" event={"ID":"0867eae6-ab5d-495b-8822-5e64be91d7f1","Type":"ContainerDied","Data":"bbba5d510ec24e0a3dd7f5bd5b9902fb48372f83e075a07d0451192346f5c73f"} Oct 30 17:01:28.391665 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:28.391614 5052 scope.go:122] "RemoveContainer" containerID="3b377054dfa0fc597c707de84836e193a2ab2f5eab1ae2b77c6bc2de89ce11cd" Oct 30 17:01:28.391783 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:28.391733 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ea77d8-64c6656497-w87sb" Oct 30 17:01:28.408306 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:28.408289 5052 scope.go:122] "RemoveContainer" containerID="3b377054dfa0fc597c707de84836e193a2ab2f5eab1ae2b77c6bc2de89ce11cd" Oct 30 17:01:28.408647 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:01:28.408618 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b377054dfa0fc597c707de84836e193a2ab2f5eab1ae2b77c6bc2de89ce11cd\": container with ID starting with 3b377054dfa0fc597c707de84836e193a2ab2f5eab1ae2b77c6bc2de89ce11cd not found: ID does not exist" containerID="3b377054dfa0fc597c707de84836e193a2ab2f5eab1ae2b77c6bc2de89ce11cd" Oct 30 17:01:28.408704 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:28.408661 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b377054dfa0fc597c707de84836e193a2ab2f5eab1ae2b77c6bc2de89ce11cd"} err="failed to get container status \"3b377054dfa0fc597c707de84836e193a2ab2f5eab1ae2b77c6bc2de89ce11cd\": rpc error: code = NotFound desc = could not find container \"3b377054dfa0fc597c707de84836e193a2ab2f5eab1ae2b77c6bc2de89ce11cd\": container with ID starting with 3b377054dfa0fc597c707de84836e193a2ab2f5eab1ae2b77c6bc2de89ce11cd not found: ID does not exist" Oct 30 17:01:28.465543 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:28.464547 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ea77d8-64c6656497-w87sb"] Oct 30 17:01:28.467095 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:28.467020 5052 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0867eae6-ab5d-495b-8822-5e64be91d7f1-openshift-service-ca\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:01:28.467095 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:28.467065 5052 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0867eae6-ab5d-495b-8822-5e64be91d7f1-metrics-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:01:28.467095 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:28.467080 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vx54p\" (UniqueName: \"kubernetes.io/projected/0867eae6-ab5d-495b-8822-5e64be91d7f1-kube-api-access-vx54p\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:01:28.467095 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:28.467094 5052 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0867eae6-ab5d-495b-8822-5e64be91d7f1-maas-api-tls\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:01:28.469284 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:28.469260 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ea77d8-64c6656497-w87sb"] Oct 30 17:01:28.515394 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:28.515374 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-689899-7c7b96964c-ltxh4" Oct 30 17:01:28.568392 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:28.568364 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50a076fb-ca0f-4c64-8c69-38b4ab22b088-kube-api-access-7k49d\" (UniqueName: \"kubernetes.io/projected/50a076fb-ca0f-4c64-8c69-38b4ab22b088-kube-api-access-7k49d\") pod \"50a076fb-ca0f-4c64-8c69-38b4ab22b088\" (UID: \"50a076fb-ca0f-4c64-8c69-38b4ab22b088\") " Oct 30 17:01:28.568729 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:28.568710 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50a076fb-ca0f-4c64-8c69-38b4ab22b088-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50a076fb-ca0f-4c64-8c69-38b4ab22b088-plugins-config-volume\") pod \"50a076fb-ca0f-4c64-8c69-38b4ab22b088\" (UID: \"50a076fb-ca0f-4c64-8c69-38b4ab22b088\") " Oct 30 17:01:28.569056 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:28.569034 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50a076fb-ca0f-4c64-8c69-38b4ab22b088-plugins-config-volume" pod "50a076fb-ca0f-4c64-8c69-38b4ab22b088" (UID: "50a076fb-ca0f-4c64-8c69-38b4ab22b088"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:01:28.569265 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:28.569248 5052 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50a076fb-ca0f-4c64-8c69-38b4ab22b088-plugins-config-volume\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:01:28.570656 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:28.570627 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50a076fb-ca0f-4c64-8c69-38b4ab22b088-kube-api-access-7k49d" pod "50a076fb-ca0f-4c64-8c69-38b4ab22b088" (UID: "50a076fb-ca0f-4c64-8c69-38b4ab22b088"). InnerVolumeSpecName "kube-api-access-7k49d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:01:28.599500 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:28.599471 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0867eae6-ab5d-495b-8822-5e64be91d7f1" path="/var/lib/kubelet/pods/0867eae6-ab5d-495b-8822-5e64be91d7f1/volumes" Oct 30 17:01:28.670929 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:28.670904 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7k49d\" (UniqueName: \"kubernetes.io/projected/50a076fb-ca0f-4c64-8c69-38b4ab22b088-kube-api-access-7k49d\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:01:29.400402 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:29.400372 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-689899-7c7b96964c-ltxh4" Oct 30 17:01:29.400402 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:29.400390 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-689899-7c7b96964c-ltxh4" event={"ID":"50a076fb-ca0f-4c64-8c69-38b4ab22b088","Type":"ContainerDied","Data":"5d64ded69aaf942b97830458b8735faac244b19f71673d4bcbced1ccb32d0153"} Oct 30 17:01:29.400959 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:29.400451 5052 scope.go:122] "RemoveContainer" containerID="8091e31f66ae35e784eab02725393affca71048ef30bacb31dcf2c92749538ab" Oct 30 17:01:29.458227 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:29.458201 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-689899-7c7b96964c-ltxh4"] Oct 30 17:01:29.471467 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:29.471442 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-689899-7c7b96964c-ltxh4"] Oct 30 17:01:30.600548 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:30.600515 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50a076fb-ca0f-4c64-8c69-38b4ab22b088" path="/var/lib/kubelet/pods/50a076fb-ca0f-4c64-8c69-38b4ab22b088/volumes" Oct 30 17:01:30.983102 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:30.983055 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ea77d8-584f58fcf-k88cs" podUID="2b967416-660d-4e1e-be9a-963007619616" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:01:40.983960 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:40.983915 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ea77d8-584f58fcf-k88cs" podUID="2b967416-660d-4e1e-be9a-963007619616" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:01:41.062973 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:41.062943 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b4214a-6fc8bc5d7b-m6vlq"] Oct 30 17:01:41.063726 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:41.063710 5052 cpu_manager.go:414] "containerMap: removing container" podUID="0867eae6-ab5d-495b-8822-5e64be91d7f1" containerName="maas-api" Oct 30 17:01:41.063801 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:41.063728 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0867eae6-ab5d-495b-8822-5e64be91d7f1" containerName="maas-api" Oct 30 17:01:41.063801 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:41.063742 5052 cpu_manager.go:414] "containerMap: removing container" podUID="c5898d00-6f84-40ef-890f-a476895b9690" containerName="maas-api" Oct 30 17:01:41.063801 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:41.063747 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5898d00-6f84-40ef-890f-a476895b9690" containerName="maas-api" Oct 30 17:01:41.063801 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:41.063764 5052 cpu_manager.go:414] "containerMap: removing container" podUID="141c2e28-1616-44fb-8e0b-275678b894b0" containerName="payload-pre-processing" Oct 30 17:01:41.063801 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:41.063769 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="141c2e28-1616-44fb-8e0b-275678b894b0" containerName="payload-pre-processing" Oct 30 17:01:41.063801 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:41.063785 5052 cpu_manager.go:414] "containerMap: removing container" podUID="cbc3abfb-2bcf-4133-9a49-65b29f06f12a" containerName="main" Oct 30 17:01:41.063801 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:41.063790 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbc3abfb-2bcf-4133-9a49-65b29f06f12a" containerName="main" Oct 30 17:01:41.064015 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:41.063806 5052 cpu_manager.go:414] "containerMap: removing container" podUID="500d3721-818e-4ad7-b087-6dfd1c1d6368" containerName="istio-proxy" Oct 30 17:01:41.064015 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:41.063812 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="500d3721-818e-4ad7-b087-6dfd1c1d6368" containerName="istio-proxy" Oct 30 17:01:41.064015 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:41.063821 5052 cpu_manager.go:414] "containerMap: removing container" podUID="dd7dff09-720a-411c-9c69-97bc38ac2178" containerName="istio-proxy" Oct 30 17:01:41.064015 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:41.063825 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd7dff09-720a-411c-9c69-97bc38ac2178" containerName="istio-proxy" Oct 30 17:01:41.064015 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:41.063833 5052 cpu_manager.go:414] "containerMap: removing container" podUID="9551ea7f-15f8-4907-aad5-f9160f7b604e" containerName="main" Oct 30 17:01:41.064015 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:41.063838 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9551ea7f-15f8-4907-aad5-f9160f7b604e" containerName="main" Oct 30 17:01:41.064015 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:41.063849 5052 cpu_manager.go:414] "containerMap: removing container" podUID="7301ced5-39dd-4e54-bff6-5262a04a0ebf" containerName="payload-processing" Oct 30 17:01:41.064015 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:41.063854 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7301ced5-39dd-4e54-bff6-5262a04a0ebf" containerName="payload-processing" Oct 30 17:01:41.064015 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:41.063865 5052 cpu_manager.go:414] "containerMap: removing container" podUID="c979b538-a34c-49dc-8269-98ef5063e17c" containerName="payload-pre-processing" Oct 30 17:01:41.064015 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:41.063870 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c979b538-a34c-49dc-8269-98ef5063e17c" containerName="payload-pre-processing" Oct 30 17:01:41.064015 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:41.063882 5052 cpu_manager.go:414] "containerMap: removing container" podUID="50a076fb-ca0f-4c64-8c69-38b4ab22b088" containerName="payload-pre-processing" Oct 30 17:01:41.064015 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:41.063886 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50a076fb-ca0f-4c64-8c69-38b4ab22b088" containerName="payload-pre-processing" Oct 30 17:01:41.064015 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:41.063893 5052 cpu_manager.go:414] "containerMap: removing container" podUID="dd7dff09-720a-411c-9c69-97bc38ac2178" containerName="istio-proxy" Oct 30 17:01:41.064015 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:41.063897 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd7dff09-720a-411c-9c69-97bc38ac2178" containerName="istio-proxy" Oct 30 17:01:41.064015 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:41.063904 5052 cpu_manager.go:414] "containerMap: removing container" podUID="67ba1b17-147c-4b27-b2bd-9660840a9778" containerName="main" Oct 30 17:01:41.064015 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:41.063908 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67ba1b17-147c-4b27-b2bd-9660840a9778" containerName="main" Oct 30 17:01:41.064015 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:41.063915 5052 cpu_manager.go:414] "containerMap: removing container" podUID="a2715b96-2ae8-4862-a787-5df13822387c" containerName="payload-processing" Oct 30 17:01:41.064015 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:41.063921 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2715b96-2ae8-4862-a787-5df13822387c" containerName="payload-processing" Oct 30 17:01:41.064015 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:41.064021 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="7301ced5-39dd-4e54-bff6-5262a04a0ebf" containerName="payload-processing" Oct 30 17:01:41.064602 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:41.064031 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="c979b538-a34c-49dc-8269-98ef5063e17c" containerName="payload-pre-processing" Oct 30 17:01:41.064602 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:41.064038 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="50a076fb-ca0f-4c64-8c69-38b4ab22b088" containerName="payload-pre-processing" Oct 30 17:01:41.064602 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:41.064044 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd7dff09-720a-411c-9c69-97bc38ac2178" containerName="istio-proxy" Oct 30 17:01:41.064602 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:41.064049 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="67ba1b17-147c-4b27-b2bd-9660840a9778" containerName="main" Oct 30 17:01:41.064602 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:41.064056 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2715b96-2ae8-4862-a787-5df13822387c" containerName="payload-processing" Oct 30 17:01:41.064602 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:41.064062 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="0867eae6-ab5d-495b-8822-5e64be91d7f1" containerName="maas-api" Oct 30 17:01:41.064602 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:41.064068 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5898d00-6f84-40ef-890f-a476895b9690" containerName="maas-api" Oct 30 17:01:41.064602 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:41.064076 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="141c2e28-1616-44fb-8e0b-275678b894b0" containerName="payload-pre-processing" Oct 30 17:01:41.064602 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:41.064083 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbc3abfb-2bcf-4133-9a49-65b29f06f12a" containerName="main" Oct 30 17:01:41.064602 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:41.064091 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="500d3721-818e-4ad7-b087-6dfd1c1d6368" containerName="istio-proxy" Oct 30 17:01:41.064602 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:41.064097 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd7dff09-720a-411c-9c69-97bc38ac2178" containerName="istio-proxy" Oct 30 17:01:41.064602 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:41.064105 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="9551ea7f-15f8-4907-aad5-f9160f7b604e" containerName="main" Oct 30 17:01:41.069012 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:41.068990 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b4214a-6fc8bc5d7b-m6vlq" Oct 30 17:01:41.090958 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:41.090932 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b4214a-6fc8bc5d7b-m6vlq"] Oct 30 17:01:41.092302 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:41.092279 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-b4214a-serving-cert\"" Oct 30 17:01:41.166097 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:41.166063 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ccbae02-fb91-4bd2-9e3b-3aa09cd5a415-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-b4214a-6fc8bc5d7b-m6vlq\" (UID: \"9ccbae02-fb91-4bd2-9e3b-3aa09cd5a415\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b4214a-6fc8bc5d7b-m6vlq" Oct 30 17:01:41.166318 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:41.166295 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ccbae02-fb91-4bd2-9e3b-3aa09cd5a415-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-b4214a-6fc8bc5d7b-m6vlq\" (UID: \"9ccbae02-fb91-4bd2-9e3b-3aa09cd5a415\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b4214a-6fc8bc5d7b-m6vlq" Oct 30 17:01:41.166399 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:41.166386 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhf2j\" (UniqueName: \"kubernetes.io/projected/9ccbae02-fb91-4bd2-9e3b-3aa09cd5a415-kube-api-access-nhf2j\") pod \"maas-api-e2e-shared-a-w1-b4214a-6fc8bc5d7b-m6vlq\" (UID: \"9ccbae02-fb91-4bd2-9e3b-3aa09cd5a415\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b4214a-6fc8bc5d7b-m6vlq" Oct 30 17:01:41.166451 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:41.166438 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ccbae02-fb91-4bd2-9e3b-3aa09cd5a415-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-b4214a-6fc8bc5d7b-m6vlq\" (UID: \"9ccbae02-fb91-4bd2-9e3b-3aa09cd5a415\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b4214a-6fc8bc5d7b-m6vlq" Oct 30 17:01:41.267803 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:41.267721 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ccbae02-fb91-4bd2-9e3b-3aa09cd5a415-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-b4214a-6fc8bc5d7b-m6vlq\" (UID: \"9ccbae02-fb91-4bd2-9e3b-3aa09cd5a415\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b4214a-6fc8bc5d7b-m6vlq" Oct 30 17:01:41.267951 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:41.267885 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ccbae02-fb91-4bd2-9e3b-3aa09cd5a415-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-b4214a-6fc8bc5d7b-m6vlq\" (UID: \"9ccbae02-fb91-4bd2-9e3b-3aa09cd5a415\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b4214a-6fc8bc5d7b-m6vlq" Oct 30 17:01:41.268047 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:41.268030 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhf2j\" (UniqueName: \"kubernetes.io/projected/9ccbae02-fb91-4bd2-9e3b-3aa09cd5a415-kube-api-access-nhf2j\") pod \"maas-api-e2e-shared-a-w1-b4214a-6fc8bc5d7b-m6vlq\" (UID: \"9ccbae02-fb91-4bd2-9e3b-3aa09cd5a415\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b4214a-6fc8bc5d7b-m6vlq" Oct 30 17:01:41.268118 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:41.268101 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ccbae02-fb91-4bd2-9e3b-3aa09cd5a415-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-b4214a-6fc8bc5d7b-m6vlq\" (UID: \"9ccbae02-fb91-4bd2-9e3b-3aa09cd5a415\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b4214a-6fc8bc5d7b-m6vlq" Oct 30 17:01:41.268492 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:41.268465 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ccbae02-fb91-4bd2-9e3b-3aa09cd5a415-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-b4214a-6fc8bc5d7b-m6vlq\" (UID: \"9ccbae02-fb91-4bd2-9e3b-3aa09cd5a415\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b4214a-6fc8bc5d7b-m6vlq" Oct 30 17:01:41.270342 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:41.270317 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ccbae02-fb91-4bd2-9e3b-3aa09cd5a415-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-b4214a-6fc8bc5d7b-m6vlq\" (UID: \"9ccbae02-fb91-4bd2-9e3b-3aa09cd5a415\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b4214a-6fc8bc5d7b-m6vlq" Oct 30 17:01:41.270432 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:41.270385 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ccbae02-fb91-4bd2-9e3b-3aa09cd5a415-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-b4214a-6fc8bc5d7b-m6vlq\" (UID: \"9ccbae02-fb91-4bd2-9e3b-3aa09cd5a415\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b4214a-6fc8bc5d7b-m6vlq" Oct 30 17:01:41.319602 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:41.313322 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhf2j\" (UniqueName: \"kubernetes.io/projected/9ccbae02-fb91-4bd2-9e3b-3aa09cd5a415-kube-api-access-nhf2j\") pod \"maas-api-e2e-shared-a-w1-b4214a-6fc8bc5d7b-m6vlq\" (UID: \"9ccbae02-fb91-4bd2-9e3b-3aa09cd5a415\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b4214a-6fc8bc5d7b-m6vlq" Oct 30 17:01:41.386016 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:41.383938 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b4214a-6fc8bc5d7b-m6vlq" Oct 30 17:01:41.698047 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:41.698005 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b4214a-6fc8bc5d7b-m6vlq"] Oct 30 17:01:41.698514 ip-10-0-131-136 kubenswrapper[5052]: W1030 17:01:41.698483 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ccbae02_fb91_4bd2_9e3b_3aa09cd5a415.slice/crio-db49d8b10074f005ccd61f899dbd1066e9d8ee4a2178777b276efae3e32d6761 WatchSource:0}: Error finding container db49d8b10074f005ccd61f899dbd1066e9d8ee4a2178777b276efae3e32d6761: Status 404 returned error can't find the container with id db49d8b10074f005ccd61f899dbd1066e9d8ee4a2178777b276efae3e32d6761 Oct 30 17:01:42.510081 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:42.510041 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b4214a-6fc8bc5d7b-m6vlq" event={"ID":"9ccbae02-fb91-4bd2-9e3b-3aa09cd5a415","Type":"ContainerStarted","Data":"c1ae373539f07c74c164c136de726c4c48ce9039bf4965f4ed55948d3bea6239"} Oct 30 17:01:42.510526 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:42.510099 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b4214a-6fc8bc5d7b-m6vlq" event={"ID":"9ccbae02-fb91-4bd2-9e3b-3aa09cd5a415","Type":"ContainerStarted","Data":"db49d8b10074f005ccd61f899dbd1066e9d8ee4a2178777b276efae3e32d6761"} Oct 30 17:01:42.510704 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:42.510683 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b4214a-6fc8bc5d7b-m6vlq" Oct 30 17:01:42.559767 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:42.559716 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b4214a-6fc8bc5d7b-m6vlq" podStartSLOduration=1.192086573 podStartE2EDuration="1.559702558s" totalImagesPullingTime="367.615985ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:01:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:01:42.555343932 +0000 UTC m=+2458.082679152" watchObservedRunningTime="2026-10-30 17:01:42.559702558 +0000 UTC m=+2458.087037777" Oct 30 17:01:44.972341 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:44.972314 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ea77d8-584f58fcf-k88cs" Oct 30 17:01:44.990496 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:44.990473 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b967416-660d-4e1e-be9a-963007619616-kube-api-access-zms84\" (UniqueName: \"kubernetes.io/projected/2b967416-660d-4e1e-be9a-963007619616-kube-api-access-zms84\") pod \"2b967416-660d-4e1e-be9a-963007619616\" (UID: \"2b967416-660d-4e1e-be9a-963007619616\") " Oct 30 17:01:44.990651 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:44.990522 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b967416-660d-4e1e-be9a-963007619616-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b967416-660d-4e1e-be9a-963007619616-plugins-config-volume\") pod \"2b967416-660d-4e1e-be9a-963007619616\" (UID: \"2b967416-660d-4e1e-be9a-963007619616\") " Oct 30 17:01:44.990897 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:44.990873 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b967416-660d-4e1e-be9a-963007619616-plugins-config-volume" pod "2b967416-660d-4e1e-be9a-963007619616" (UID: "2b967416-660d-4e1e-be9a-963007619616"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:01:44.991112 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:44.991095 5052 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b967416-660d-4e1e-be9a-963007619616-plugins-config-volume\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:01:44.992558 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:44.992532 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b967416-660d-4e1e-be9a-963007619616-kube-api-access-zms84" pod "2b967416-660d-4e1e-be9a-963007619616" (UID: "2b967416-660d-4e1e-be9a-963007619616"). InnerVolumeSpecName "kube-api-access-zms84". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:01:45.092787 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:45.092747 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zms84\" (UniqueName: \"kubernetes.io/projected/2b967416-660d-4e1e-be9a-963007619616-kube-api-access-zms84\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:01:45.542451 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:45.542419 5052 generic.go:356] "Generic (PLEG): container finished" podID="2b967416-660d-4e1e-be9a-963007619616" containerID="a638ce3ad204fcc6ef0208a71e2f98b04b7a9ff9de546fa1d7867f0297fa296e" exitCode=0 Oct 30 17:01:45.542671 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:45.542467 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ea77d8-584f58fcf-k88cs" Oct 30 17:01:45.542671 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:45.542492 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ea77d8-584f58fcf-k88cs" event={"ID":"2b967416-660d-4e1e-be9a-963007619616","Type":"ContainerDied","Data":"a638ce3ad204fcc6ef0208a71e2f98b04b7a9ff9de546fa1d7867f0297fa296e"} Oct 30 17:01:45.542671 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:45.542545 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ea77d8-584f58fcf-k88cs" event={"ID":"2b967416-660d-4e1e-be9a-963007619616","Type":"ContainerDied","Data":"37cbf0655ebf7518ddd061ca7873257f0d00d995b60fe275304e633ee30bd8dc"} Oct 30 17:01:45.542671 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:45.542573 5052 scope.go:122] "RemoveContainer" containerID="a638ce3ad204fcc6ef0208a71e2f98b04b7a9ff9de546fa1d7867f0297fa296e" Oct 30 17:01:45.554406 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:45.554382 5052 scope.go:122] "RemoveContainer" containerID="a638ce3ad204fcc6ef0208a71e2f98b04b7a9ff9de546fa1d7867f0297fa296e" Oct 30 17:01:45.555572 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:01:45.555550 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a638ce3ad204fcc6ef0208a71e2f98b04b7a9ff9de546fa1d7867f0297fa296e\": container with ID starting with a638ce3ad204fcc6ef0208a71e2f98b04b7a9ff9de546fa1d7867f0297fa296e not found: ID does not exist" containerID="a638ce3ad204fcc6ef0208a71e2f98b04b7a9ff9de546fa1d7867f0297fa296e" Oct 30 17:01:45.555707 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:45.555596 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a638ce3ad204fcc6ef0208a71e2f98b04b7a9ff9de546fa1d7867f0297fa296e"} err="failed to get container status \"a638ce3ad204fcc6ef0208a71e2f98b04b7a9ff9de546fa1d7867f0297fa296e\": rpc error: code = NotFound desc = could not find container \"a638ce3ad204fcc6ef0208a71e2f98b04b7a9ff9de546fa1d7867f0297fa296e\": container with ID starting with a638ce3ad204fcc6ef0208a71e2f98b04b7a9ff9de546fa1d7867f0297fa296e not found: ID does not exist" Oct 30 17:01:45.660554 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:45.659886 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ea77d8-584f58fcf-k88cs"] Oct 30 17:01:45.663668 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:45.663634 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ea77d8-584f58fcf-k88cs"] Oct 30 17:01:46.599133 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:46.599097 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b967416-660d-4e1e-be9a-963007619616" path="/var/lib/kubelet/pods/2b967416-660d-4e1e-be9a-963007619616/volumes" Oct 30 17:01:46.968353 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:01:46.968319 5052 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b967416_660d_4e1e_be9a_963007619616.slice/crio-a638ce3ad204fcc6ef0208a71e2f98b04b7a9ff9de546fa1d7867f0297fa296e.scope\": RecentStats: unable to find data in memory cache]" Oct 30 17:01:47.421352 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:47.421313 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-ea77d8-openshift-default-68fd566499-wqdpf"] Oct 30 17:01:47.421777 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:47.421714 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-ea77d8-openshift-default-68fd566499-wqdpf" podUID="2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e" containerName="istio-proxy" containerID="cri-o://16c232ff5edc96e38866097a543394dd5e754303930d96105ed7f288fee430f1" gracePeriod=30 Oct 30 17:01:48.533345 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:48.533316 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b4214a-6fc8bc5d7b-m6vlq" Oct 30 17:01:48.841869 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:01:48.841788 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 17:01:50.969023 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:50.968961 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-p6cdk"] Oct 30 17:01:50.969550 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:50.969234 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-p6cdk" podUID="fff40615-0fac-46d8-ad06-9fe5942628dd" containerName="istio-proxy" containerID="cri-o://3a14f6a0a7b3107db8ee0362834771ff79e779c6c05b2bdae99879b2482caeee" gracePeriod=30 Oct 30 17:01:50.980651 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:50.980628 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-b4xqs"] Oct 30 17:01:50.980861 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:50.980838 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-b4xqs" podUID="157dcd12-c7ff-488a-9b49-fc2a36012728" containerName="istio-proxy" containerID="cri-o://8ff156124dae0f864e03d67d92029d77e36dfc3b4a12774dddd2f52482683256" gracePeriod=30 Oct 30 17:01:50.985943 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:50.985912 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-8w9ml"] Oct 30 17:01:50.986554 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:50.986122 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-8w9ml" podUID="ceae9daf-1d9c-4510-bb01-b001d2fc9151" containerName="istio-proxy" containerID="cri-o://74784cc27ad61fd80786a90a94d39f17556f1e7993f452deb76ff87492c70380" gracePeriod=30 Oct 30 17:01:50.993686 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:50.993664 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-wcwwf"] Oct 30 17:01:50.993878 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:50.993858 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-wcwwf" podUID="a89cc50e-1a94-4b08-80af-963baa6f4927" containerName="istio-proxy" containerID="cri-o://8619a13fe20d3a5dec404c8e15d80778913a856f4f65a26dc3f359c1f4022c85" gracePeriod=30 Oct 30 17:01:51.798818 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:01:51.798785 5052 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b967416_660d_4e1e_be9a_963007619616.slice/crio-a638ce3ad204fcc6ef0208a71e2f98b04b7a9ff9de546fa1d7867f0297fa296e.scope\": RecentStats: unable to find data in memory cache]" Oct 30 17:01:52.656994 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:52.656961 5052 generic.go:356] "Generic (PLEG): container finished" podID="2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e" containerID="16c232ff5edc96e38866097a543394dd5e754303930d96105ed7f288fee430f1" exitCode=0 Oct 30 17:01:52.657338 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:52.657034 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ea77d8-openshift-default-68fd566499-wqdpf" event={"ID":"2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e","Type":"ContainerDied","Data":"16c232ff5edc96e38866097a543394dd5e754303930d96105ed7f288fee430f1"} Oct 30 17:01:53.120136 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:53.120112 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ea77d8-openshift-default-68fd566499-wqdpf" Oct 30 17:01:53.215107 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:53.215006 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e-credential-socket\") pod \"2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e\" (UID: \"2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e\") " Oct 30 17:01:53.215286 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:53.215153 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e-istiod-ca-cert\") pod \"2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e\" (UID: \"2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e\") " Oct 30 17:01:53.215370 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:53.215287 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e-istio-data\") pod \"2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e\" (UID: \"2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e\") " Oct 30 17:01:53.215428 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:53.215366 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e-credential-socket" pod "2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e" (UID: "2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:01:53.215428 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:53.215420 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e-istio-podinfo\") pod \"2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e\" (UID: \"2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e\") " Oct 30 17:01:53.215534 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:53.215454 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e-workload-certs\") pod \"2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e\" (UID: \"2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e\") " Oct 30 17:01:53.215534 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:53.215491 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e-workload-socket\") pod \"2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e\" (UID: \"2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e\") " Oct 30 17:01:53.215672 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:53.215551 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e-istio-envoy\") pod \"2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e\" (UID: \"2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e\") " Oct 30 17:01:53.215672 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:53.215595 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e-istio-token\" (UniqueName: \"kubernetes.io/projected/2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e-istio-token\") pod \"2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e\" (UID: \"2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e\") " Oct 30 17:01:53.215784 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:53.215698 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e-kube-api-access-r5dpw\" (UniqueName: \"kubernetes.io/projected/2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e-kube-api-access-r5dpw\") pod \"2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e\" (UID: \"2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e\") " Oct 30 17:01:53.215784 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:53.215765 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e-workload-certs" pod "2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e" (UID: "2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:01:53.215900 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:53.215814 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e-istiod-ca-cert" pod "2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e" (UID: "2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:01:53.216109 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:53.216059 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e-workload-socket" pod "2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e" (UID: "2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:01:53.216234 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:53.216128 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e-istio-data" pod "2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e" (UID: "2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:01:53.216614 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:53.216481 5052 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e-credential-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:01:53.216614 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:53.216510 5052 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e-istiod-ca-cert\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:01:53.216614 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:53.216525 5052 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e-istio-data\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:01:53.216614 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:53.216537 5052 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e-workload-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:01:53.216614 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:53.216551 5052 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e-workload-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:01:53.218636 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:53.218566 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e-istio-envoy" pod "2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e" (UID: "2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:01:53.218636 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:53.218604 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e-kube-api-access-r5dpw" pod "2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e" (UID: "2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e"). InnerVolumeSpecName "kube-api-access-r5dpw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:01:53.218758 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:53.218640 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e-istio-token" pod "2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e" (UID: "2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:01:53.219127 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:53.219108 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e-istio-podinfo" pod "2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e" (UID: "2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:01:53.320295 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:53.320213 5052 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e-istio-podinfo\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:01:53.320295 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:53.320249 5052 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e-istio-envoy\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:01:53.320295 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:53.320258 5052 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e-istio-token\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:01:53.320295 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:53.320267 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5dpw\" (UniqueName: \"kubernetes.io/projected/2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e-kube-api-access-r5dpw\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:01:53.665323 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:53.665279 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ea77d8-openshift-default-68fd566499-wqdpf" event={"ID":"2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e","Type":"ContainerDied","Data":"079834d7714bff7816beafeafe0d878be4ac03097a0186c2e7d5ff2311a00a20"} Oct 30 17:01:53.665778 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:53.665339 5052 scope.go:122] "RemoveContainer" containerID="16c232ff5edc96e38866097a543394dd5e754303930d96105ed7f288fee430f1" Oct 30 17:01:53.665778 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:53.665297 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ea77d8-openshift-default-68fd566499-wqdpf" Oct 30 17:01:53.733048 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:53.733017 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-ea77d8-openshift-default-68fd566499-wqdpf"] Oct 30 17:01:53.746125 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:53.746090 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-ea77d8-openshift-default-68fd566499-wqdpf"] Oct 30 17:01:54.599183 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:54.599151 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e" path="/var/lib/kubelet/pods/2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e/volumes" Oct 30 17:01:56.314729 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.314703 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-b4xqs" Oct 30 17:01:56.352797 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.352764 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-8w9ml" Oct 30 17:01:56.379392 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.379357 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/157dcd12-c7ff-488a-9b49-fc2a36012728-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/157dcd12-c7ff-488a-9b49-fc2a36012728-istio-data\") pod \"157dcd12-c7ff-488a-9b49-fc2a36012728\" (UID: \"157dcd12-c7ff-488a-9b49-fc2a36012728\") " Oct 30 17:01:56.379624 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.379421 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/157dcd12-c7ff-488a-9b49-fc2a36012728-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/157dcd12-c7ff-488a-9b49-fc2a36012728-istio-envoy\") pod \"157dcd12-c7ff-488a-9b49-fc2a36012728\" (UID: \"157dcd12-c7ff-488a-9b49-fc2a36012728\") " Oct 30 17:01:56.379624 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.379506 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/157dcd12-c7ff-488a-9b49-fc2a36012728-kube-api-access-6jfmp\" (UniqueName: \"kubernetes.io/projected/157dcd12-c7ff-488a-9b49-fc2a36012728-kube-api-access-6jfmp\") pod \"157dcd12-c7ff-488a-9b49-fc2a36012728\" (UID: \"157dcd12-c7ff-488a-9b49-fc2a36012728\") " Oct 30 17:01:56.379624 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.379557 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/157dcd12-c7ff-488a-9b49-fc2a36012728-istio-token\" (UniqueName: \"kubernetes.io/projected/157dcd12-c7ff-488a-9b49-fc2a36012728-istio-token\") pod \"157dcd12-c7ff-488a-9b49-fc2a36012728\" (UID: \"157dcd12-c7ff-488a-9b49-fc2a36012728\") " Oct 30 17:01:56.379861 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.379694 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/157dcd12-c7ff-488a-9b49-fc2a36012728-istio-data" pod "157dcd12-c7ff-488a-9b49-fc2a36012728" (UID: "157dcd12-c7ff-488a-9b49-fc2a36012728"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:01:56.379861 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.379722 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/157dcd12-c7ff-488a-9b49-fc2a36012728-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/157dcd12-c7ff-488a-9b49-fc2a36012728-istiod-ca-cert\") pod \"157dcd12-c7ff-488a-9b49-fc2a36012728\" (UID: \"157dcd12-c7ff-488a-9b49-fc2a36012728\") " Oct 30 17:01:56.379861 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.379785 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/157dcd12-c7ff-488a-9b49-fc2a36012728-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/157dcd12-c7ff-488a-9b49-fc2a36012728-istio-podinfo\") pod \"157dcd12-c7ff-488a-9b49-fc2a36012728\" (UID: \"157dcd12-c7ff-488a-9b49-fc2a36012728\") " Oct 30 17:01:56.380034 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.379860 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/157dcd12-c7ff-488a-9b49-fc2a36012728-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/157dcd12-c7ff-488a-9b49-fc2a36012728-workload-socket\") pod \"157dcd12-c7ff-488a-9b49-fc2a36012728\" (UID: \"157dcd12-c7ff-488a-9b49-fc2a36012728\") " Oct 30 17:01:56.380034 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.379898 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/157dcd12-c7ff-488a-9b49-fc2a36012728-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/157dcd12-c7ff-488a-9b49-fc2a36012728-workload-certs\") pod \"157dcd12-c7ff-488a-9b49-fc2a36012728\" (UID: \"157dcd12-c7ff-488a-9b49-fc2a36012728\") " Oct 30 17:01:56.380034 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.379962 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/157dcd12-c7ff-488a-9b49-fc2a36012728-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/157dcd12-c7ff-488a-9b49-fc2a36012728-credential-socket\") pod \"157dcd12-c7ff-488a-9b49-fc2a36012728\" (UID: \"157dcd12-c7ff-488a-9b49-fc2a36012728\") " Oct 30 17:01:56.380358 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.380331 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/157dcd12-c7ff-488a-9b49-fc2a36012728-workload-socket" pod "157dcd12-c7ff-488a-9b49-fc2a36012728" (UID: "157dcd12-c7ff-488a-9b49-fc2a36012728"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:01:56.381380 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.381354 5052 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/157dcd12-c7ff-488a-9b49-fc2a36012728-istio-data\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:01:56.381495 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.381383 5052 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/157dcd12-c7ff-488a-9b49-fc2a36012728-workload-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:01:56.381742 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.381625 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/157dcd12-c7ff-488a-9b49-fc2a36012728-workload-certs" pod "157dcd12-c7ff-488a-9b49-fc2a36012728" (UID: "157dcd12-c7ff-488a-9b49-fc2a36012728"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:01:56.381826 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.381811 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/157dcd12-c7ff-488a-9b49-fc2a36012728-credential-socket" pod "157dcd12-c7ff-488a-9b49-fc2a36012728" (UID: "157dcd12-c7ff-488a-9b49-fc2a36012728"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:01:56.383294 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.383149 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/157dcd12-c7ff-488a-9b49-fc2a36012728-istiod-ca-cert" pod "157dcd12-c7ff-488a-9b49-fc2a36012728" (UID: "157dcd12-c7ff-488a-9b49-fc2a36012728"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:01:56.391941 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.391893 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/157dcd12-c7ff-488a-9b49-fc2a36012728-istio-podinfo" pod "157dcd12-c7ff-488a-9b49-fc2a36012728" (UID: "157dcd12-c7ff-488a-9b49-fc2a36012728"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:01:56.392047 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.391980 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/157dcd12-c7ff-488a-9b49-fc2a36012728-istio-envoy" pod "157dcd12-c7ff-488a-9b49-fc2a36012728" (UID: "157dcd12-c7ff-488a-9b49-fc2a36012728"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:01:56.393846 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.393819 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/157dcd12-c7ff-488a-9b49-fc2a36012728-istio-token" pod "157dcd12-c7ff-488a-9b49-fc2a36012728" (UID: "157dcd12-c7ff-488a-9b49-fc2a36012728"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:01:56.394099 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.394080 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/157dcd12-c7ff-488a-9b49-fc2a36012728-kube-api-access-6jfmp" pod "157dcd12-c7ff-488a-9b49-fc2a36012728" (UID: "157dcd12-c7ff-488a-9b49-fc2a36012728"). InnerVolumeSpecName "kube-api-access-6jfmp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:01:56.420076 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.420048 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-wcwwf" Oct 30 17:01:56.431756 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.431427 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-p6cdk" Oct 30 17:01:56.481992 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.481959 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ceae9daf-1d9c-4510-bb01-b001d2fc9151-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ceae9daf-1d9c-4510-bb01-b001d2fc9151-istio-data\") pod \"ceae9daf-1d9c-4510-bb01-b001d2fc9151\" (UID: \"ceae9daf-1d9c-4510-bb01-b001d2fc9151\") " Oct 30 17:01:56.482202 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.482049 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ceae9daf-1d9c-4510-bb01-b001d2fc9151-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ceae9daf-1d9c-4510-bb01-b001d2fc9151-istio-envoy\") pod \"ceae9daf-1d9c-4510-bb01-b001d2fc9151\" (UID: \"ceae9daf-1d9c-4510-bb01-b001d2fc9151\") " Oct 30 17:01:56.482202 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.482106 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a89cc50e-1a94-4b08-80af-963baa6f4927-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a89cc50e-1a94-4b08-80af-963baa6f4927-istio-data\") pod \"a89cc50e-1a94-4b08-80af-963baa6f4927\" (UID: \"a89cc50e-1a94-4b08-80af-963baa6f4927\") " Oct 30 17:01:56.482340 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.482201 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ceae9daf-1d9c-4510-bb01-b001d2fc9151-istio-token\" (UniqueName: \"kubernetes.io/projected/ceae9daf-1d9c-4510-bb01-b001d2fc9151-istio-token\") pod \"ceae9daf-1d9c-4510-bb01-b001d2fc9151\" (UID: \"ceae9daf-1d9c-4510-bb01-b001d2fc9151\") " Oct 30 17:01:56.482340 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.482241 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ceae9daf-1d9c-4510-bb01-b001d2fc9151-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ceae9daf-1d9c-4510-bb01-b001d2fc9151-workload-certs\") pod \"ceae9daf-1d9c-4510-bb01-b001d2fc9151\" (UID: \"ceae9daf-1d9c-4510-bb01-b001d2fc9151\") " Oct 30 17:01:56.482340 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.482280 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a89cc50e-1a94-4b08-80af-963baa6f4927-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a89cc50e-1a94-4b08-80af-963baa6f4927-istiod-ca-cert\") pod \"a89cc50e-1a94-4b08-80af-963baa6f4927\" (UID: \"a89cc50e-1a94-4b08-80af-963baa6f4927\") " Oct 30 17:01:56.482340 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.482312 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ceae9daf-1d9c-4510-bb01-b001d2fc9151-istio-data" pod "ceae9daf-1d9c-4510-bb01-b001d2fc9151" (UID: "ceae9daf-1d9c-4510-bb01-b001d2fc9151"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:01:56.482340 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.482333 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a89cc50e-1a94-4b08-80af-963baa6f4927-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a89cc50e-1a94-4b08-80af-963baa6f4927-workload-socket\") pod \"a89cc50e-1a94-4b08-80af-963baa6f4927\" (UID: \"a89cc50e-1a94-4b08-80af-963baa6f4927\") " Oct 30 17:01:56.482633 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.482382 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a89cc50e-1a94-4b08-80af-963baa6f4927-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a89cc50e-1a94-4b08-80af-963baa6f4927-credential-socket\") pod \"a89cc50e-1a94-4b08-80af-963baa6f4927\" (UID: \"a89cc50e-1a94-4b08-80af-963baa6f4927\") " Oct 30 17:01:56.482633 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.482416 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a89cc50e-1a94-4b08-80af-963baa6f4927-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a89cc50e-1a94-4b08-80af-963baa6f4927-istio-envoy\") pod \"a89cc50e-1a94-4b08-80af-963baa6f4927\" (UID: \"a89cc50e-1a94-4b08-80af-963baa6f4927\") " Oct 30 17:01:56.482633 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.482448 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a89cc50e-1a94-4b08-80af-963baa6f4927-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a89cc50e-1a94-4b08-80af-963baa6f4927-istio-podinfo\") pod \"a89cc50e-1a94-4b08-80af-963baa6f4927\" (UID: \"a89cc50e-1a94-4b08-80af-963baa6f4927\") " Oct 30 17:01:56.482633 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.482497 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ceae9daf-1d9c-4510-bb01-b001d2fc9151-kube-api-access-f5lxl\" (UniqueName: \"kubernetes.io/projected/ceae9daf-1d9c-4510-bb01-b001d2fc9151-kube-api-access-f5lxl\") pod \"ceae9daf-1d9c-4510-bb01-b001d2fc9151\" (UID: \"ceae9daf-1d9c-4510-bb01-b001d2fc9151\") " Oct 30 17:01:56.482633 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.482567 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a89cc50e-1a94-4b08-80af-963baa6f4927-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a89cc50e-1a94-4b08-80af-963baa6f4927-workload-certs\") pod \"a89cc50e-1a94-4b08-80af-963baa6f4927\" (UID: \"a89cc50e-1a94-4b08-80af-963baa6f4927\") " Oct 30 17:01:56.482633 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.482608 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ceae9daf-1d9c-4510-bb01-b001d2fc9151-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ceae9daf-1d9c-4510-bb01-b001d2fc9151-istiod-ca-cert\") pod \"ceae9daf-1d9c-4510-bb01-b001d2fc9151\" (UID: \"ceae9daf-1d9c-4510-bb01-b001d2fc9151\") " Oct 30 17:01:56.482948 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.482641 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a89cc50e-1a94-4b08-80af-963baa6f4927-istio-token\" (UniqueName: \"kubernetes.io/projected/a89cc50e-1a94-4b08-80af-963baa6f4927-istio-token\") pod \"a89cc50e-1a94-4b08-80af-963baa6f4927\" (UID: \"a89cc50e-1a94-4b08-80af-963baa6f4927\") " Oct 30 17:01:56.482948 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.482736 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ceae9daf-1d9c-4510-bb01-b001d2fc9151-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ceae9daf-1d9c-4510-bb01-b001d2fc9151-istio-podinfo\") pod \"ceae9daf-1d9c-4510-bb01-b001d2fc9151\" (UID: \"ceae9daf-1d9c-4510-bb01-b001d2fc9151\") " Oct 30 17:01:56.482948 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.482787 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ceae9daf-1d9c-4510-bb01-b001d2fc9151-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceae9daf-1d9c-4510-bb01-b001d2fc9151-credential-socket\") pod \"ceae9daf-1d9c-4510-bb01-b001d2fc9151\" (UID: \"ceae9daf-1d9c-4510-bb01-b001d2fc9151\") " Oct 30 17:01:56.482948 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.482851 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ceae9daf-1d9c-4510-bb01-b001d2fc9151-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceae9daf-1d9c-4510-bb01-b001d2fc9151-workload-socket\") pod \"ceae9daf-1d9c-4510-bb01-b001d2fc9151\" (UID: \"ceae9daf-1d9c-4510-bb01-b001d2fc9151\") " Oct 30 17:01:56.482948 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.482912 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a89cc50e-1a94-4b08-80af-963baa6f4927-kube-api-access-twcnd\" (UniqueName: \"kubernetes.io/projected/a89cc50e-1a94-4b08-80af-963baa6f4927-kube-api-access-twcnd\") pod \"a89cc50e-1a94-4b08-80af-963baa6f4927\" (UID: \"a89cc50e-1a94-4b08-80af-963baa6f4927\") " Oct 30 17:01:56.483489 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.483469 5052 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/157dcd12-c7ff-488a-9b49-fc2a36012728-istio-envoy\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:01:56.483688 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.483672 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jfmp\" (UniqueName: \"kubernetes.io/projected/157dcd12-c7ff-488a-9b49-fc2a36012728-kube-api-access-6jfmp\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:01:56.483793 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.483780 5052 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/157dcd12-c7ff-488a-9b49-fc2a36012728-istio-token\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:01:56.483911 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.483898 5052 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/157dcd12-c7ff-488a-9b49-fc2a36012728-istiod-ca-cert\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:01:56.484302 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.484006 5052 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/157dcd12-c7ff-488a-9b49-fc2a36012728-istio-podinfo\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:01:56.485408 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.484316 5052 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/157dcd12-c7ff-488a-9b49-fc2a36012728-workload-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:01:56.485408 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.484333 5052 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/157dcd12-c7ff-488a-9b49-fc2a36012728-credential-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:01:56.485408 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.484349 5052 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ceae9daf-1d9c-4510-bb01-b001d2fc9151-istio-data\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:01:56.485408 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.483937 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a89cc50e-1a94-4b08-80af-963baa6f4927-workload-certs" pod "a89cc50e-1a94-4b08-80af-963baa6f4927" (UID: "a89cc50e-1a94-4b08-80af-963baa6f4927"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:01:56.485408 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.484136 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ceae9daf-1d9c-4510-bb01-b001d2fc9151-istiod-ca-cert" pod "ceae9daf-1d9c-4510-bb01-b001d2fc9151" (UID: "ceae9daf-1d9c-4510-bb01-b001d2fc9151"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:01:56.485408 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.484368 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ceae9daf-1d9c-4510-bb01-b001d2fc9151-workload-certs" pod "ceae9daf-1d9c-4510-bb01-b001d2fc9151" (UID: "ceae9daf-1d9c-4510-bb01-b001d2fc9151"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:01:56.485408 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.484159 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a89cc50e-1a94-4b08-80af-963baa6f4927-workload-socket" pod "a89cc50e-1a94-4b08-80af-963baa6f4927" (UID: "a89cc50e-1a94-4b08-80af-963baa6f4927"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:01:56.485408 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.484511 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a89cc50e-1a94-4b08-80af-963baa6f4927-istiod-ca-cert" pod "a89cc50e-1a94-4b08-80af-963baa6f4927" (UID: "a89cc50e-1a94-4b08-80af-963baa6f4927"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:01:56.485408 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.484628 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ceae9daf-1d9c-4510-bb01-b001d2fc9151-workload-socket" pod "ceae9daf-1d9c-4510-bb01-b001d2fc9151" (UID: "ceae9daf-1d9c-4510-bb01-b001d2fc9151"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:01:56.485408 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.484819 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a89cc50e-1a94-4b08-80af-963baa6f4927-istio-data" pod "a89cc50e-1a94-4b08-80af-963baa6f4927" (UID: "a89cc50e-1a94-4b08-80af-963baa6f4927"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:01:56.485408 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.485115 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ceae9daf-1d9c-4510-bb01-b001d2fc9151-credential-socket" pod "ceae9daf-1d9c-4510-bb01-b001d2fc9151" (UID: "ceae9daf-1d9c-4510-bb01-b001d2fc9151"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:01:56.490003 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.489971 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ceae9daf-1d9c-4510-bb01-b001d2fc9151-istio-envoy" pod "ceae9daf-1d9c-4510-bb01-b001d2fc9151" (UID: "ceae9daf-1d9c-4510-bb01-b001d2fc9151"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:01:56.490224 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.490193 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a89cc50e-1a94-4b08-80af-963baa6f4927-credential-socket" pod "a89cc50e-1a94-4b08-80af-963baa6f4927" (UID: "a89cc50e-1a94-4b08-80af-963baa6f4927"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:01:56.490345 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.490320 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a89cc50e-1a94-4b08-80af-963baa6f4927-kube-api-access-twcnd" pod "a89cc50e-1a94-4b08-80af-963baa6f4927" (UID: "a89cc50e-1a94-4b08-80af-963baa6f4927"). InnerVolumeSpecName "kube-api-access-twcnd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:01:56.492281 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.492259 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a89cc50e-1a94-4b08-80af-963baa6f4927-istio-envoy" pod "a89cc50e-1a94-4b08-80af-963baa6f4927" (UID: "a89cc50e-1a94-4b08-80af-963baa6f4927"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:01:56.492390 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.492304 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a89cc50e-1a94-4b08-80af-963baa6f4927-istio-podinfo" pod "a89cc50e-1a94-4b08-80af-963baa6f4927" (UID: "a89cc50e-1a94-4b08-80af-963baa6f4927"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:01:56.495870 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.495818 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a89cc50e-1a94-4b08-80af-963baa6f4927-istio-token" pod "a89cc50e-1a94-4b08-80af-963baa6f4927" (UID: "a89cc50e-1a94-4b08-80af-963baa6f4927"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:01:56.496109 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.496090 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ceae9daf-1d9c-4510-bb01-b001d2fc9151-istio-token" pod "ceae9daf-1d9c-4510-bb01-b001d2fc9151" (UID: "ceae9daf-1d9c-4510-bb01-b001d2fc9151"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:01:56.496197 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.496150 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ceae9daf-1d9c-4510-bb01-b001d2fc9151-kube-api-access-f5lxl" pod "ceae9daf-1d9c-4510-bb01-b001d2fc9151" (UID: "ceae9daf-1d9c-4510-bb01-b001d2fc9151"). InnerVolumeSpecName "kube-api-access-f5lxl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:01:56.498483 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.498081 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ceae9daf-1d9c-4510-bb01-b001d2fc9151-istio-podinfo" pod "ceae9daf-1d9c-4510-bb01-b001d2fc9151" (UID: "ceae9daf-1d9c-4510-bb01-b001d2fc9151"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:01:56.585541 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.585450 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fff40615-0fac-46d8-ad06-9fe5942628dd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fff40615-0fac-46d8-ad06-9fe5942628dd-workload-certs\") pod \"fff40615-0fac-46d8-ad06-9fe5942628dd\" (UID: \"fff40615-0fac-46d8-ad06-9fe5942628dd\") " Oct 30 17:01:56.585541 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.585514 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fff40615-0fac-46d8-ad06-9fe5942628dd-kube-api-access-cgkrx\" (UniqueName: \"kubernetes.io/projected/fff40615-0fac-46d8-ad06-9fe5942628dd-kube-api-access-cgkrx\") pod \"fff40615-0fac-46d8-ad06-9fe5942628dd\" (UID: \"fff40615-0fac-46d8-ad06-9fe5942628dd\") " Oct 30 17:01:56.585756 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.585554 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fff40615-0fac-46d8-ad06-9fe5942628dd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fff40615-0fac-46d8-ad06-9fe5942628dd-credential-socket\") pod \"fff40615-0fac-46d8-ad06-9fe5942628dd\" (UID: \"fff40615-0fac-46d8-ad06-9fe5942628dd\") " Oct 30 17:01:56.585756 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.585661 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fff40615-0fac-46d8-ad06-9fe5942628dd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fff40615-0fac-46d8-ad06-9fe5942628dd-istio-envoy\") pod \"fff40615-0fac-46d8-ad06-9fe5942628dd\" (UID: \"fff40615-0fac-46d8-ad06-9fe5942628dd\") " Oct 30 17:01:56.585756 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.585727 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fff40615-0fac-46d8-ad06-9fe5942628dd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fff40615-0fac-46d8-ad06-9fe5942628dd-workload-socket\") pod \"fff40615-0fac-46d8-ad06-9fe5942628dd\" (UID: \"fff40615-0fac-46d8-ad06-9fe5942628dd\") " Oct 30 17:01:56.585880 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.585761 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fff40615-0fac-46d8-ad06-9fe5942628dd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fff40615-0fac-46d8-ad06-9fe5942628dd-istiod-ca-cert\") pod \"fff40615-0fac-46d8-ad06-9fe5942628dd\" (UID: \"fff40615-0fac-46d8-ad06-9fe5942628dd\") " Oct 30 17:01:56.585880 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.585804 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fff40615-0fac-46d8-ad06-9fe5942628dd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fff40615-0fac-46d8-ad06-9fe5942628dd-istio-podinfo\") pod \"fff40615-0fac-46d8-ad06-9fe5942628dd\" (UID: \"fff40615-0fac-46d8-ad06-9fe5942628dd\") " Oct 30 17:01:56.585880 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.585820 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fff40615-0fac-46d8-ad06-9fe5942628dd-workload-certs" pod "fff40615-0fac-46d8-ad06-9fe5942628dd" (UID: "fff40615-0fac-46d8-ad06-9fe5942628dd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:01:56.586125 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.585933 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fff40615-0fac-46d8-ad06-9fe5942628dd-istio-token\" (UniqueName: \"kubernetes.io/projected/fff40615-0fac-46d8-ad06-9fe5942628dd-istio-token\") pod \"fff40615-0fac-46d8-ad06-9fe5942628dd\" (UID: \"fff40615-0fac-46d8-ad06-9fe5942628dd\") " Oct 30 17:01:56.586125 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.585998 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fff40615-0fac-46d8-ad06-9fe5942628dd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fff40615-0fac-46d8-ad06-9fe5942628dd-istio-data\") pod \"fff40615-0fac-46d8-ad06-9fe5942628dd\" (UID: \"fff40615-0fac-46d8-ad06-9fe5942628dd\") " Oct 30 17:01:56.586258 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.586197 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fff40615-0fac-46d8-ad06-9fe5942628dd-workload-socket" pod "fff40615-0fac-46d8-ad06-9fe5942628dd" (UID: "fff40615-0fac-46d8-ad06-9fe5942628dd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:01:56.586304 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.586280 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fff40615-0fac-46d8-ad06-9fe5942628dd-istiod-ca-cert" pod "fff40615-0fac-46d8-ad06-9fe5942628dd" (UID: "fff40615-0fac-46d8-ad06-9fe5942628dd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:01:56.586554 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.586531 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fff40615-0fac-46d8-ad06-9fe5942628dd-istio-data" pod "fff40615-0fac-46d8-ad06-9fe5942628dd" (UID: "fff40615-0fac-46d8-ad06-9fe5942628dd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:01:56.586750 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.586611 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fff40615-0fac-46d8-ad06-9fe5942628dd-credential-socket" pod "fff40615-0fac-46d8-ad06-9fe5942628dd" (UID: "fff40615-0fac-46d8-ad06-9fe5942628dd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:01:56.586919 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.586830 5052 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fff40615-0fac-46d8-ad06-9fe5942628dd-istio-data\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:01:56.586919 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.586853 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twcnd\" (UniqueName: \"kubernetes.io/projected/a89cc50e-1a94-4b08-80af-963baa6f4927-kube-api-access-twcnd\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:01:56.586919 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.586869 5052 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ceae9daf-1d9c-4510-bb01-b001d2fc9151-istio-envoy\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:01:56.586919 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.586882 5052 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a89cc50e-1a94-4b08-80af-963baa6f4927-istio-data\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:01:56.586919 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.586897 5052 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ceae9daf-1d9c-4510-bb01-b001d2fc9151-istio-token\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:01:56.586919 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.586910 5052 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ceae9daf-1d9c-4510-bb01-b001d2fc9151-workload-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:01:56.586919 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.586922 5052 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fff40615-0fac-46d8-ad06-9fe5942628dd-workload-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:01:56.586919 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.586934 5052 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a89cc50e-1a94-4b08-80af-963baa6f4927-istiod-ca-cert\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:01:56.586919 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.586948 5052 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fff40615-0fac-46d8-ad06-9fe5942628dd-credential-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:01:56.587476 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.586962 5052 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a89cc50e-1a94-4b08-80af-963baa6f4927-workload-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:01:56.587476 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.586974 5052 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a89cc50e-1a94-4b08-80af-963baa6f4927-credential-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:01:56.587476 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.586987 5052 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a89cc50e-1a94-4b08-80af-963baa6f4927-istio-envoy\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:01:56.587476 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.587001 5052 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a89cc50e-1a94-4b08-80af-963baa6f4927-istio-podinfo\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:01:56.587476 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.587017 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5lxl\" (UniqueName: \"kubernetes.io/projected/ceae9daf-1d9c-4510-bb01-b001d2fc9151-kube-api-access-f5lxl\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:01:56.587476 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.587032 5052 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a89cc50e-1a94-4b08-80af-963baa6f4927-workload-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:01:56.587476 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.587045 5052 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ceae9daf-1d9c-4510-bb01-b001d2fc9151-istiod-ca-cert\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:01:56.587476 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.587058 5052 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a89cc50e-1a94-4b08-80af-963baa6f4927-istio-token\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:01:56.587476 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.587070 5052 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fff40615-0fac-46d8-ad06-9fe5942628dd-workload-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:01:56.587476 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.587097 5052 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fff40615-0fac-46d8-ad06-9fe5942628dd-istiod-ca-cert\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:01:56.587476 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.587111 5052 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ceae9daf-1d9c-4510-bb01-b001d2fc9151-istio-podinfo\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:01:56.587476 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.587123 5052 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceae9daf-1d9c-4510-bb01-b001d2fc9151-credential-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:01:56.587476 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.587138 5052 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceae9daf-1d9c-4510-bb01-b001d2fc9151-workload-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:01:56.590083 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.590052 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fff40615-0fac-46d8-ad06-9fe5942628dd-istio-token" pod "fff40615-0fac-46d8-ad06-9fe5942628dd" (UID: "fff40615-0fac-46d8-ad06-9fe5942628dd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:01:56.590797 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.590772 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fff40615-0fac-46d8-ad06-9fe5942628dd-kube-api-access-cgkrx" pod "fff40615-0fac-46d8-ad06-9fe5942628dd" (UID: "fff40615-0fac-46d8-ad06-9fe5942628dd"). InnerVolumeSpecName "kube-api-access-cgkrx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:01:56.590866 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.590849 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fff40615-0fac-46d8-ad06-9fe5942628dd-istio-envoy" pod "fff40615-0fac-46d8-ad06-9fe5942628dd" (UID: "fff40615-0fac-46d8-ad06-9fe5942628dd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:01:56.592293 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.592268 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fff40615-0fac-46d8-ad06-9fe5942628dd-istio-podinfo" pod "fff40615-0fac-46d8-ad06-9fe5942628dd" (UID: "fff40615-0fac-46d8-ad06-9fe5942628dd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:01:56.688786 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.688753 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgkrx\" (UniqueName: \"kubernetes.io/projected/fff40615-0fac-46d8-ad06-9fe5942628dd-kube-api-access-cgkrx\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:01:56.688786 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.688767 5052 generic.go:356] "Generic (PLEG): container finished" podID="a89cc50e-1a94-4b08-80af-963baa6f4927" containerID="8619a13fe20d3a5dec404c8e15d80778913a856f4f65a26dc3f359c1f4022c85" exitCode=0 Oct 30 17:01:56.689028 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.688810 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-wcwwf" event={"ID":"a89cc50e-1a94-4b08-80af-963baa6f4927","Type":"ContainerDied","Data":"8619a13fe20d3a5dec404c8e15d80778913a856f4f65a26dc3f359c1f4022c85"} Oct 30 17:01:56.689028 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.688834 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-wcwwf" Oct 30 17:01:56.689028 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.688885 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-wcwwf" event={"ID":"a89cc50e-1a94-4b08-80af-963baa6f4927","Type":"ContainerDied","Data":"6402918c6cc2119b518f8f6bb5bb158b2f547d423b019a82afda1bd61f9b8321"} Oct 30 17:01:56.689028 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.688786 5052 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fff40615-0fac-46d8-ad06-9fe5942628dd-istio-envoy\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:01:56.689028 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.688911 5052 scope.go:122] "RemoveContainer" containerID="8619a13fe20d3a5dec404c8e15d80778913a856f4f65a26dc3f359c1f4022c85" Oct 30 17:01:56.689028 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.688925 5052 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fff40615-0fac-46d8-ad06-9fe5942628dd-istio-podinfo\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:01:56.689028 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.688945 5052 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fff40615-0fac-46d8-ad06-9fe5942628dd-istio-token\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:01:56.690489 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.690467 5052 generic.go:356] "Generic (PLEG): container finished" podID="157dcd12-c7ff-488a-9b49-fc2a36012728" containerID="8ff156124dae0f864e03d67d92029d77e36dfc3b4a12774dddd2f52482683256" exitCode=0 Oct 30 17:01:56.690619 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.690528 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-b4xqs" Oct 30 17:01:56.690619 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.690559 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-b4xqs" event={"ID":"157dcd12-c7ff-488a-9b49-fc2a36012728","Type":"ContainerDied","Data":"8ff156124dae0f864e03d67d92029d77e36dfc3b4a12774dddd2f52482683256"} Oct 30 17:01:56.690752 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.690624 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-b4xqs" event={"ID":"157dcd12-c7ff-488a-9b49-fc2a36012728","Type":"ContainerDied","Data":"90788c96c84390861bd52774e75de7a46e46cf5a307b620f220e5218f94d943f"} Oct 30 17:01:56.692313 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.692291 5052 generic.go:356] "Generic (PLEG): container finished" podID="ceae9daf-1d9c-4510-bb01-b001d2fc9151" containerID="74784cc27ad61fd80786a90a94d39f17556f1e7993f452deb76ff87492c70380" exitCode=0 Oct 30 17:01:56.692422 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.692331 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-8w9ml" Oct 30 17:01:56.692422 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.692366 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-8w9ml" event={"ID":"ceae9daf-1d9c-4510-bb01-b001d2fc9151","Type":"ContainerDied","Data":"74784cc27ad61fd80786a90a94d39f17556f1e7993f452deb76ff87492c70380"} Oct 30 17:01:56.692422 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.692410 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-8w9ml" event={"ID":"ceae9daf-1d9c-4510-bb01-b001d2fc9151","Type":"ContainerDied","Data":"3fa6e719c2eacc351d1fd8edad06dbe14dd933767d314dca2cf2146f71655854"} Oct 30 17:01:56.694062 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.694033 5052 generic.go:356] "Generic (PLEG): container finished" podID="fff40615-0fac-46d8-ad06-9fe5942628dd" containerID="3a14f6a0a7b3107db8ee0362834771ff79e779c6c05b2bdae99879b2482caeee" exitCode=0 Oct 30 17:01:56.694245 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.694079 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-p6cdk" event={"ID":"fff40615-0fac-46d8-ad06-9fe5942628dd","Type":"ContainerDied","Data":"3a14f6a0a7b3107db8ee0362834771ff79e779c6c05b2bdae99879b2482caeee"} Oct 30 17:01:56.694245 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.694089 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-p6cdk" Oct 30 17:01:56.694245 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.694111 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-p6cdk" event={"ID":"fff40615-0fac-46d8-ad06-9fe5942628dd","Type":"ContainerDied","Data":"e56efcbd87b681bfa4074029c78d75863ccf102be23813f063a199ff8b97c0f3"} Oct 30 17:01:56.701186 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.701171 5052 scope.go:122] "RemoveContainer" containerID="8619a13fe20d3a5dec404c8e15d80778913a856f4f65a26dc3f359c1f4022c85" Oct 30 17:01:56.701485 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:01:56.701467 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8619a13fe20d3a5dec404c8e15d80778913a856f4f65a26dc3f359c1f4022c85\": container with ID starting with 8619a13fe20d3a5dec404c8e15d80778913a856f4f65a26dc3f359c1f4022c85 not found: ID does not exist" containerID="8619a13fe20d3a5dec404c8e15d80778913a856f4f65a26dc3f359c1f4022c85" Oct 30 17:01:56.701540 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.701494 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8619a13fe20d3a5dec404c8e15d80778913a856f4f65a26dc3f359c1f4022c85"} err="failed to get container status \"8619a13fe20d3a5dec404c8e15d80778913a856f4f65a26dc3f359c1f4022c85\": rpc error: code = NotFound desc = could not find container \"8619a13fe20d3a5dec404c8e15d80778913a856f4f65a26dc3f359c1f4022c85\": container with ID starting with 8619a13fe20d3a5dec404c8e15d80778913a856f4f65a26dc3f359c1f4022c85 not found: ID does not exist" Oct 30 17:01:56.701540 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.701517 5052 scope.go:122] "RemoveContainer" containerID="8ff156124dae0f864e03d67d92029d77e36dfc3b4a12774dddd2f52482683256" Oct 30 17:01:56.711218 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.711202 5052 scope.go:122] "RemoveContainer" containerID="8ff156124dae0f864e03d67d92029d77e36dfc3b4a12774dddd2f52482683256" Oct 30 17:01:56.711505 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:01:56.711485 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ff156124dae0f864e03d67d92029d77e36dfc3b4a12774dddd2f52482683256\": container with ID starting with 8ff156124dae0f864e03d67d92029d77e36dfc3b4a12774dddd2f52482683256 not found: ID does not exist" containerID="8ff156124dae0f864e03d67d92029d77e36dfc3b4a12774dddd2f52482683256" Oct 30 17:01:56.711550 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.711512 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ff156124dae0f864e03d67d92029d77e36dfc3b4a12774dddd2f52482683256"} err="failed to get container status \"8ff156124dae0f864e03d67d92029d77e36dfc3b4a12774dddd2f52482683256\": rpc error: code = NotFound desc = could not find container \"8ff156124dae0f864e03d67d92029d77e36dfc3b4a12774dddd2f52482683256\": container with ID starting with 8ff156124dae0f864e03d67d92029d77e36dfc3b4a12774dddd2f52482683256 not found: ID does not exist" Oct 30 17:01:56.711550 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.711529 5052 scope.go:122] "RemoveContainer" containerID="74784cc27ad61fd80786a90a94d39f17556f1e7993f452deb76ff87492c70380" Oct 30 17:01:56.720988 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.720956 5052 scope.go:122] "RemoveContainer" containerID="74784cc27ad61fd80786a90a94d39f17556f1e7993f452deb76ff87492c70380" Oct 30 17:01:56.721229 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:01:56.721211 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74784cc27ad61fd80786a90a94d39f17556f1e7993f452deb76ff87492c70380\": container with ID starting with 74784cc27ad61fd80786a90a94d39f17556f1e7993f452deb76ff87492c70380 not found: ID does not exist" containerID="74784cc27ad61fd80786a90a94d39f17556f1e7993f452deb76ff87492c70380" Oct 30 17:01:56.721272 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.721235 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74784cc27ad61fd80786a90a94d39f17556f1e7993f452deb76ff87492c70380"} err="failed to get container status \"74784cc27ad61fd80786a90a94d39f17556f1e7993f452deb76ff87492c70380\": rpc error: code = NotFound desc = could not find container \"74784cc27ad61fd80786a90a94d39f17556f1e7993f452deb76ff87492c70380\": container with ID starting with 74784cc27ad61fd80786a90a94d39f17556f1e7993f452deb76ff87492c70380 not found: ID does not exist" Oct 30 17:01:56.721272 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.721250 5052 scope.go:122] "RemoveContainer" containerID="3a14f6a0a7b3107db8ee0362834771ff79e779c6c05b2bdae99879b2482caeee" Oct 30 17:01:56.733083 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.733063 5052 scope.go:122] "RemoveContainer" containerID="3a14f6a0a7b3107db8ee0362834771ff79e779c6c05b2bdae99879b2482caeee" Oct 30 17:01:56.733342 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:01:56.733322 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a14f6a0a7b3107db8ee0362834771ff79e779c6c05b2bdae99879b2482caeee\": container with ID starting with 3a14f6a0a7b3107db8ee0362834771ff79e779c6c05b2bdae99879b2482caeee not found: ID does not exist" containerID="3a14f6a0a7b3107db8ee0362834771ff79e779c6c05b2bdae99879b2482caeee" Oct 30 17:01:56.733426 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.733355 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a14f6a0a7b3107db8ee0362834771ff79e779c6c05b2bdae99879b2482caeee"} err="failed to get container status \"3a14f6a0a7b3107db8ee0362834771ff79e779c6c05b2bdae99879b2482caeee\": rpc error: code = NotFound desc = could not find container \"3a14f6a0a7b3107db8ee0362834771ff79e779c6c05b2bdae99879b2482caeee\": container with ID starting with 3a14f6a0a7b3107db8ee0362834771ff79e779c6c05b2bdae99879b2482caeee not found: ID does not exist" Oct 30 17:01:56.747027 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.746997 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-p6cdk"] Oct 30 17:01:56.751589 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.751560 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-p6cdk"] Oct 30 17:01:56.790236 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.790206 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-8w9ml"] Oct 30 17:01:56.793795 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.793771 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-8w9ml"] Oct 30 17:01:56.834891 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.834865 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-wcwwf"] Oct 30 17:01:56.845054 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.844992 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-wcwwf"] Oct 30 17:01:56.881927 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.881899 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-b4xqs"] Oct 30 17:01:56.888618 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:56.888576 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-b4xqs"] Oct 30 17:01:58.599400 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:58.599358 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="157dcd12-c7ff-488a-9b49-fc2a36012728" path="/var/lib/kubelet/pods/157dcd12-c7ff-488a-9b49-fc2a36012728/volumes" Oct 30 17:01:58.600096 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:58.600074 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a89cc50e-1a94-4b08-80af-963baa6f4927" path="/var/lib/kubelet/pods/a89cc50e-1a94-4b08-80af-963baa6f4927/volumes" Oct 30 17:01:58.600622 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:58.600601 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ceae9daf-1d9c-4510-bb01-b001d2fc9151" path="/var/lib/kubelet/pods/ceae9daf-1d9c-4510-bb01-b001d2fc9151/volumes" Oct 30 17:01:58.601449 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:01:58.601428 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fff40615-0fac-46d8-ad06-9fe5942628dd" path="/var/lib/kubelet/pods/fff40615-0fac-46d8-ad06-9fe5942628dd/volumes" Oct 30 17:02:01.889725 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:02:01.889688 5052 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b967416_660d_4e1e_be9a_963007619616.slice/crio-a638ce3ad204fcc6ef0208a71e2f98b04b7a9ff9de546fa1d7867f0297fa296e.scope\": RecentStats: unable to find data in memory cache]" Oct 30 17:02:01.893026 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:02:01.892985 5052 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b967416_660d_4e1e_be9a_963007619616.slice/crio-a638ce3ad204fcc6ef0208a71e2f98b04b7a9ff9de546fa1d7867f0297fa296e.scope\": RecentStats: unable to find data in memory cache]" Oct 30 17:02:10.380120 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:10.380087 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-c8db42/sub-test-model-c8db42-kserve-6676499fbc-2wrm4"] Oct 30 17:02:10.380801 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:10.380784 5052 cpu_manager.go:414] "containerMap: removing container" podUID="a89cc50e-1a94-4b08-80af-963baa6f4927" containerName="istio-proxy" Oct 30 17:02:10.380801 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:10.380800 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a89cc50e-1a94-4b08-80af-963baa6f4927" containerName="istio-proxy" Oct 30 17:02:10.380944 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:10.380811 5052 cpu_manager.go:414] "containerMap: removing container" podUID="ceae9daf-1d9c-4510-bb01-b001d2fc9151" containerName="istio-proxy" Oct 30 17:02:10.380944 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:10.380816 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ceae9daf-1d9c-4510-bb01-b001d2fc9151" containerName="istio-proxy" Oct 30 17:02:10.380944 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:10.380827 5052 cpu_manager.go:414] "containerMap: removing container" podUID="2b967416-660d-4e1e-be9a-963007619616" containerName="payload-pre-processing" Oct 30 17:02:10.380944 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:10.380834 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b967416-660d-4e1e-be9a-963007619616" containerName="payload-pre-processing" Oct 30 17:02:10.380944 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:10.380848 5052 cpu_manager.go:414] "containerMap: removing container" podUID="2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e" containerName="istio-proxy" Oct 30 17:02:10.380944 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:10.380853 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e" containerName="istio-proxy" Oct 30 17:02:10.380944 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:10.380863 5052 cpu_manager.go:414] "containerMap: removing container" podUID="fff40615-0fac-46d8-ad06-9fe5942628dd" containerName="istio-proxy" Oct 30 17:02:10.380944 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:10.380868 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fff40615-0fac-46d8-ad06-9fe5942628dd" containerName="istio-proxy" Oct 30 17:02:10.380944 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:10.380892 5052 cpu_manager.go:414] "containerMap: removing container" podUID="157dcd12-c7ff-488a-9b49-fc2a36012728" containerName="istio-proxy" Oct 30 17:02:10.380944 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:10.380896 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="157dcd12-c7ff-488a-9b49-fc2a36012728" containerName="istio-proxy" Oct 30 17:02:10.381472 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:10.380997 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="157dcd12-c7ff-488a-9b49-fc2a36012728" containerName="istio-proxy" Oct 30 17:02:10.381472 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:10.381010 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="a89cc50e-1a94-4b08-80af-963baa6f4927" containerName="istio-proxy" Oct 30 17:02:10.381472 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:10.381016 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="ceae9daf-1d9c-4510-bb01-b001d2fc9151" containerName="istio-proxy" Oct 30 17:02:10.381472 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:10.381023 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b967416-660d-4e1e-be9a-963007619616" containerName="payload-pre-processing" Oct 30 17:02:10.381472 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:10.381031 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b5e7ff8-4590-4afd-bb81-cc7df5dfd20e" containerName="istio-proxy" Oct 30 17:02:10.381472 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:10.381036 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="fff40615-0fac-46d8-ad06-9fe5942628dd" containerName="istio-proxy" Oct 30 17:02:10.386394 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:10.386368 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-c8db42/sub-test-model-c8db42-kserve-6676499fbc-2wrm4" Oct 30 17:02:10.411258 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:10.411220 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-c8db42/sub-test-model-c8db42-kserve-6676499fbc-2wrm4"] Oct 30 17:02:10.416753 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:10.416721 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-c8db42\"/\"kube-root-ca.crt\"" Oct 30 17:02:10.416913 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:10.416864 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-c8db42\"/\"openshift-service-ca.crt\"" Oct 30 17:02:10.416913 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:10.416883 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-c8db42\"/\"sub-test-model-c8db42-kserve-self-signed-certs\"" Oct 30 17:02:10.417153 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:10.417128 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-c8db42\"/\"default-dockercfg-gmrlp\"" Oct 30 17:02:10.476601 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:10.476507 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c1e0fa7-8b1e-430a-b3ff-cef2a4a7bbd1-dshm\") pod \"sub-test-model-c8db42-kserve-6676499fbc-2wrm4\" (UID: \"9c1e0fa7-8b1e-430a-b3ff-cef2a4a7bbd1\") " pod="ai-tenant-e2e-shared-b-w2-c8db42/sub-test-model-c8db42-kserve-6676499fbc-2wrm4" Oct 30 17:02:10.476601 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:10.476558 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbhdw\" (UniqueName: \"kubernetes.io/projected/9c1e0fa7-8b1e-430a-b3ff-cef2a4a7bbd1-kube-api-access-cbhdw\") pod \"sub-test-model-c8db42-kserve-6676499fbc-2wrm4\" (UID: \"9c1e0fa7-8b1e-430a-b3ff-cef2a4a7bbd1\") " pod="ai-tenant-e2e-shared-b-w2-c8db42/sub-test-model-c8db42-kserve-6676499fbc-2wrm4" Oct 30 17:02:10.476823 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:10.476707 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c1e0fa7-8b1e-430a-b3ff-cef2a4a7bbd1-tmp-dir\") pod \"sub-test-model-c8db42-kserve-6676499fbc-2wrm4\" (UID: \"9c1e0fa7-8b1e-430a-b3ff-cef2a4a7bbd1\") " pod="ai-tenant-e2e-shared-b-w2-c8db42/sub-test-model-c8db42-kserve-6676499fbc-2wrm4" Oct 30 17:02:10.476823 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:10.476763 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c1e0fa7-8b1e-430a-b3ff-cef2a4a7bbd1-tls-certs\") pod \"sub-test-model-c8db42-kserve-6676499fbc-2wrm4\" (UID: \"9c1e0fa7-8b1e-430a-b3ff-cef2a4a7bbd1\") " pod="ai-tenant-e2e-shared-b-w2-c8db42/sub-test-model-c8db42-kserve-6676499fbc-2wrm4" Oct 30 17:02:10.476903 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:10.476882 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c1e0fa7-8b1e-430a-b3ff-cef2a4a7bbd1-home\") pod \"sub-test-model-c8db42-kserve-6676499fbc-2wrm4\" (UID: \"9c1e0fa7-8b1e-430a-b3ff-cef2a4a7bbd1\") " pod="ai-tenant-e2e-shared-b-w2-c8db42/sub-test-model-c8db42-kserve-6676499fbc-2wrm4" Oct 30 17:02:10.476947 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:10.476933 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c1e0fa7-8b1e-430a-b3ff-cef2a4a7bbd1-model-cache\") pod \"sub-test-model-c8db42-kserve-6676499fbc-2wrm4\" (UID: \"9c1e0fa7-8b1e-430a-b3ff-cef2a4a7bbd1\") " pod="ai-tenant-e2e-shared-b-w2-c8db42/sub-test-model-c8db42-kserve-6676499fbc-2wrm4" Oct 30 17:02:10.577970 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:10.577933 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbhdw\" (UniqueName: \"kubernetes.io/projected/9c1e0fa7-8b1e-430a-b3ff-cef2a4a7bbd1-kube-api-access-cbhdw\") pod \"sub-test-model-c8db42-kserve-6676499fbc-2wrm4\" (UID: \"9c1e0fa7-8b1e-430a-b3ff-cef2a4a7bbd1\") " pod="ai-tenant-e2e-shared-b-w2-c8db42/sub-test-model-c8db42-kserve-6676499fbc-2wrm4" Oct 30 17:02:10.578136 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:10.578012 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c1e0fa7-8b1e-430a-b3ff-cef2a4a7bbd1-tmp-dir\") pod \"sub-test-model-c8db42-kserve-6676499fbc-2wrm4\" (UID: \"9c1e0fa7-8b1e-430a-b3ff-cef2a4a7bbd1\") " pod="ai-tenant-e2e-shared-b-w2-c8db42/sub-test-model-c8db42-kserve-6676499fbc-2wrm4" Oct 30 17:02:10.578136 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:10.578065 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c1e0fa7-8b1e-430a-b3ff-cef2a4a7bbd1-tls-certs\") pod \"sub-test-model-c8db42-kserve-6676499fbc-2wrm4\" (UID: \"9c1e0fa7-8b1e-430a-b3ff-cef2a4a7bbd1\") " pod="ai-tenant-e2e-shared-b-w2-c8db42/sub-test-model-c8db42-kserve-6676499fbc-2wrm4" Oct 30 17:02:10.578270 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:10.578168 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c1e0fa7-8b1e-430a-b3ff-cef2a4a7bbd1-home\") pod \"sub-test-model-c8db42-kserve-6676499fbc-2wrm4\" (UID: \"9c1e0fa7-8b1e-430a-b3ff-cef2a4a7bbd1\") " pod="ai-tenant-e2e-shared-b-w2-c8db42/sub-test-model-c8db42-kserve-6676499fbc-2wrm4" Oct 30 17:02:10.578270 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:10.578241 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c1e0fa7-8b1e-430a-b3ff-cef2a4a7bbd1-model-cache\") pod \"sub-test-model-c8db42-kserve-6676499fbc-2wrm4\" (UID: \"9c1e0fa7-8b1e-430a-b3ff-cef2a4a7bbd1\") " pod="ai-tenant-e2e-shared-b-w2-c8db42/sub-test-model-c8db42-kserve-6676499fbc-2wrm4" Oct 30 17:02:10.578382 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:10.578332 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c1e0fa7-8b1e-430a-b3ff-cef2a4a7bbd1-dshm\") pod \"sub-test-model-c8db42-kserve-6676499fbc-2wrm4\" (UID: \"9c1e0fa7-8b1e-430a-b3ff-cef2a4a7bbd1\") " pod="ai-tenant-e2e-shared-b-w2-c8db42/sub-test-model-c8db42-kserve-6676499fbc-2wrm4" Oct 30 17:02:10.578464 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:10.578439 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c1e0fa7-8b1e-430a-b3ff-cef2a4a7bbd1-tmp-dir\") pod \"sub-test-model-c8db42-kserve-6676499fbc-2wrm4\" (UID: \"9c1e0fa7-8b1e-430a-b3ff-cef2a4a7bbd1\") " pod="ai-tenant-e2e-shared-b-w2-c8db42/sub-test-model-c8db42-kserve-6676499fbc-2wrm4" Oct 30 17:02:10.578636 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:10.578575 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c1e0fa7-8b1e-430a-b3ff-cef2a4a7bbd1-home\") pod \"sub-test-model-c8db42-kserve-6676499fbc-2wrm4\" (UID: \"9c1e0fa7-8b1e-430a-b3ff-cef2a4a7bbd1\") " pod="ai-tenant-e2e-shared-b-w2-c8db42/sub-test-model-c8db42-kserve-6676499fbc-2wrm4" Oct 30 17:02:10.578741 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:10.578607 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c1e0fa7-8b1e-430a-b3ff-cef2a4a7bbd1-model-cache\") pod \"sub-test-model-c8db42-kserve-6676499fbc-2wrm4\" (UID: \"9c1e0fa7-8b1e-430a-b3ff-cef2a4a7bbd1\") " pod="ai-tenant-e2e-shared-b-w2-c8db42/sub-test-model-c8db42-kserve-6676499fbc-2wrm4" Oct 30 17:02:10.580432 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:10.580409 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c1e0fa7-8b1e-430a-b3ff-cef2a4a7bbd1-dshm\") pod \"sub-test-model-c8db42-kserve-6676499fbc-2wrm4\" (UID: \"9c1e0fa7-8b1e-430a-b3ff-cef2a4a7bbd1\") " pod="ai-tenant-e2e-shared-b-w2-c8db42/sub-test-model-c8db42-kserve-6676499fbc-2wrm4" Oct 30 17:02:10.580906 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:10.580885 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c1e0fa7-8b1e-430a-b3ff-cef2a4a7bbd1-tls-certs\") pod \"sub-test-model-c8db42-kserve-6676499fbc-2wrm4\" (UID: \"9c1e0fa7-8b1e-430a-b3ff-cef2a4a7bbd1\") " pod="ai-tenant-e2e-shared-b-w2-c8db42/sub-test-model-c8db42-kserve-6676499fbc-2wrm4" Oct 30 17:02:10.643424 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:10.626770 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbhdw\" (UniqueName: \"kubernetes.io/projected/9c1e0fa7-8b1e-430a-b3ff-cef2a4a7bbd1-kube-api-access-cbhdw\") pod \"sub-test-model-c8db42-kserve-6676499fbc-2wrm4\" (UID: \"9c1e0fa7-8b1e-430a-b3ff-cef2a4a7bbd1\") " pod="ai-tenant-e2e-shared-b-w2-c8db42/sub-test-model-c8db42-kserve-6676499fbc-2wrm4" Oct 30 17:02:10.713958 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:10.713842 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-c8db42/sub-test-model-c8db42-kserve-6676499fbc-2wrm4" Oct 30 17:02:11.035548 ip-10-0-131-136 kubenswrapper[5052]: W1030 17:02:11.035517 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9c1e0fa7_8b1e_430a_b3ff_cef2a4a7bbd1.slice/crio-4772117ce90fbd88cb7137806f79094e2f03800050331ddbc92c2b996f1a55a6 WatchSource:0}: Error finding container 4772117ce90fbd88cb7137806f79094e2f03800050331ddbc92c2b996f1a55a6: Status 404 returned error can't find the container with id 4772117ce90fbd88cb7137806f79094e2f03800050331ddbc92c2b996f1a55a6 Oct 30 17:02:11.037189 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:11.037130 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-c8db42/sub-test-model-c8db42-kserve-6676499fbc-2wrm4"] Oct 30 17:02:11.905694 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:11.905651 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-c8db42/sub-test-model-c8db42-kserve-6676499fbc-2wrm4" event={"ID":"9c1e0fa7-8b1e-430a-b3ff-cef2a4a7bbd1","Type":"ContainerStarted","Data":"f478601357926d58a3acffcabf0afdd3760d8f9676fab7bab4bf64f0f27d401b"} Oct 30 17:02:11.906193 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:11.905702 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-c8db42/sub-test-model-c8db42-kserve-6676499fbc-2wrm4" event={"ID":"9c1e0fa7-8b1e-430a-b3ff-cef2a4a7bbd1","Type":"ContainerStarted","Data":"4772117ce90fbd88cb7137806f79094e2f03800050331ddbc92c2b996f1a55a6"} Oct 30 17:02:11.953951 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:11.953882 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-c8db42/sub-test-model-c8db42-kserve-6676499fbc-2wrm4" podStartSLOduration=1.953860747 podStartE2EDuration="1.953860747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:02:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:02:11.948762402 +0000 UTC m=+2487.476097622" watchObservedRunningTime="2026-10-30 17:02:11.953860747 +0000 UTC m=+2487.481195966" Oct 30 17:02:11.997114 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:02:11.997078 5052 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b967416_660d_4e1e_be9a_963007619616.slice/crio-a638ce3ad204fcc6ef0208a71e2f98b04b7a9ff9de546fa1d7867f0297fa296e.scope\": RecentStats: unable to find data in memory cache]" Oct 30 17:02:12.714407 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:12.714371 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-c8db42/sub-test-model-c8db42-kserve-6676499fbc-2wrm4" Oct 30 17:02:14.929690 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:14.929655 5052 generic.go:356] "Generic (PLEG): container finished" podID="d6c3dee8-d01e-445d-b8a4-96b7f57493ec" containerID="6e801b7a1159f0947dc899e0f8a3597f7d099583f63a7e759acb93e810488d39" exitCode=0 Oct 30 17:02:14.930129 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:14.929731 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-x5p9m" event={"ID":"d6c3dee8-d01e-445d-b8a4-96b7f57493ec","Type":"ContainerDied","Data":"6e801b7a1159f0947dc899e0f8a3597f7d099583f63a7e759acb93e810488d39"} Oct 30 17:02:14.930281 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:14.930214 5052 scope.go:122] "RemoveContainer" containerID="6e801b7a1159f0947dc899e0f8a3597f7d099583f63a7e759acb93e810488d39" Oct 30 17:02:14.932559 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:14.932520 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:02:14.932702 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:14.932639 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:02:14.932702 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:14.932691 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:02:15.769384 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:15.769349 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-x5p9m" Oct 30 17:02:15.959352 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:15.959310 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-x5p9m" event={"ID":"d6c3dee8-d01e-445d-b8a4-96b7f57493ec","Type":"ContainerStarted","Data":"b6db49b82fee9c70b44841dcc72fbd0811f6ef15f1630ab7b4ea93e4e3d52770"} Oct 30 17:02:16.385373 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:16.385337 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-x5p9m" Oct 30 17:02:16.386978 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:16.386948 5052 patch_prober.go:29] interesting pod/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-x5p9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Oct 30 17:02:16.387109 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:16.386998 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-x5p9m" podUID="d6c3dee8-d01e-445d-b8a4-96b7f57493ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Oct 30 17:02:16.896507 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:02:16.896471 5052 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b967416_660d_4e1e_be9a_963007619616.slice/crio-a638ce3ad204fcc6ef0208a71e2f98b04b7a9ff9de546fa1d7867f0297fa296e.scope\": RecentStats: unable to find data in memory cache]" Oct 30 17:02:17.385977 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:17.385940 5052 patch_prober.go:29] interesting pod/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-x5p9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Oct 30 17:02:17.386469 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:17.386001 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-x5p9m" podUID="d6c3dee8-d01e-445d-b8a4-96b7f57493ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Oct 30 17:02:18.386508 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:18.386471 5052 patch_prober.go:29] interesting pod/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-x5p9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Oct 30 17:02:18.386940 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:18.386558 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-x5p9m" podUID="d6c3dee8-d01e-445d-b8a4-96b7f57493ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Oct 30 17:02:18.862405 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:02:18.862371 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 17:02:19.391622 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:19.390032 5052 patch_prober.go:29] interesting pod/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-x5p9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Oct 30 17:02:19.391622 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:19.390103 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-x5p9m" podUID="d6c3dee8-d01e-445d-b8a4-96b7f57493ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Oct 30 17:02:20.386432 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:20.386399 5052 patch_prober.go:29] interesting pod/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-x5p9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Oct 30 17:02:20.386664 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:20.386457 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-x5p9m" podUID="d6c3dee8-d01e-445d-b8a4-96b7f57493ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Oct 30 17:02:20.714110 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:20.714024 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-c8db42/sub-test-model-c8db42-kserve-6676499fbc-2wrm4" Oct 30 17:02:20.756345 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:20.756307 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-c8db42/sub-test-model-c8db42-kserve-6676499fbc-2wrm4" Oct 30 17:02:21.008427 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:21.008343 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-c8db42/sub-test-model-c8db42-kserve-6676499fbc-2wrm4" Oct 30 17:02:21.386303 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:21.386215 5052 patch_prober.go:29] interesting pod/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-x5p9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Oct 30 17:02:21.386303 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:21.386271 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-x5p9m" podUID="d6c3dee8-d01e-445d-b8a4-96b7f57493ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Oct 30 17:02:22.020186 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:02:22.020148 5052 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b967416_660d_4e1e_be9a_963007619616.slice/crio-a638ce3ad204fcc6ef0208a71e2f98b04b7a9ff9de546fa1d7867f0297fa296e.scope\": RecentStats: unable to find data in memory cache]" Oct 30 17:02:22.390910 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:22.390826 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-x5p9m" Oct 30 17:02:22.391351 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:22.391337 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-x5p9m" Oct 30 17:02:22.392112 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:22.392088 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-x5p9m" Oct 30 17:02:23.176479 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:23.176446 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-9e533f/auth-test-model-9e533f-kserve-6dbc95dc5c-rfqgk"] Oct 30 17:02:23.181467 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:23.181445 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-9e533f/auth-test-model-9e533f-kserve-6dbc95dc5c-rfqgk" Oct 30 17:02:23.219134 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:23.219092 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-9e533f\"/\"kube-root-ca.crt\"" Oct 30 17:02:23.219313 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:23.219169 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-9e533f\"/\"auth-test-model-9e533f-kserve-self-signed-certs\"" Oct 30 17:02:23.219313 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:23.219228 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-9e533f\"/\"openshift-service-ca.crt\"" Oct 30 17:02:23.219313 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:23.219304 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-9e533f\"/\"default-dockercfg-8j2k2\"" Oct 30 17:02:23.222593 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:23.222560 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-9e533f/auth-test-model-9e533f-kserve-6dbc95dc5c-rfqgk"] Oct 30 17:02:23.322100 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:23.322068 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njxtd\" (UniqueName: \"kubernetes.io/projected/321c0696-5cc2-4038-8783-5d92da061c2d-kube-api-access-njxtd\") pod \"auth-test-model-9e533f-kserve-6dbc95dc5c-rfqgk\" (UID: \"321c0696-5cc2-4038-8783-5d92da061c2d\") " pod="ai-tenant-e2e-shared-b-w0-9e533f/auth-test-model-9e533f-kserve-6dbc95dc5c-rfqgk" Oct 30 17:02:23.322282 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:23.322105 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/321c0696-5cc2-4038-8783-5d92da061c2d-model-cache\") pod \"auth-test-model-9e533f-kserve-6dbc95dc5c-rfqgk\" (UID: \"321c0696-5cc2-4038-8783-5d92da061c2d\") " pod="ai-tenant-e2e-shared-b-w0-9e533f/auth-test-model-9e533f-kserve-6dbc95dc5c-rfqgk" Oct 30 17:02:23.322282 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:23.322251 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/321c0696-5cc2-4038-8783-5d92da061c2d-home\") pod \"auth-test-model-9e533f-kserve-6dbc95dc5c-rfqgk\" (UID: \"321c0696-5cc2-4038-8783-5d92da061c2d\") " pod="ai-tenant-e2e-shared-b-w0-9e533f/auth-test-model-9e533f-kserve-6dbc95dc5c-rfqgk" Oct 30 17:02:23.322382 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:23.322287 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/321c0696-5cc2-4038-8783-5d92da061c2d-tmp-dir\") pod \"auth-test-model-9e533f-kserve-6dbc95dc5c-rfqgk\" (UID: \"321c0696-5cc2-4038-8783-5d92da061c2d\") " pod="ai-tenant-e2e-shared-b-w0-9e533f/auth-test-model-9e533f-kserve-6dbc95dc5c-rfqgk" Oct 30 17:02:23.322382 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:23.322374 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/321c0696-5cc2-4038-8783-5d92da061c2d-tls-certs\") pod \"auth-test-model-9e533f-kserve-6dbc95dc5c-rfqgk\" (UID: \"321c0696-5cc2-4038-8783-5d92da061c2d\") " pod="ai-tenant-e2e-shared-b-w0-9e533f/auth-test-model-9e533f-kserve-6dbc95dc5c-rfqgk" Oct 30 17:02:23.322460 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:23.322414 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/321c0696-5cc2-4038-8783-5d92da061c2d-dshm\") pod \"auth-test-model-9e533f-kserve-6dbc95dc5c-rfqgk\" (UID: \"321c0696-5cc2-4038-8783-5d92da061c2d\") " pod="ai-tenant-e2e-shared-b-w0-9e533f/auth-test-model-9e533f-kserve-6dbc95dc5c-rfqgk" Oct 30 17:02:23.424180 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:23.423840 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njxtd\" (UniqueName: \"kubernetes.io/projected/321c0696-5cc2-4038-8783-5d92da061c2d-kube-api-access-njxtd\") pod \"auth-test-model-9e533f-kserve-6dbc95dc5c-rfqgk\" (UID: \"321c0696-5cc2-4038-8783-5d92da061c2d\") " pod="ai-tenant-e2e-shared-b-w0-9e533f/auth-test-model-9e533f-kserve-6dbc95dc5c-rfqgk" Oct 30 17:02:23.450653 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:23.448816 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/321c0696-5cc2-4038-8783-5d92da061c2d-model-cache\") pod \"auth-test-model-9e533f-kserve-6dbc95dc5c-rfqgk\" (UID: \"321c0696-5cc2-4038-8783-5d92da061c2d\") " pod="ai-tenant-e2e-shared-b-w0-9e533f/auth-test-model-9e533f-kserve-6dbc95dc5c-rfqgk" Oct 30 17:02:23.450653 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:23.449013 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/321c0696-5cc2-4038-8783-5d92da061c2d-home\") pod \"auth-test-model-9e533f-kserve-6dbc95dc5c-rfqgk\" (UID: \"321c0696-5cc2-4038-8783-5d92da061c2d\") " pod="ai-tenant-e2e-shared-b-w0-9e533f/auth-test-model-9e533f-kserve-6dbc95dc5c-rfqgk" Oct 30 17:02:23.450653 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:23.449062 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/321c0696-5cc2-4038-8783-5d92da061c2d-tmp-dir\") pod \"auth-test-model-9e533f-kserve-6dbc95dc5c-rfqgk\" (UID: \"321c0696-5cc2-4038-8783-5d92da061c2d\") " pod="ai-tenant-e2e-shared-b-w0-9e533f/auth-test-model-9e533f-kserve-6dbc95dc5c-rfqgk" Oct 30 17:02:23.450653 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:23.449187 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/321c0696-5cc2-4038-8783-5d92da061c2d-tls-certs\") pod \"auth-test-model-9e533f-kserve-6dbc95dc5c-rfqgk\" (UID: \"321c0696-5cc2-4038-8783-5d92da061c2d\") " pod="ai-tenant-e2e-shared-b-w0-9e533f/auth-test-model-9e533f-kserve-6dbc95dc5c-rfqgk" Oct 30 17:02:23.450653 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:23.449240 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/321c0696-5cc2-4038-8783-5d92da061c2d-dshm\") pod \"auth-test-model-9e533f-kserve-6dbc95dc5c-rfqgk\" (UID: \"321c0696-5cc2-4038-8783-5d92da061c2d\") " pod="ai-tenant-e2e-shared-b-w0-9e533f/auth-test-model-9e533f-kserve-6dbc95dc5c-rfqgk" Oct 30 17:02:23.450653 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:23.450157 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/321c0696-5cc2-4038-8783-5d92da061c2d-home\") pod \"auth-test-model-9e533f-kserve-6dbc95dc5c-rfqgk\" (UID: \"321c0696-5cc2-4038-8783-5d92da061c2d\") " pod="ai-tenant-e2e-shared-b-w0-9e533f/auth-test-model-9e533f-kserve-6dbc95dc5c-rfqgk" Oct 30 17:02:23.450653 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:23.450424 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/321c0696-5cc2-4038-8783-5d92da061c2d-model-cache\") pod \"auth-test-model-9e533f-kserve-6dbc95dc5c-rfqgk\" (UID: \"321c0696-5cc2-4038-8783-5d92da061c2d\") " pod="ai-tenant-e2e-shared-b-w0-9e533f/auth-test-model-9e533f-kserve-6dbc95dc5c-rfqgk" Oct 30 17:02:23.451204 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:23.450676 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/321c0696-5cc2-4038-8783-5d92da061c2d-tmp-dir\") pod \"auth-test-model-9e533f-kserve-6dbc95dc5c-rfqgk\" (UID: \"321c0696-5cc2-4038-8783-5d92da061c2d\") " pod="ai-tenant-e2e-shared-b-w0-9e533f/auth-test-model-9e533f-kserve-6dbc95dc5c-rfqgk" Oct 30 17:02:23.498786 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:23.496500 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/321c0696-5cc2-4038-8783-5d92da061c2d-dshm\") pod \"auth-test-model-9e533f-kserve-6dbc95dc5c-rfqgk\" (UID: \"321c0696-5cc2-4038-8783-5d92da061c2d\") " pod="ai-tenant-e2e-shared-b-w0-9e533f/auth-test-model-9e533f-kserve-6dbc95dc5c-rfqgk" Oct 30 17:02:23.498786 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:23.497248 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njxtd\" (UniqueName: \"kubernetes.io/projected/321c0696-5cc2-4038-8783-5d92da061c2d-kube-api-access-njxtd\") pod \"auth-test-model-9e533f-kserve-6dbc95dc5c-rfqgk\" (UID: \"321c0696-5cc2-4038-8783-5d92da061c2d\") " pod="ai-tenant-e2e-shared-b-w0-9e533f/auth-test-model-9e533f-kserve-6dbc95dc5c-rfqgk" Oct 30 17:02:23.498786 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:23.498258 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/321c0696-5cc2-4038-8783-5d92da061c2d-tls-certs\") pod \"auth-test-model-9e533f-kserve-6dbc95dc5c-rfqgk\" (UID: \"321c0696-5cc2-4038-8783-5d92da061c2d\") " pod="ai-tenant-e2e-shared-b-w0-9e533f/auth-test-model-9e533f-kserve-6dbc95dc5c-rfqgk" Oct 30 17:02:23.511607 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:23.508361 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-9e533f/auth-test-model-9e533f-kserve-6dbc95dc5c-rfqgk" Oct 30 17:02:23.876508 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:23.876465 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-9e533f/auth-test-model-9e533f-kserve-6dbc95dc5c-rfqgk"] Oct 30 17:02:23.877356 ip-10-0-131-136 kubenswrapper[5052]: W1030 17:02:23.877329 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod321c0696_5cc2_4038_8783_5d92da061c2d.slice/crio-f42411f7f262c6380dd40371a012754f2f8b1495606967d5c93b60a1d9c88fe2 WatchSource:0}: Error finding container f42411f7f262c6380dd40371a012754f2f8b1495606967d5c93b60a1d9c88fe2: Status 404 returned error can't find the container with id f42411f7f262c6380dd40371a012754f2f8b1495606967d5c93b60a1d9c88fe2 Oct 30 17:02:24.020407 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:24.020364 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-9e533f/auth-test-model-9e533f-kserve-6dbc95dc5c-rfqgk" event={"ID":"321c0696-5cc2-4038-8783-5d92da061c2d","Type":"ContainerStarted","Data":"ac7dc35476d346b2b331b560b82f81313e935c61cb9267acb1b287ccab94b489"} Oct 30 17:02:24.020611 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:24.020414 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-9e533f/auth-test-model-9e533f-kserve-6dbc95dc5c-rfqgk" event={"ID":"321c0696-5cc2-4038-8783-5d92da061c2d","Type":"ContainerStarted","Data":"f42411f7f262c6380dd40371a012754f2f8b1495606967d5c93b60a1d9c88fe2"} Oct 30 17:02:24.064742 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:24.064694 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-9e533f/auth-test-model-9e533f-kserve-6dbc95dc5c-rfqgk" podStartSLOduration=1.064679587 podStartE2EDuration="1.064679587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:02:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:02:24.063839233 +0000 UTC m=+2499.591174474" watchObservedRunningTime="2026-10-30 17:02:24.064679587 +0000 UTC m=+2499.592014805" Oct 30 17:02:24.509531 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:24.509498 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-9e533f/auth-test-model-9e533f-kserve-6dbc95dc5c-rfqgk" Oct 30 17:02:27.499902 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:27.499867 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f92caf-openshift-default-56d7b858bc-wg985"] Oct 30 17:02:27.500429 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:27.500059 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-f92caf-openshift-default-56d7b858bc-wg985" podUID="e47df7d5-b3d9-4ba4-91a7-7a530ab85616" containerName="istio-proxy" containerID="cri-o://1a9aac3d8750fbb0e7e87936730cc26ebfd9d548bb5164bcf6bc21ec551e462f" gracePeriod=30 Oct 30 17:02:31.980245 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:02:31.980209 5052 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b967416_660d_4e1e_be9a_963007619616.slice/crio-a638ce3ad204fcc6ef0208a71e2f98b04b7a9ff9de546fa1d7867f0297fa296e.scope\": RecentStats: unable to find data in memory cache]" Oct 30 17:02:32.036900 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:02:32.036862 5052 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b967416_660d_4e1e_be9a_963007619616.slice/crio-a638ce3ad204fcc6ef0208a71e2f98b04b7a9ff9de546fa1d7867f0297fa296e.scope\": RecentStats: unable to find data in memory cache]" Oct 30 17:02:32.803264 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:32.803240 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f92caf-openshift-default-56d7b858bc-wg985" Oct 30 17:02:32.915645 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:32.915568 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e47df7d5-b3d9-4ba4-91a7-7a530ab85616-istio-token\" (UniqueName: \"kubernetes.io/projected/e47df7d5-b3d9-4ba4-91a7-7a530ab85616-istio-token\") pod \"e47df7d5-b3d9-4ba4-91a7-7a530ab85616\" (UID: \"e47df7d5-b3d9-4ba4-91a7-7a530ab85616\") " Oct 30 17:02:32.915829 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:32.915690 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e47df7d5-b3d9-4ba4-91a7-7a530ab85616-kube-api-access-zl7zm\" (UniqueName: \"kubernetes.io/projected/e47df7d5-b3d9-4ba4-91a7-7a530ab85616-kube-api-access-zl7zm\") pod \"e47df7d5-b3d9-4ba4-91a7-7a530ab85616\" (UID: \"e47df7d5-b3d9-4ba4-91a7-7a530ab85616\") " Oct 30 17:02:32.915829 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:32.915766 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e47df7d5-b3d9-4ba4-91a7-7a530ab85616-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e47df7d5-b3d9-4ba4-91a7-7a530ab85616-workload-socket\") pod \"e47df7d5-b3d9-4ba4-91a7-7a530ab85616\" (UID: \"e47df7d5-b3d9-4ba4-91a7-7a530ab85616\") " Oct 30 17:02:32.915829 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:32.915804 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e47df7d5-b3d9-4ba4-91a7-7a530ab85616-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e47df7d5-b3d9-4ba4-91a7-7a530ab85616-istio-envoy\") pod \"e47df7d5-b3d9-4ba4-91a7-7a530ab85616\" (UID: \"e47df7d5-b3d9-4ba4-91a7-7a530ab85616\") " Oct 30 17:02:32.916001 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:32.915906 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e47df7d5-b3d9-4ba4-91a7-7a530ab85616-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e47df7d5-b3d9-4ba4-91a7-7a530ab85616-credential-socket\") pod \"e47df7d5-b3d9-4ba4-91a7-7a530ab85616\" (UID: \"e47df7d5-b3d9-4ba4-91a7-7a530ab85616\") " Oct 30 17:02:32.916069 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:32.916011 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e47df7d5-b3d9-4ba4-91a7-7a530ab85616-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e47df7d5-b3d9-4ba4-91a7-7a530ab85616-istio-podinfo\") pod \"e47df7d5-b3d9-4ba4-91a7-7a530ab85616\" (UID: \"e47df7d5-b3d9-4ba4-91a7-7a530ab85616\") " Oct 30 17:02:32.916069 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:32.916050 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e47df7d5-b3d9-4ba4-91a7-7a530ab85616-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e47df7d5-b3d9-4ba4-91a7-7a530ab85616-istio-data\") pod \"e47df7d5-b3d9-4ba4-91a7-7a530ab85616\" (UID: \"e47df7d5-b3d9-4ba4-91a7-7a530ab85616\") " Oct 30 17:02:32.916179 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:32.916112 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e47df7d5-b3d9-4ba4-91a7-7a530ab85616-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e47df7d5-b3d9-4ba4-91a7-7a530ab85616-istiod-ca-cert\") pod \"e47df7d5-b3d9-4ba4-91a7-7a530ab85616\" (UID: \"e47df7d5-b3d9-4ba4-91a7-7a530ab85616\") " Oct 30 17:02:32.916290 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:32.916270 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e47df7d5-b3d9-4ba4-91a7-7a530ab85616-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e47df7d5-b3d9-4ba4-91a7-7a530ab85616-workload-certs\") pod \"e47df7d5-b3d9-4ba4-91a7-7a530ab85616\" (UID: \"e47df7d5-b3d9-4ba4-91a7-7a530ab85616\") " Oct 30 17:02:32.916367 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:32.916283 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e47df7d5-b3d9-4ba4-91a7-7a530ab85616-credential-socket" pod "e47df7d5-b3d9-4ba4-91a7-7a530ab85616" (UID: "e47df7d5-b3d9-4ba4-91a7-7a530ab85616"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:02:32.916616 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:32.916576 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e47df7d5-b3d9-4ba4-91a7-7a530ab85616-workload-socket" pod "e47df7d5-b3d9-4ba4-91a7-7a530ab85616" (UID: "e47df7d5-b3d9-4ba4-91a7-7a530ab85616"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:02:32.916754 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:32.916644 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e47df7d5-b3d9-4ba4-91a7-7a530ab85616-istiod-ca-cert" pod "e47df7d5-b3d9-4ba4-91a7-7a530ab85616" (UID: "e47df7d5-b3d9-4ba4-91a7-7a530ab85616"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:02:32.916754 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:32.916523 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e47df7d5-b3d9-4ba4-91a7-7a530ab85616-istio-data" pod "e47df7d5-b3d9-4ba4-91a7-7a530ab85616" (UID: "e47df7d5-b3d9-4ba4-91a7-7a530ab85616"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:02:32.917002 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:32.916977 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e47df7d5-b3d9-4ba4-91a7-7a530ab85616-workload-certs" pod "e47df7d5-b3d9-4ba4-91a7-7a530ab85616" (UID: "e47df7d5-b3d9-4ba4-91a7-7a530ab85616"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:02:32.917082 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:32.916986 5052 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e47df7d5-b3d9-4ba4-91a7-7a530ab85616-istio-data\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:02:32.917082 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:32.917035 5052 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e47df7d5-b3d9-4ba4-91a7-7a530ab85616-istiod-ca-cert\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:02:32.917082 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:32.917050 5052 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e47df7d5-b3d9-4ba4-91a7-7a530ab85616-workload-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:02:32.917082 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:32.917063 5052 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e47df7d5-b3d9-4ba4-91a7-7a530ab85616-credential-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:02:32.918539 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:32.918504 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e47df7d5-b3d9-4ba4-91a7-7a530ab85616-kube-api-access-zl7zm" pod "e47df7d5-b3d9-4ba4-91a7-7a530ab85616" (UID: "e47df7d5-b3d9-4ba4-91a7-7a530ab85616"). InnerVolumeSpecName "kube-api-access-zl7zm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:02:32.919089 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:32.919061 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e47df7d5-b3d9-4ba4-91a7-7a530ab85616-istio-podinfo" pod "e47df7d5-b3d9-4ba4-91a7-7a530ab85616" (UID: "e47df7d5-b3d9-4ba4-91a7-7a530ab85616"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:02:32.919265 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:32.919247 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e47df7d5-b3d9-4ba4-91a7-7a530ab85616-istio-token" pod "e47df7d5-b3d9-4ba4-91a7-7a530ab85616" (UID: "e47df7d5-b3d9-4ba4-91a7-7a530ab85616"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:02:32.919493 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:32.919476 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e47df7d5-b3d9-4ba4-91a7-7a530ab85616-istio-envoy" pod "e47df7d5-b3d9-4ba4-91a7-7a530ab85616" (UID: "e47df7d5-b3d9-4ba4-91a7-7a530ab85616"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:02:33.018478 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:33.018445 5052 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e47df7d5-b3d9-4ba4-91a7-7a530ab85616-workload-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:02:33.018478 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:33.018476 5052 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e47df7d5-b3d9-4ba4-91a7-7a530ab85616-istio-token\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:02:33.018912 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:33.018486 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zl7zm\" (UniqueName: \"kubernetes.io/projected/e47df7d5-b3d9-4ba4-91a7-7a530ab85616-kube-api-access-zl7zm\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:02:33.018912 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:33.018495 5052 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e47df7d5-b3d9-4ba4-91a7-7a530ab85616-istio-envoy\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:02:33.018912 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:33.018503 5052 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e47df7d5-b3d9-4ba4-91a7-7a530ab85616-istio-podinfo\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:02:33.091865 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:33.091770 5052 generic.go:356] "Generic (PLEG): container finished" podID="e47df7d5-b3d9-4ba4-91a7-7a530ab85616" containerID="1a9aac3d8750fbb0e7e87936730cc26ebfd9d548bb5164bcf6bc21ec551e462f" exitCode=0 Oct 30 17:02:33.091865 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:33.091821 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f92caf-openshift-default-56d7b858bc-wg985" Oct 30 17:02:33.092104 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:33.091845 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f92caf-openshift-default-56d7b858bc-wg985" event={"ID":"e47df7d5-b3d9-4ba4-91a7-7a530ab85616","Type":"ContainerDied","Data":"1a9aac3d8750fbb0e7e87936730cc26ebfd9d548bb5164bcf6bc21ec551e462f"} Oct 30 17:02:33.092104 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:33.091920 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f92caf-openshift-default-56d7b858bc-wg985" event={"ID":"e47df7d5-b3d9-4ba4-91a7-7a530ab85616","Type":"ContainerDied","Data":"9dedd91c60662c49eb431a67efb1f39ed0dac5ed7bf5fcabdb31df2472fa4a78"} Oct 30 17:02:33.092104 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:33.091950 5052 scope.go:122] "RemoveContainer" containerID="1a9aac3d8750fbb0e7e87936730cc26ebfd9d548bb5164bcf6bc21ec551e462f" Oct 30 17:02:33.106885 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:33.106857 5052 scope.go:122] "RemoveContainer" containerID="f9f1ac74236c917e6bf027705dfb0f20b260a1dbef4d2363b76d7d39cd18f6e8" Oct 30 17:02:33.117001 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:33.116977 5052 scope.go:122] "RemoveContainer" containerID="1a9aac3d8750fbb0e7e87936730cc26ebfd9d548bb5164bcf6bc21ec551e462f" Oct 30 17:02:33.117289 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:02:33.117265 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a9aac3d8750fbb0e7e87936730cc26ebfd9d548bb5164bcf6bc21ec551e462f\": container with ID starting with 1a9aac3d8750fbb0e7e87936730cc26ebfd9d548bb5164bcf6bc21ec551e462f not found: ID does not exist" containerID="1a9aac3d8750fbb0e7e87936730cc26ebfd9d548bb5164bcf6bc21ec551e462f" Oct 30 17:02:33.117416 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:33.117296 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a9aac3d8750fbb0e7e87936730cc26ebfd9d548bb5164bcf6bc21ec551e462f"} err="failed to get container status \"1a9aac3d8750fbb0e7e87936730cc26ebfd9d548bb5164bcf6bc21ec551e462f\": rpc error: code = NotFound desc = could not find container \"1a9aac3d8750fbb0e7e87936730cc26ebfd9d548bb5164bcf6bc21ec551e462f\": container with ID starting with 1a9aac3d8750fbb0e7e87936730cc26ebfd9d548bb5164bcf6bc21ec551e462f not found: ID does not exist" Oct 30 17:02:33.117416 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:33.117313 5052 scope.go:122] "RemoveContainer" containerID="f9f1ac74236c917e6bf027705dfb0f20b260a1dbef4d2363b76d7d39cd18f6e8" Oct 30 17:02:33.117634 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:02:33.117614 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9f1ac74236c917e6bf027705dfb0f20b260a1dbef4d2363b76d7d39cd18f6e8\": container with ID starting with f9f1ac74236c917e6bf027705dfb0f20b260a1dbef4d2363b76d7d39cd18f6e8 not found: ID does not exist" containerID="f9f1ac74236c917e6bf027705dfb0f20b260a1dbef4d2363b76d7d39cd18f6e8" Oct 30 17:02:33.117692 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:33.117640 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9f1ac74236c917e6bf027705dfb0f20b260a1dbef4d2363b76d7d39cd18f6e8"} err="failed to get container status \"f9f1ac74236c917e6bf027705dfb0f20b260a1dbef4d2363b76d7d39cd18f6e8\": rpc error: code = NotFound desc = could not find container \"f9f1ac74236c917e6bf027705dfb0f20b260a1dbef4d2363b76d7d39cd18f6e8\": container with ID starting with f9f1ac74236c917e6bf027705dfb0f20b260a1dbef4d2363b76d7d39cd18f6e8 not found: ID does not exist" Oct 30 17:02:33.153065 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:33.153032 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f92caf-openshift-default-56d7b858bc-wg985"] Oct 30 17:02:33.158840 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:33.158813 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-f92caf-openshift-default-56d7b858bc-wg985"] Oct 30 17:02:33.509076 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:33.509035 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-9e533f/auth-test-model-9e533f-kserve-6dbc95dc5c-rfqgk" Oct 30 17:02:33.522078 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:33.522055 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-9e533f/auth-test-model-9e533f-kserve-6dbc95dc5c-rfqgk" Oct 30 17:02:34.119374 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:34.119344 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-9e533f/auth-test-model-9e533f-kserve-6dbc95dc5c-rfqgk" Oct 30 17:02:34.603220 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:34.603178 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e47df7d5-b3d9-4ba4-91a7-7a530ab85616" path="/var/lib/kubelet/pods/e47df7d5-b3d9-4ba4-91a7-7a530ab85616/volumes" Oct 30 17:02:39.734479 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:39.734441 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w6-830ba9/auth-test-model-830ba9-kserve-7cb5656f77-vshw4"] Oct 30 17:02:39.735195 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:39.735180 5052 cpu_manager.go:414] "containerMap: removing container" podUID="e47df7d5-b3d9-4ba4-91a7-7a530ab85616" containerName="istio-proxy" Oct 30 17:02:39.735195 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:39.735195 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e47df7d5-b3d9-4ba4-91a7-7a530ab85616" containerName="istio-proxy" Oct 30 17:02:39.735274 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:39.735237 5052 cpu_manager.go:414] "containerMap: removing container" podUID="e47df7d5-b3d9-4ba4-91a7-7a530ab85616" containerName="istio-proxy" Oct 30 17:02:39.735274 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:39.735242 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e47df7d5-b3d9-4ba4-91a7-7a530ab85616" containerName="istio-proxy" Oct 30 17:02:39.735338 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:39.735330 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="e47df7d5-b3d9-4ba4-91a7-7a530ab85616" containerName="istio-proxy" Oct 30 17:02:39.740467 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:39.740451 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-830ba9/auth-test-model-830ba9-kserve-7cb5656f77-vshw4" Oct 30 17:02:39.779618 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:39.779573 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w6-830ba9\"/\"openshift-service-ca.crt\"" Oct 30 17:02:39.779787 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:39.779668 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w6-830ba9\"/\"default-dockercfg-g7bk7\"" Oct 30 17:02:39.779787 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:39.779707 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w6-830ba9\"/\"auth-test-model-830ba9-kserve-self-signed-certs\"" Oct 30 17:02:39.779787 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:39.779738 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w6-830ba9\"/\"kube-root-ca.crt\"" Oct 30 17:02:39.784840 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:39.784814 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w6-830ba9/auth-test-model-830ba9-kserve-7cb5656f77-vshw4"] Oct 30 17:02:39.812683 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:39.812648 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dafdc56a-7a9f-4ed2-b4ab-bf4e25f04437-home\") pod \"auth-test-model-830ba9-kserve-7cb5656f77-vshw4\" (UID: \"dafdc56a-7a9f-4ed2-b4ab-bf4e25f04437\") " pod="ai-tenant-e2e-shared-a-w6-830ba9/auth-test-model-830ba9-kserve-7cb5656f77-vshw4" Oct 30 17:02:39.812865 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:39.812840 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vvfn\" (UniqueName: \"kubernetes.io/projected/dafdc56a-7a9f-4ed2-b4ab-bf4e25f04437-kube-api-access-5vvfn\") pod \"auth-test-model-830ba9-kserve-7cb5656f77-vshw4\" (UID: \"dafdc56a-7a9f-4ed2-b4ab-bf4e25f04437\") " pod="ai-tenant-e2e-shared-a-w6-830ba9/auth-test-model-830ba9-kserve-7cb5656f77-vshw4" Oct 30 17:02:39.813017 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:39.812935 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dafdc56a-7a9f-4ed2-b4ab-bf4e25f04437-dshm\") pod \"auth-test-model-830ba9-kserve-7cb5656f77-vshw4\" (UID: \"dafdc56a-7a9f-4ed2-b4ab-bf4e25f04437\") " pod="ai-tenant-e2e-shared-a-w6-830ba9/auth-test-model-830ba9-kserve-7cb5656f77-vshw4" Oct 30 17:02:39.813121 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:39.813097 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dafdc56a-7a9f-4ed2-b4ab-bf4e25f04437-model-cache\") pod \"auth-test-model-830ba9-kserve-7cb5656f77-vshw4\" (UID: \"dafdc56a-7a9f-4ed2-b4ab-bf4e25f04437\") " pod="ai-tenant-e2e-shared-a-w6-830ba9/auth-test-model-830ba9-kserve-7cb5656f77-vshw4" Oct 30 17:02:39.813195 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:39.813133 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dafdc56a-7a9f-4ed2-b4ab-bf4e25f04437-tls-certs\") pod \"auth-test-model-830ba9-kserve-7cb5656f77-vshw4\" (UID: \"dafdc56a-7a9f-4ed2-b4ab-bf4e25f04437\") " pod="ai-tenant-e2e-shared-a-w6-830ba9/auth-test-model-830ba9-kserve-7cb5656f77-vshw4" Oct 30 17:02:39.813284 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:39.813267 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dafdc56a-7a9f-4ed2-b4ab-bf4e25f04437-tmp-dir\") pod \"auth-test-model-830ba9-kserve-7cb5656f77-vshw4\" (UID: \"dafdc56a-7a9f-4ed2-b4ab-bf4e25f04437\") " pod="ai-tenant-e2e-shared-a-w6-830ba9/auth-test-model-830ba9-kserve-7cb5656f77-vshw4" Oct 30 17:02:39.914953 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:39.914915 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dafdc56a-7a9f-4ed2-b4ab-bf4e25f04437-dshm\") pod \"auth-test-model-830ba9-kserve-7cb5656f77-vshw4\" (UID: \"dafdc56a-7a9f-4ed2-b4ab-bf4e25f04437\") " pod="ai-tenant-e2e-shared-a-w6-830ba9/auth-test-model-830ba9-kserve-7cb5656f77-vshw4" Oct 30 17:02:39.915123 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:39.915034 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dafdc56a-7a9f-4ed2-b4ab-bf4e25f04437-model-cache\") pod \"auth-test-model-830ba9-kserve-7cb5656f77-vshw4\" (UID: \"dafdc56a-7a9f-4ed2-b4ab-bf4e25f04437\") " pod="ai-tenant-e2e-shared-a-w6-830ba9/auth-test-model-830ba9-kserve-7cb5656f77-vshw4" Oct 30 17:02:39.915123 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:39.915062 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dafdc56a-7a9f-4ed2-b4ab-bf4e25f04437-tls-certs\") pod \"auth-test-model-830ba9-kserve-7cb5656f77-vshw4\" (UID: \"dafdc56a-7a9f-4ed2-b4ab-bf4e25f04437\") " pod="ai-tenant-e2e-shared-a-w6-830ba9/auth-test-model-830ba9-kserve-7cb5656f77-vshw4" Oct 30 17:02:39.915218 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:39.915140 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dafdc56a-7a9f-4ed2-b4ab-bf4e25f04437-tmp-dir\") pod \"auth-test-model-830ba9-kserve-7cb5656f77-vshw4\" (UID: \"dafdc56a-7a9f-4ed2-b4ab-bf4e25f04437\") " pod="ai-tenant-e2e-shared-a-w6-830ba9/auth-test-model-830ba9-kserve-7cb5656f77-vshw4" Oct 30 17:02:39.915218 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:39.915182 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dafdc56a-7a9f-4ed2-b4ab-bf4e25f04437-home\") pod \"auth-test-model-830ba9-kserve-7cb5656f77-vshw4\" (UID: \"dafdc56a-7a9f-4ed2-b4ab-bf4e25f04437\") " pod="ai-tenant-e2e-shared-a-w6-830ba9/auth-test-model-830ba9-kserve-7cb5656f77-vshw4" Oct 30 17:02:39.915312 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:39.915274 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vvfn\" (UniqueName: \"kubernetes.io/projected/dafdc56a-7a9f-4ed2-b4ab-bf4e25f04437-kube-api-access-5vvfn\") pod \"auth-test-model-830ba9-kserve-7cb5656f77-vshw4\" (UID: \"dafdc56a-7a9f-4ed2-b4ab-bf4e25f04437\") " pod="ai-tenant-e2e-shared-a-w6-830ba9/auth-test-model-830ba9-kserve-7cb5656f77-vshw4" Oct 30 17:02:39.915461 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:39.915437 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dafdc56a-7a9f-4ed2-b4ab-bf4e25f04437-model-cache\") pod \"auth-test-model-830ba9-kserve-7cb5656f77-vshw4\" (UID: \"dafdc56a-7a9f-4ed2-b4ab-bf4e25f04437\") " pod="ai-tenant-e2e-shared-a-w6-830ba9/auth-test-model-830ba9-kserve-7cb5656f77-vshw4" Oct 30 17:02:39.915699 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:39.915540 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dafdc56a-7a9f-4ed2-b4ab-bf4e25f04437-tmp-dir\") pod \"auth-test-model-830ba9-kserve-7cb5656f77-vshw4\" (UID: \"dafdc56a-7a9f-4ed2-b4ab-bf4e25f04437\") " pod="ai-tenant-e2e-shared-a-w6-830ba9/auth-test-model-830ba9-kserve-7cb5656f77-vshw4" Oct 30 17:02:39.915699 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:39.915673 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dafdc56a-7a9f-4ed2-b4ab-bf4e25f04437-home\") pod \"auth-test-model-830ba9-kserve-7cb5656f77-vshw4\" (UID: \"dafdc56a-7a9f-4ed2-b4ab-bf4e25f04437\") " pod="ai-tenant-e2e-shared-a-w6-830ba9/auth-test-model-830ba9-kserve-7cb5656f77-vshw4" Oct 30 17:02:39.917330 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:39.917307 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dafdc56a-7a9f-4ed2-b4ab-bf4e25f04437-dshm\") pod \"auth-test-model-830ba9-kserve-7cb5656f77-vshw4\" (UID: \"dafdc56a-7a9f-4ed2-b4ab-bf4e25f04437\") " pod="ai-tenant-e2e-shared-a-w6-830ba9/auth-test-model-830ba9-kserve-7cb5656f77-vshw4" Oct 30 17:02:39.917653 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:39.917637 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dafdc56a-7a9f-4ed2-b4ab-bf4e25f04437-tls-certs\") pod \"auth-test-model-830ba9-kserve-7cb5656f77-vshw4\" (UID: \"dafdc56a-7a9f-4ed2-b4ab-bf4e25f04437\") " pod="ai-tenant-e2e-shared-a-w6-830ba9/auth-test-model-830ba9-kserve-7cb5656f77-vshw4" Oct 30 17:02:40.002654 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:39.996301 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vvfn\" (UniqueName: \"kubernetes.io/projected/dafdc56a-7a9f-4ed2-b4ab-bf4e25f04437-kube-api-access-5vvfn\") pod \"auth-test-model-830ba9-kserve-7cb5656f77-vshw4\" (UID: \"dafdc56a-7a9f-4ed2-b4ab-bf4e25f04437\") " pod="ai-tenant-e2e-shared-a-w6-830ba9/auth-test-model-830ba9-kserve-7cb5656f77-vshw4" Oct 30 17:02:40.054456 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:40.054361 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-830ba9/auth-test-model-830ba9-kserve-7cb5656f77-vshw4" Oct 30 17:02:40.378806 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:40.378780 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w6-830ba9/auth-test-model-830ba9-kserve-7cb5656f77-vshw4"] Oct 30 17:02:40.379455 ip-10-0-131-136 kubenswrapper[5052]: W1030 17:02:40.379428 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddafdc56a_7a9f_4ed2_b4ab_bf4e25f04437.slice/crio-01ef032675d126119ae152229e8a11f10919c5e61a97c9174e0747a86c5d9c46 WatchSource:0}: Error finding container 01ef032675d126119ae152229e8a11f10919c5e61a97c9174e0747a86c5d9c46: Status 404 returned error can't find the container with id 01ef032675d126119ae152229e8a11f10919c5e61a97c9174e0747a86c5d9c46 Oct 30 17:02:41.172374 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:41.172341 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-830ba9/auth-test-model-830ba9-kserve-7cb5656f77-vshw4" event={"ID":"dafdc56a-7a9f-4ed2-b4ab-bf4e25f04437","Type":"ContainerStarted","Data":"7c744aee30d2410c8d1b97543c85976cceecbbe6af2c157bdc67ca94a0815e8e"} Oct 30 17:02:41.172833 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:41.172387 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-830ba9/auth-test-model-830ba9-kserve-7cb5656f77-vshw4" event={"ID":"dafdc56a-7a9f-4ed2-b4ab-bf4e25f04437","Type":"ContainerStarted","Data":"01ef032675d126119ae152229e8a11f10919c5e61a97c9174e0747a86c5d9c46"} Oct 30 17:02:41.217364 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:41.217303 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w6-830ba9/auth-test-model-830ba9-kserve-7cb5656f77-vshw4" podStartSLOduration=2.2172869139999998 podStartE2EDuration="2.217286914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:02:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:02:41.21719514 +0000 UTC m=+2516.744530362" watchObservedRunningTime="2026-10-30 17:02:41.217286914 +0000 UTC m=+2516.744622135" Oct 30 17:02:42.054488 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:42.054456 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w6-830ba9/auth-test-model-830ba9-kserve-7cb5656f77-vshw4" Oct 30 17:02:42.126074 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:02:42.126043 5052 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b967416_660d_4e1e_be9a_963007619616.slice/crio-a638ce3ad204fcc6ef0208a71e2f98b04b7a9ff9de546fa1d7867f0297fa296e.scope\": RecentStats: unable to find data in memory cache]" Oct 30 17:02:43.309044 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:02:43.309007 5052 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b967416_660d_4e1e_be9a_963007619616.slice/crio-a638ce3ad204fcc6ef0208a71e2f98b04b7a9ff9de546fa1d7867f0297fa296e.scope\": RecentStats: unable to find data in memory cache]" Oct 30 17:02:43.309490 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:02:43.309464 5052 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b967416_660d_4e1e_be9a_963007619616.slice/crio-a638ce3ad204fcc6ef0208a71e2f98b04b7a9ff9de546fa1d7867f0297fa296e.scope\": RecentStats: unable to find data in memory cache]" Oct 30 17:02:44.694673 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:02:44.694642 5052 fsHandler.go:119] failed to collect filesystem stats - rootDiskErr: could not stat "/var/lib/containers/storage/overlay/76e869b347bc53d4269044dc1521f2a86c3a3ef61150991e89fd8b7834ed98cf/diff" to get inode usage: stat /var/lib/containers/storage/overlay/76e869b347bc53d4269044dc1521f2a86c3a3ef61150991e89fd8b7834ed98cf/diff: no such file or directory, extraDiskErr: could not stat "/var/log/pods/openshift-ingress_payload-pre-processing-e2e-inf-a-ea77d8-584f58fcf-k88cs_2b967416-660d-4e1e-be9a-963007619616/payload-pre-processing/0.log" to get inode usage: stat /var/log/pods/openshift-ingress_payload-pre-processing-e2e-inf-a-ea77d8-584f58fcf-k88cs_2b967416-660d-4e1e-be9a-963007619616/payload-pre-processing/0.log: no such file or directory Oct 30 17:02:47.822632 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:47.822591 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cdwcj"] Oct 30 17:02:48.843917 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:02:48.843873 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 17:02:50.055076 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:50.055043 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w6-830ba9/auth-test-model-830ba9-kserve-7cb5656f77-vshw4" Oct 30 17:02:50.208916 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:50.208886 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w6-830ba9/auth-test-model-830ba9-kserve-7cb5656f77-vshw4" Oct 30 17:02:50.296562 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:02:50.296526 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w6-830ba9/auth-test-model-830ba9-kserve-7cb5656f77-vshw4" Oct 30 17:03:18.842193 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:03:18.842141 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 17:03:20.788197 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:03:20.788163 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w6-e5bfb0/auth-test-model-e5bfb0-kserve-8599ccd689-kdckc"] Oct 30 17:03:20.789287 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:03:20.789264 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="e47df7d5-b3d9-4ba4-91a7-7a530ab85616" containerName="istio-proxy" Oct 30 17:03:20.793147 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:03:20.793124 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-e5bfb0/auth-test-model-e5bfb0-kserve-8599ccd689-kdckc" Oct 30 17:03:20.824596 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:03:20.824551 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w6-e5bfb0\"/\"kube-root-ca.crt\"" Oct 30 17:03:20.824864 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:03:20.824831 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w6-e5bfb0\"/\"openshift-service-ca.crt\"" Oct 30 17:03:20.825078 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:03:20.825056 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w6-e5bfb0\"/\"auth-test-model-e5bfb0-kserve-self-signed-certs\"" Oct 30 17:03:20.827165 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:03:20.827141 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w6-e5bfb0/auth-test-model-e5bfb0-kserve-8599ccd689-kdckc"] Oct 30 17:03:20.831057 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:03:20.831029 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w6-e5bfb0\"/\"default-dockercfg-xdjwm\"" Oct 30 17:03:20.884391 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:03:20.884361 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jvr9\" (UniqueName: \"kubernetes.io/projected/64fa47f5-0840-429e-9dd6-2a4a0a20ce57-kube-api-access-8jvr9\") pod \"auth-test-model-e5bfb0-kserve-8599ccd689-kdckc\" (UID: \"64fa47f5-0840-429e-9dd6-2a4a0a20ce57\") " pod="ai-tenant-e2e-shared-b-w6-e5bfb0/auth-test-model-e5bfb0-kserve-8599ccd689-kdckc" Oct 30 17:03:20.884591 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:03:20.884522 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64fa47f5-0840-429e-9dd6-2a4a0a20ce57-tls-certs\") pod \"auth-test-model-e5bfb0-kserve-8599ccd689-kdckc\" (UID: \"64fa47f5-0840-429e-9dd6-2a4a0a20ce57\") " pod="ai-tenant-e2e-shared-b-w6-e5bfb0/auth-test-model-e5bfb0-kserve-8599ccd689-kdckc" Oct 30 17:03:20.884678 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:03:20.884598 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64fa47f5-0840-429e-9dd6-2a4a0a20ce57-home\") pod \"auth-test-model-e5bfb0-kserve-8599ccd689-kdckc\" (UID: \"64fa47f5-0840-429e-9dd6-2a4a0a20ce57\") " pod="ai-tenant-e2e-shared-b-w6-e5bfb0/auth-test-model-e5bfb0-kserve-8599ccd689-kdckc" Oct 30 17:03:20.884678 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:03:20.884657 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64fa47f5-0840-429e-9dd6-2a4a0a20ce57-tmp-dir\") pod \"auth-test-model-e5bfb0-kserve-8599ccd689-kdckc\" (UID: \"64fa47f5-0840-429e-9dd6-2a4a0a20ce57\") " pod="ai-tenant-e2e-shared-b-w6-e5bfb0/auth-test-model-e5bfb0-kserve-8599ccd689-kdckc" Oct 30 17:03:20.884792 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:03:20.884708 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64fa47f5-0840-429e-9dd6-2a4a0a20ce57-model-cache\") pod \"auth-test-model-e5bfb0-kserve-8599ccd689-kdckc\" (UID: \"64fa47f5-0840-429e-9dd6-2a4a0a20ce57\") " pod="ai-tenant-e2e-shared-b-w6-e5bfb0/auth-test-model-e5bfb0-kserve-8599ccd689-kdckc" Oct 30 17:03:20.884792 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:03:20.884733 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64fa47f5-0840-429e-9dd6-2a4a0a20ce57-dshm\") pod \"auth-test-model-e5bfb0-kserve-8599ccd689-kdckc\" (UID: \"64fa47f5-0840-429e-9dd6-2a4a0a20ce57\") " pod="ai-tenant-e2e-shared-b-w6-e5bfb0/auth-test-model-e5bfb0-kserve-8599ccd689-kdckc" Oct 30 17:03:20.986210 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:03:20.986159 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64fa47f5-0840-429e-9dd6-2a4a0a20ce57-tmp-dir\") pod \"auth-test-model-e5bfb0-kserve-8599ccd689-kdckc\" (UID: \"64fa47f5-0840-429e-9dd6-2a4a0a20ce57\") " pod="ai-tenant-e2e-shared-b-w6-e5bfb0/auth-test-model-e5bfb0-kserve-8599ccd689-kdckc" Oct 30 17:03:20.986411 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:03:20.986332 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64fa47f5-0840-429e-9dd6-2a4a0a20ce57-model-cache\") pod \"auth-test-model-e5bfb0-kserve-8599ccd689-kdckc\" (UID: \"64fa47f5-0840-429e-9dd6-2a4a0a20ce57\") " pod="ai-tenant-e2e-shared-b-w6-e5bfb0/auth-test-model-e5bfb0-kserve-8599ccd689-kdckc" Oct 30 17:03:20.986411 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:03:20.986391 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64fa47f5-0840-429e-9dd6-2a4a0a20ce57-dshm\") pod \"auth-test-model-e5bfb0-kserve-8599ccd689-kdckc\" (UID: \"64fa47f5-0840-429e-9dd6-2a4a0a20ce57\") " pod="ai-tenant-e2e-shared-b-w6-e5bfb0/auth-test-model-e5bfb0-kserve-8599ccd689-kdckc" Oct 30 17:03:20.986565 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:03:20.986549 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jvr9\" (UniqueName: \"kubernetes.io/projected/64fa47f5-0840-429e-9dd6-2a4a0a20ce57-kube-api-access-8jvr9\") pod \"auth-test-model-e5bfb0-kserve-8599ccd689-kdckc\" (UID: \"64fa47f5-0840-429e-9dd6-2a4a0a20ce57\") " pod="ai-tenant-e2e-shared-b-w6-e5bfb0/auth-test-model-e5bfb0-kserve-8599ccd689-kdckc" Oct 30 17:03:20.986666 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:03:20.986634 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64fa47f5-0840-429e-9dd6-2a4a0a20ce57-tmp-dir\") pod \"auth-test-model-e5bfb0-kserve-8599ccd689-kdckc\" (UID: \"64fa47f5-0840-429e-9dd6-2a4a0a20ce57\") " pod="ai-tenant-e2e-shared-b-w6-e5bfb0/auth-test-model-e5bfb0-kserve-8599ccd689-kdckc" Oct 30 17:03:20.986718 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:03:20.986674 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64fa47f5-0840-429e-9dd6-2a4a0a20ce57-model-cache\") pod \"auth-test-model-e5bfb0-kserve-8599ccd689-kdckc\" (UID: \"64fa47f5-0840-429e-9dd6-2a4a0a20ce57\") " pod="ai-tenant-e2e-shared-b-w6-e5bfb0/auth-test-model-e5bfb0-kserve-8599ccd689-kdckc" Oct 30 17:03:20.986782 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:03:20.986757 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64fa47f5-0840-429e-9dd6-2a4a0a20ce57-tls-certs\") pod \"auth-test-model-e5bfb0-kserve-8599ccd689-kdckc\" (UID: \"64fa47f5-0840-429e-9dd6-2a4a0a20ce57\") " pod="ai-tenant-e2e-shared-b-w6-e5bfb0/auth-test-model-e5bfb0-kserve-8599ccd689-kdckc" Oct 30 17:03:20.986865 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:03:20.986844 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64fa47f5-0840-429e-9dd6-2a4a0a20ce57-home\") pod \"auth-test-model-e5bfb0-kserve-8599ccd689-kdckc\" (UID: \"64fa47f5-0840-429e-9dd6-2a4a0a20ce57\") " pod="ai-tenant-e2e-shared-b-w6-e5bfb0/auth-test-model-e5bfb0-kserve-8599ccd689-kdckc" Oct 30 17:03:20.987245 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:03:20.987218 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64fa47f5-0840-429e-9dd6-2a4a0a20ce57-home\") pod \"auth-test-model-e5bfb0-kserve-8599ccd689-kdckc\" (UID: \"64fa47f5-0840-429e-9dd6-2a4a0a20ce57\") " pod="ai-tenant-e2e-shared-b-w6-e5bfb0/auth-test-model-e5bfb0-kserve-8599ccd689-kdckc" Oct 30 17:03:20.989041 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:03:20.989021 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64fa47f5-0840-429e-9dd6-2a4a0a20ce57-dshm\") pod \"auth-test-model-e5bfb0-kserve-8599ccd689-kdckc\" (UID: \"64fa47f5-0840-429e-9dd6-2a4a0a20ce57\") " pod="ai-tenant-e2e-shared-b-w6-e5bfb0/auth-test-model-e5bfb0-kserve-8599ccd689-kdckc" Oct 30 17:03:20.989168 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:03:20.989156 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64fa47f5-0840-429e-9dd6-2a4a0a20ce57-tls-certs\") pod \"auth-test-model-e5bfb0-kserve-8599ccd689-kdckc\" (UID: \"64fa47f5-0840-429e-9dd6-2a4a0a20ce57\") " pod="ai-tenant-e2e-shared-b-w6-e5bfb0/auth-test-model-e5bfb0-kserve-8599ccd689-kdckc" Oct 30 17:03:21.070649 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:03:21.070130 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jvr9\" (UniqueName: \"kubernetes.io/projected/64fa47f5-0840-429e-9dd6-2a4a0a20ce57-kube-api-access-8jvr9\") pod \"auth-test-model-e5bfb0-kserve-8599ccd689-kdckc\" (UID: \"64fa47f5-0840-429e-9dd6-2a4a0a20ce57\") " pod="ai-tenant-e2e-shared-b-w6-e5bfb0/auth-test-model-e5bfb0-kserve-8599ccd689-kdckc" Oct 30 17:03:21.126013 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:03:21.125927 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-e5bfb0/auth-test-model-e5bfb0-kserve-8599ccd689-kdckc" Oct 30 17:03:21.445928 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:03:21.445898 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w6-e5bfb0/auth-test-model-e5bfb0-kserve-8599ccd689-kdckc"] Oct 30 17:03:21.446657 ip-10-0-131-136 kubenswrapper[5052]: W1030 17:03:21.446628 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod64fa47f5_0840_429e_9dd6_2a4a0a20ce57.slice/crio-1438a9707802e06c08930d50bcc15eff78e2a6b21d00b5568cd2cc16a2bf22ca WatchSource:0}: Error finding container 1438a9707802e06c08930d50bcc15eff78e2a6b21d00b5568cd2cc16a2bf22ca: Status 404 returned error can't find the container with id 1438a9707802e06c08930d50bcc15eff78e2a6b21d00b5568cd2cc16a2bf22ca Oct 30 17:03:21.487940 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:03:21.487902 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-e5bfb0/auth-test-model-e5bfb0-kserve-8599ccd689-kdckc" event={"ID":"64fa47f5-0840-429e-9dd6-2a4a0a20ce57","Type":"ContainerStarted","Data":"1438a9707802e06c08930d50bcc15eff78e2a6b21d00b5568cd2cc16a2bf22ca"} Oct 30 17:03:22.496483 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:03:22.496433 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-e5bfb0/auth-test-model-e5bfb0-kserve-8599ccd689-kdckc" event={"ID":"64fa47f5-0840-429e-9dd6-2a4a0a20ce57","Type":"ContainerStarted","Data":"4fd42315b563a7869560037df1845f10783f52712c3d75380928f6c46dbd3894"} Oct 30 17:03:22.541304 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:03:22.541255 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w6-e5bfb0/auth-test-model-e5bfb0-kserve-8599ccd689-kdckc" podStartSLOduration=2.541241849 podStartE2EDuration="2.541241849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:03:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:03:22.540327721 +0000 UTC m=+2558.067662938" watchObservedRunningTime="2026-10-30 17:03:22.541241849 +0000 UTC m=+2558.068577108" Oct 30 17:03:23.126698 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:03:23.126666 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w6-e5bfb0/auth-test-model-e5bfb0-kserve-8599ccd689-kdckc" Oct 30 17:03:31.126281 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:03:31.126237 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w6-e5bfb0/auth-test-model-e5bfb0-kserve-8599ccd689-kdckc" Oct 30 17:03:31.140088 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:03:31.140053 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w6-e5bfb0/auth-test-model-e5bfb0-kserve-8599ccd689-kdckc" Oct 30 17:03:31.631927 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:03:31.631890 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w6-e5bfb0/auth-test-model-e5bfb0-kserve-8599ccd689-kdckc" Oct 30 17:03:48.842669 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:03:48.842634 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 17:03:56.406466 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:03:56.406424 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-hmw2l"] Oct 30 17:03:56.412167 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:03:56.412143 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-hmw2l" Oct 30 17:03:56.430416 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:03:56.430385 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w5-1b7bb3-openshift-default-dockercfg-gqvnj\"" Oct 30 17:03:56.454920 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:03:56.454888 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-hmw2l"] Oct 30 17:03:56.502275 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:03:56.502237 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/952ac6ad-c9ff-4078-b361-201e411053de-istiod-ca-cert\") pod \"e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-hmw2l\" (UID: \"952ac6ad-c9ff-4078-b361-201e411053de\") " pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-hmw2l" Oct 30 17:03:56.502488 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:03:56.502285 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/952ac6ad-c9ff-4078-b361-201e411053de-istio-podinfo\") pod \"e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-hmw2l\" (UID: \"952ac6ad-c9ff-4078-b361-201e411053de\") " pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-hmw2l" Oct 30 17:03:56.502549 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:03:56.502482 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/952ac6ad-c9ff-4078-b361-201e411053de-istio-data\") pod \"e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-hmw2l\" (UID: \"952ac6ad-c9ff-4078-b361-201e411053de\") " pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-hmw2l" Oct 30 17:03:56.502549 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:03:56.502510 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/952ac6ad-c9ff-4078-b361-201e411053de-workload-socket\") pod \"e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-hmw2l\" (UID: \"952ac6ad-c9ff-4078-b361-201e411053de\") " pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-hmw2l" Oct 30 17:03:56.502708 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:03:56.502683 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsvmf\" (UniqueName: \"kubernetes.io/projected/952ac6ad-c9ff-4078-b361-201e411053de-kube-api-access-vsvmf\") pod \"e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-hmw2l\" (UID: \"952ac6ad-c9ff-4078-b361-201e411053de\") " pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-hmw2l" Oct 30 17:03:56.502823 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:03:56.502807 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/952ac6ad-c9ff-4078-b361-201e411053de-istio-token\") pod \"e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-hmw2l\" (UID: \"952ac6ad-c9ff-4078-b361-201e411053de\") " pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-hmw2l" Oct 30 17:03:56.502860 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:03:56.502847 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/952ac6ad-c9ff-4078-b361-201e411053de-credential-socket\") pod \"e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-hmw2l\" (UID: \"952ac6ad-c9ff-4078-b361-201e411053de\") " pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-hmw2l" Oct 30 17:03:56.502895 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:03:56.502874 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/952ac6ad-c9ff-4078-b361-201e411053de-workload-certs\") pod \"e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-hmw2l\" (UID: \"952ac6ad-c9ff-4078-b361-201e411053de\") " pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-hmw2l" Oct 30 17:03:56.502968 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:03:56.502957 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/952ac6ad-c9ff-4078-b361-201e411053de-istio-envoy\") pod \"e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-hmw2l\" (UID: \"952ac6ad-c9ff-4078-b361-201e411053de\") " pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-hmw2l" Oct 30 17:03:56.604485 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:03:56.604450 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/952ac6ad-c9ff-4078-b361-201e411053de-istio-data\") pod \"e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-hmw2l\" (UID: \"952ac6ad-c9ff-4078-b361-201e411053de\") " pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-hmw2l" Oct 30 17:03:56.604485 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:03:56.604487 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/952ac6ad-c9ff-4078-b361-201e411053de-workload-socket\") pod \"e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-hmw2l\" (UID: \"952ac6ad-c9ff-4078-b361-201e411053de\") " pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-hmw2l" Oct 30 17:03:56.604709 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:03:56.604535 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsvmf\" (UniqueName: \"kubernetes.io/projected/952ac6ad-c9ff-4078-b361-201e411053de-kube-api-access-vsvmf\") pod \"e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-hmw2l\" (UID: \"952ac6ad-c9ff-4078-b361-201e411053de\") " pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-hmw2l" Oct 30 17:03:56.604709 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:03:56.604637 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/952ac6ad-c9ff-4078-b361-201e411053de-istio-token\") pod \"e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-hmw2l\" (UID: \"952ac6ad-c9ff-4078-b361-201e411053de\") " pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-hmw2l" Oct 30 17:03:56.604709 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:03:56.604669 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/952ac6ad-c9ff-4078-b361-201e411053de-credential-socket\") pod \"e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-hmw2l\" (UID: \"952ac6ad-c9ff-4078-b361-201e411053de\") " pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-hmw2l" Oct 30 17:03:56.604709 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:03:56.604692 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/952ac6ad-c9ff-4078-b361-201e411053de-workload-certs\") pod \"e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-hmw2l\" (UID: \"952ac6ad-c9ff-4078-b361-201e411053de\") " pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-hmw2l" Oct 30 17:03:56.604896 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:03:56.604760 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/952ac6ad-c9ff-4078-b361-201e411053de-istio-envoy\") pod \"e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-hmw2l\" (UID: \"952ac6ad-c9ff-4078-b361-201e411053de\") " pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-hmw2l" Oct 30 17:03:56.604896 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:03:56.604815 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/952ac6ad-c9ff-4078-b361-201e411053de-istiod-ca-cert\") pod \"e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-hmw2l\" (UID: \"952ac6ad-c9ff-4078-b361-201e411053de\") " pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-hmw2l" Oct 30 17:03:56.604896 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:03:56.604843 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/952ac6ad-c9ff-4078-b361-201e411053de-istio-data\") pod \"e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-hmw2l\" (UID: \"952ac6ad-c9ff-4078-b361-201e411053de\") " pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-hmw2l" Oct 30 17:03:56.604896 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:03:56.604847 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/952ac6ad-c9ff-4078-b361-201e411053de-istio-podinfo\") pod \"e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-hmw2l\" (UID: \"952ac6ad-c9ff-4078-b361-201e411053de\") " pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-hmw2l" Oct 30 17:03:56.605128 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:03:56.604914 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/952ac6ad-c9ff-4078-b361-201e411053de-workload-socket\") pod \"e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-hmw2l\" (UID: \"952ac6ad-c9ff-4078-b361-201e411053de\") " pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-hmw2l" Oct 30 17:03:56.605128 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:03:56.605048 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/952ac6ad-c9ff-4078-b361-201e411053de-credential-socket\") pod \"e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-hmw2l\" (UID: \"952ac6ad-c9ff-4078-b361-201e411053de\") " pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-hmw2l" Oct 30 17:03:56.605293 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:03:56.605272 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/952ac6ad-c9ff-4078-b361-201e411053de-workload-certs\") pod \"e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-hmw2l\" (UID: \"952ac6ad-c9ff-4078-b361-201e411053de\") " pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-hmw2l" Oct 30 17:03:56.605664 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:03:56.605640 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/952ac6ad-c9ff-4078-b361-201e411053de-istiod-ca-cert\") pod \"e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-hmw2l\" (UID: \"952ac6ad-c9ff-4078-b361-201e411053de\") " pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-hmw2l" Oct 30 17:03:56.607180 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:03:56.607158 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/952ac6ad-c9ff-4078-b361-201e411053de-istio-envoy\") pod \"e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-hmw2l\" (UID: \"952ac6ad-c9ff-4078-b361-201e411053de\") " pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-hmw2l" Oct 30 17:03:56.607498 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:03:56.607457 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/952ac6ad-c9ff-4078-b361-201e411053de-istio-podinfo\") pod \"e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-hmw2l\" (UID: \"952ac6ad-c9ff-4078-b361-201e411053de\") " pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-hmw2l" Oct 30 17:03:56.630203 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:03:56.630168 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/952ac6ad-c9ff-4078-b361-201e411053de-istio-token\") pod \"e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-hmw2l\" (UID: \"952ac6ad-c9ff-4078-b361-201e411053de\") " pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-hmw2l" Oct 30 17:03:56.630345 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:03:56.630319 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsvmf\" (UniqueName: \"kubernetes.io/projected/952ac6ad-c9ff-4078-b361-201e411053de-kube-api-access-vsvmf\") pod \"e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-hmw2l\" (UID: \"952ac6ad-c9ff-4078-b361-201e411053de\") " pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-hmw2l" Oct 30 17:03:56.731942 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:03:56.731807 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-hmw2l" Oct 30 17:03:57.104160 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:03:57.104130 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-hmw2l"] Oct 30 17:03:57.105783 ip-10-0-131-136 kubenswrapper[5052]: W1030 17:03:57.105754 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod952ac6ad_c9ff_4078_b361_201e411053de.slice/crio-02bf8af5b14b9f869522fab434a6cdacda3b7258d150632f042a61d77f8b7849 WatchSource:0}: Error finding container 02bf8af5b14b9f869522fab434a6cdacda3b7258d150632f042a61d77f8b7849: Status 404 returned error can't find the container with id 02bf8af5b14b9f869522fab434a6cdacda3b7258d150632f042a61d77f8b7849 Oct 30 17:03:57.109018 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:03:57.108977 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:03:57.109117 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:03:57.109061 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:03:57.109117 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:03:57.109102 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:03:57.826343 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:03:57.826302 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-hmw2l" event={"ID":"952ac6ad-c9ff-4078-b361-201e411053de","Type":"ContainerStarted","Data":"5299695c33208f41d7994a89ac6c857de3b4177d02fb5c5b37705b4a012ce689"} Oct 30 17:03:57.826776 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:03:57.826367 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-hmw2l" event={"ID":"952ac6ad-c9ff-4078-b361-201e411053de","Type":"ContainerStarted","Data":"02bf8af5b14b9f869522fab434a6cdacda3b7258d150632f042a61d77f8b7849"} Oct 30 17:03:57.888338 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:03:57.888277 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-hmw2l" podStartSLOduration=1.8882625050000001 podStartE2EDuration="1.888262505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:03:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:03:57.886564117 +0000 UTC m=+2593.413899337" watchObservedRunningTime="2026-10-30 17:03:57.888262505 +0000 UTC m=+2593.415597723" Oct 30 17:03:58.749463 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:03:58.749425 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-hmw2l" Oct 30 17:03:58.749846 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:03:58.749821 5052 patch_prober.go:29] interesting pod/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-hmw2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Oct 30 17:03:58.750016 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:03:58.749992 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-hmw2l" podUID="952ac6ad-c9ff-4078-b361-201e411053de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Oct 30 17:03:59.747604 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:03:59.746981 5052 patch_prober.go:29] interesting pod/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-hmw2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Oct 30 17:03:59.747604 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:03:59.747042 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-hmw2l" podUID="952ac6ad-c9ff-4078-b361-201e411053de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Oct 30 17:04:00.745605 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:04:00.738899 5052 patch_prober.go:29] interesting pod/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-hmw2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Oct 30 17:04:00.745605 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:04:00.738961 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-hmw2l" podUID="952ac6ad-c9ff-4078-b361-201e411053de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Oct 30 17:04:01.752605 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:04:01.750904 5052 patch_prober.go:29] interesting pod/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-hmw2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Oct 30 17:04:01.752605 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:04:01.750966 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-hmw2l" podUID="952ac6ad-c9ff-4078-b361-201e411053de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Oct 30 17:04:02.747603 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:04:02.746673 5052 patch_prober.go:29] interesting pod/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-hmw2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Oct 30 17:04:02.747603 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:04:02.746737 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-hmw2l" podUID="952ac6ad-c9ff-4078-b361-201e411053de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Oct 30 17:04:03.738101 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:04:03.738072 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-hmw2l" Oct 30 17:04:03.738532 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:04:03.738412 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-hmw2l" Oct 30 17:04:03.739146 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:04:03.739126 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-hmw2l" Oct 30 17:04:16.412678 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:04:16.412636 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cdwcj"] Oct 30 17:04:18.963047 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:04:18.963007 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 17:04:34.546230 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:04:34.546181 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b16279-d87cd4586-gmr6p"] Oct 30 17:04:34.551080 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:04:34.551058 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b16279-d87cd4586-gmr6p" Oct 30 17:04:34.570537 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:04:34.570508 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b16279-d87cd4586-gmr6p"] Oct 30 17:04:34.579473 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:04:34.579443 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-b16279\"" Oct 30 17:04:34.579473 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:04:34.579448 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-b16279-dockercfg-2hncv\"" Oct 30 17:04:34.709476 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:04:34.709431 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nldgp\" (UniqueName: \"kubernetes.io/projected/49192ccf-145d-47db-b495-46cd4daeb9cf-kube-api-access-nldgp\") pod \"payload-processing-e2e-ipp-a-b16279-d87cd4586-gmr6p\" (UID: \"49192ccf-145d-47db-b495-46cd4daeb9cf\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b16279-d87cd4586-gmr6p" Oct 30 17:04:34.709863 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:04:34.709830 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49192ccf-145d-47db-b495-46cd4daeb9cf-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-b16279-d87cd4586-gmr6p\" (UID: \"49192ccf-145d-47db-b495-46cd4daeb9cf\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b16279-d87cd4586-gmr6p" Oct 30 17:04:34.824705 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:04:34.817552 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49192ccf-145d-47db-b495-46cd4daeb9cf-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-b16279-d87cd4586-gmr6p\" (UID: \"49192ccf-145d-47db-b495-46cd4daeb9cf\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b16279-d87cd4586-gmr6p" Oct 30 17:04:34.825149 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:04:34.825117 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nldgp\" (UniqueName: \"kubernetes.io/projected/49192ccf-145d-47db-b495-46cd4daeb9cf-kube-api-access-nldgp\") pod \"payload-processing-e2e-ipp-a-b16279-d87cd4586-gmr6p\" (UID: \"49192ccf-145d-47db-b495-46cd4daeb9cf\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b16279-d87cd4586-gmr6p" Oct 30 17:04:34.825686 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:04:34.823484 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49192ccf-145d-47db-b495-46cd4daeb9cf-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-b16279-d87cd4586-gmr6p\" (UID: \"49192ccf-145d-47db-b495-46cd4daeb9cf\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b16279-d87cd4586-gmr6p" Oct 30 17:04:34.881767 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:04:34.880663 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nldgp\" (UniqueName: \"kubernetes.io/projected/49192ccf-145d-47db-b495-46cd4daeb9cf-kube-api-access-nldgp\") pod \"payload-processing-e2e-ipp-a-b16279-d87cd4586-gmr6p\" (UID: \"49192ccf-145d-47db-b495-46cd4daeb9cf\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b16279-d87cd4586-gmr6p" Oct 30 17:04:35.167500 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:04:35.167469 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b16279-d87cd4586-gmr6p" Oct 30 17:04:35.482706 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:04:35.482669 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b16279-d87cd4586-gmr6p"] Oct 30 17:04:35.483518 ip-10-0-131-136 kubenswrapper[5052]: W1030 17:04:35.483484 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49192ccf_145d_47db_b495_46cd4daeb9cf.slice/crio-bb9f34e846e0a0f1894fc27378ee70757bed885acd15b5c4cbf5a0c206490ddb WatchSource:0}: Error finding container bb9f34e846e0a0f1894fc27378ee70757bed885acd15b5c4cbf5a0c206490ddb: Status 404 returned error can't find the container with id bb9f34e846e0a0f1894fc27378ee70757bed885acd15b5c4cbf5a0c206490ddb Oct 30 17:04:36.212218 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:04:36.212177 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b16279-d87cd4586-gmr6p" event={"ID":"49192ccf-145d-47db-b495-46cd4daeb9cf","Type":"ContainerStarted","Data":"33d1f191e869e2df72f712d02e97e83148efa2edd2d4963015d85799dfe08c97"} Oct 30 17:04:36.212688 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:04:36.212239 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b16279-d87cd4586-gmr6p" event={"ID":"49192ccf-145d-47db-b495-46cd4daeb9cf","Type":"ContainerStarted","Data":"bb9f34e846e0a0f1894fc27378ee70757bed885acd15b5c4cbf5a0c206490ddb"} Oct 30 17:04:36.212688 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:04:36.212261 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-b16279-d87cd4586-gmr6p" Oct 30 17:04:36.264224 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:04:36.264147 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-b16279-d87cd4586-gmr6p" podStartSLOduration=2.264118565 podStartE2EDuration="2.264118565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:04:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:04:36.26196595 +0000 UTC m=+2631.789301170" watchObservedRunningTime="2026-10-30 17:04:36.264118565 +0000 UTC m=+2631.791453810" Oct 30 17:04:39.509135 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:04:39.509099 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-lhnp2"] Oct 30 17:04:39.514062 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:04:39.514042 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-lhnp2" Oct 30 17:04:39.544919 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:04:39.544893 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-c8db42-openshift-default-dockercfg-nsgbm\"" Oct 30 17:04:39.553164 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:04:39.553135 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-lhnp2"] Oct 30 17:04:39.598632 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:04:39.598562 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mbcw\" (UniqueName: \"kubernetes.io/projected/523c6f22-3610-4d8f-a048-acb661bb23b5-kube-api-access-7mbcw\") pod \"e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-lhnp2\" (UID: \"523c6f22-3610-4d8f-a048-acb661bb23b5\") " pod="openshift-ingress/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-lhnp2" Oct 30 17:04:39.598781 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:04:39.598699 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/523c6f22-3610-4d8f-a048-acb661bb23b5-istio-podinfo\") pod \"e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-lhnp2\" (UID: \"523c6f22-3610-4d8f-a048-acb661bb23b5\") " pod="openshift-ingress/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-lhnp2" Oct 30 17:04:39.598781 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:04:39.598739 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/523c6f22-3610-4d8f-a048-acb661bb23b5-workload-socket\") pod \"e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-lhnp2\" (UID: \"523c6f22-3610-4d8f-a048-acb661bb23b5\") " pod="openshift-ingress/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-lhnp2" Oct 30 17:04:39.598856 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:04:39.598778 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/523c6f22-3610-4d8f-a048-acb661bb23b5-workload-certs\") pod \"e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-lhnp2\" (UID: \"523c6f22-3610-4d8f-a048-acb661bb23b5\") " pod="openshift-ingress/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-lhnp2" Oct 30 17:04:39.598910 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:04:39.598890 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/523c6f22-3610-4d8f-a048-acb661bb23b5-istiod-ca-cert\") pod \"e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-lhnp2\" (UID: \"523c6f22-3610-4d8f-a048-acb661bb23b5\") " pod="openshift-ingress/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-lhnp2" Oct 30 17:04:39.598955 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:04:39.598924 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/523c6f22-3610-4d8f-a048-acb661bb23b5-istio-envoy\") pod \"e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-lhnp2\" (UID: \"523c6f22-3610-4d8f-a048-acb661bb23b5\") " pod="openshift-ingress/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-lhnp2" Oct 30 17:04:39.598997 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:04:39.598987 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/523c6f22-3610-4d8f-a048-acb661bb23b5-istio-data\") pod \"e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-lhnp2\" (UID: \"523c6f22-3610-4d8f-a048-acb661bb23b5\") " pod="openshift-ingress/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-lhnp2" Oct 30 17:04:39.599036 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:04:39.599010 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/523c6f22-3610-4d8f-a048-acb661bb23b5-istio-token\") pod \"e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-lhnp2\" (UID: \"523c6f22-3610-4d8f-a048-acb661bb23b5\") " pod="openshift-ingress/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-lhnp2" Oct 30 17:04:39.599148 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:04:39.599132 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/523c6f22-3610-4d8f-a048-acb661bb23b5-credential-socket\") pod \"e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-lhnp2\" (UID: \"523c6f22-3610-4d8f-a048-acb661bb23b5\") " pod="openshift-ingress/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-lhnp2" Oct 30 17:04:39.700942 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:04:39.700911 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/523c6f22-3610-4d8f-a048-acb661bb23b5-credential-socket\") pod \"e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-lhnp2\" (UID: \"523c6f22-3610-4d8f-a048-acb661bb23b5\") " pod="openshift-ingress/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-lhnp2" Oct 30 17:04:39.701109 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:04:39.701009 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mbcw\" (UniqueName: \"kubernetes.io/projected/523c6f22-3610-4d8f-a048-acb661bb23b5-kube-api-access-7mbcw\") pod \"e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-lhnp2\" (UID: \"523c6f22-3610-4d8f-a048-acb661bb23b5\") " pod="openshift-ingress/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-lhnp2" Oct 30 17:04:39.701109 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:04:39.701071 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/523c6f22-3610-4d8f-a048-acb661bb23b5-istio-podinfo\") pod \"e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-lhnp2\" (UID: \"523c6f22-3610-4d8f-a048-acb661bb23b5\") " pod="openshift-ingress/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-lhnp2" Oct 30 17:04:39.701109 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:04:39.701104 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/523c6f22-3610-4d8f-a048-acb661bb23b5-workload-socket\") pod \"e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-lhnp2\" (UID: \"523c6f22-3610-4d8f-a048-acb661bb23b5\") " pod="openshift-ingress/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-lhnp2" Oct 30 17:04:39.701239 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:04:39.701177 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/523c6f22-3610-4d8f-a048-acb661bb23b5-workload-certs\") pod \"e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-lhnp2\" (UID: \"523c6f22-3610-4d8f-a048-acb661bb23b5\") " pod="openshift-ingress/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-lhnp2" Oct 30 17:04:39.701355 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:04:39.701273 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/523c6f22-3610-4d8f-a048-acb661bb23b5-istiod-ca-cert\") pod \"e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-lhnp2\" (UID: \"523c6f22-3610-4d8f-a048-acb661bb23b5\") " pod="openshift-ingress/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-lhnp2" Oct 30 17:04:39.701355 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:04:39.701309 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/523c6f22-3610-4d8f-a048-acb661bb23b5-istio-envoy\") pod \"e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-lhnp2\" (UID: \"523c6f22-3610-4d8f-a048-acb661bb23b5\") " pod="openshift-ingress/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-lhnp2" Oct 30 17:04:39.701473 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:04:39.701373 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/523c6f22-3610-4d8f-a048-acb661bb23b5-credential-socket\") pod \"e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-lhnp2\" (UID: \"523c6f22-3610-4d8f-a048-acb661bb23b5\") " pod="openshift-ingress/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-lhnp2" Oct 30 17:04:39.701473 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:04:39.701381 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/523c6f22-3610-4d8f-a048-acb661bb23b5-istio-data\") pod \"e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-lhnp2\" (UID: \"523c6f22-3610-4d8f-a048-acb661bb23b5\") " pod="openshift-ingress/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-lhnp2" Oct 30 17:04:39.701473 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:04:39.701432 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/523c6f22-3610-4d8f-a048-acb661bb23b5-istio-token\") pod \"e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-lhnp2\" (UID: \"523c6f22-3610-4d8f-a048-acb661bb23b5\") " pod="openshift-ingress/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-lhnp2" Oct 30 17:04:39.701680 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:04:39.701472 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/523c6f22-3610-4d8f-a048-acb661bb23b5-workload-socket\") pod \"e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-lhnp2\" (UID: \"523c6f22-3610-4d8f-a048-acb661bb23b5\") " pod="openshift-ingress/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-lhnp2" Oct 30 17:04:39.701680 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:04:39.701569 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/523c6f22-3610-4d8f-a048-acb661bb23b5-workload-certs\") pod \"e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-lhnp2\" (UID: \"523c6f22-3610-4d8f-a048-acb661bb23b5\") " pod="openshift-ingress/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-lhnp2" Oct 30 17:04:39.701940 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:04:39.701916 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/523c6f22-3610-4d8f-a048-acb661bb23b5-istio-data\") pod \"e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-lhnp2\" (UID: \"523c6f22-3610-4d8f-a048-acb661bb23b5\") " pod="openshift-ingress/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-lhnp2" Oct 30 17:04:39.702104 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:04:39.702083 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/523c6f22-3610-4d8f-a048-acb661bb23b5-istiod-ca-cert\") pod \"e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-lhnp2\" (UID: \"523c6f22-3610-4d8f-a048-acb661bb23b5\") " pod="openshift-ingress/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-lhnp2" Oct 30 17:04:39.703968 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:04:39.703942 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/523c6f22-3610-4d8f-a048-acb661bb23b5-istio-envoy\") pod \"e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-lhnp2\" (UID: \"523c6f22-3610-4d8f-a048-acb661bb23b5\") " pod="openshift-ingress/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-lhnp2" Oct 30 17:04:39.704090 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:04:39.704074 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/523c6f22-3610-4d8f-a048-acb661bb23b5-istio-podinfo\") pod \"e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-lhnp2\" (UID: \"523c6f22-3610-4d8f-a048-acb661bb23b5\") " pod="openshift-ingress/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-lhnp2" Oct 30 17:04:39.741165 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:04:39.741130 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/523c6f22-3610-4d8f-a048-acb661bb23b5-istio-token\") pod \"e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-lhnp2\" (UID: \"523c6f22-3610-4d8f-a048-acb661bb23b5\") " pod="openshift-ingress/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-lhnp2" Oct 30 17:04:39.747607 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:04:39.742261 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mbcw\" (UniqueName: \"kubernetes.io/projected/523c6f22-3610-4d8f-a048-acb661bb23b5-kube-api-access-7mbcw\") pod \"e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-lhnp2\" (UID: \"523c6f22-3610-4d8f-a048-acb661bb23b5\") " pod="openshift-ingress/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-lhnp2" Oct 30 17:04:39.828920 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:04:39.828827 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-lhnp2" Oct 30 17:04:40.158895 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:04:40.158863 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-lhnp2"] Oct 30 17:04:40.159252 ip-10-0-131-136 kubenswrapper[5052]: W1030 17:04:40.159225 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod523c6f22_3610_4d8f_a048_acb661bb23b5.slice/crio-ec959e07938e0cf97fd08ff8aa55404910a2e49ccc71acac6ca8fd3e40cfea07 WatchSource:0}: Error finding container ec959e07938e0cf97fd08ff8aa55404910a2e49ccc71acac6ca8fd3e40cfea07: Status 404 returned error can't find the container with id ec959e07938e0cf97fd08ff8aa55404910a2e49ccc71acac6ca8fd3e40cfea07 Oct 30 17:04:40.162070 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:04:40.162035 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:04:40.162167 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:04:40.162100 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:04:40.162167 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:04:40.162130 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:04:40.242677 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:04:40.242644 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-lhnp2" event={"ID":"523c6f22-3610-4d8f-a048-acb661bb23b5","Type":"ContainerStarted","Data":"84641a29cb8d4fbba4808b2e6538c65d21414b19dcdf929c0de0923de41a3b57"} Oct 30 17:04:40.242799 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:04:40.242699 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-lhnp2" event={"ID":"523c6f22-3610-4d8f-a048-acb661bb23b5","Type":"ContainerStarted","Data":"ec959e07938e0cf97fd08ff8aa55404910a2e49ccc71acac6ca8fd3e40cfea07"} Oct 30 17:04:40.292455 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:04:40.292391 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-lhnp2" podStartSLOduration=1.292372123 podStartE2EDuration="1.292372123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:04:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:04:40.289720259 +0000 UTC m=+2635.817055473" watchObservedRunningTime="2026-10-30 17:04:40.292372123 +0000 UTC m=+2635.819707346" Oct 30 17:04:40.829553 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:04:40.829517 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-lhnp2" Oct 30 17:04:41.854853 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:04:41.854768 5052 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-lhnp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body= Oct 30 17:04:41.855326 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:04:41.855088 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-lhnp2" podUID="523c6f22-3610-4d8f-a048-acb661bb23b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" Oct 30 17:04:42.852756 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:04:42.852718 5052 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-lhnp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body= Oct 30 17:04:42.852979 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:04:42.852837 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-lhnp2" podUID="523c6f22-3610-4d8f-a048-acb661bb23b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" Oct 30 17:04:43.829233 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:04:43.829175 5052 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-lhnp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body= Oct 30 17:04:43.829878 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:04:43.829823 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-lhnp2" podUID="523c6f22-3610-4d8f-a048-acb661bb23b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" Oct 30 17:04:44.852720 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:04:44.852685 5052 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-lhnp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body= Oct 30 17:04:44.853201 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:04:44.853173 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-lhnp2" podUID="523c6f22-3610-4d8f-a048-acb661bb23b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" Oct 30 17:04:45.834800 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:04:45.834749 5052 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-lhnp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body= Oct 30 17:04:45.835125 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:04:45.835079 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-lhnp2" podUID="523c6f22-3610-4d8f-a048-acb661bb23b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" Oct 30 17:04:47.065926 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:04:47.065886 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-lhnp2" Oct 30 17:04:47.066920 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:04:47.066891 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-lhnp2" Oct 30 17:04:47.069081 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:04:47.069045 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-lhnp2" Oct 30 17:04:47.223146 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:04:47.223114 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-b16279-d87cd4586-gmr6p" Oct 30 17:04:48.845149 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:04:48.845102 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 17:05:08.143705 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:05:08.143672 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-9e533f/auth-test-model-9e533f-kserve-6dbc95dc5c-rfqgk"] Oct 30 17:05:18.841573 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:05:18.841536 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 17:05:23.490633 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:05:23.490555 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-9e533f/auth-test-model-9e533f-kserve-6dbc95dc5c-rfqgk" podUID="321c0696-5cc2-4038-8783-5d92da061c2d" containerName="main" containerID="cri-o://ac7dc35476d346b2b331b560b82f81313e935c61cb9267acb1b287ccab94b489" gracePeriod=45 Oct 30 17:05:23.787650 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:05:23.787623 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-9e533f/auth-test-model-9e533f-kserve-6dbc95dc5c-rfqgk" Oct 30 17:05:23.862509 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:05:23.862475 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/321c0696-5cc2-4038-8783-5d92da061c2d-kube-api-access-njxtd\" (UniqueName: \"kubernetes.io/projected/321c0696-5cc2-4038-8783-5d92da061c2d-kube-api-access-njxtd\") pod \"321c0696-5cc2-4038-8783-5d92da061c2d\" (UID: \"321c0696-5cc2-4038-8783-5d92da061c2d\") " Oct 30 17:05:23.862715 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:05:23.862545 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/321c0696-5cc2-4038-8783-5d92da061c2d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/321c0696-5cc2-4038-8783-5d92da061c2d-tmp-dir\") pod \"321c0696-5cc2-4038-8783-5d92da061c2d\" (UID: \"321c0696-5cc2-4038-8783-5d92da061c2d\") " Oct 30 17:05:23.862715 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:05:23.862577 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/321c0696-5cc2-4038-8783-5d92da061c2d-tls-certs\" (UniqueName: \"kubernetes.io/secret/321c0696-5cc2-4038-8783-5d92da061c2d-tls-certs\") pod \"321c0696-5cc2-4038-8783-5d92da061c2d\" (UID: \"321c0696-5cc2-4038-8783-5d92da061c2d\") " Oct 30 17:05:23.862886 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:05:23.862770 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/321c0696-5cc2-4038-8783-5d92da061c2d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/321c0696-5cc2-4038-8783-5d92da061c2d-model-cache\") pod \"321c0696-5cc2-4038-8783-5d92da061c2d\" (UID: \"321c0696-5cc2-4038-8783-5d92da061c2d\") " Oct 30 17:05:23.862969 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:05:23.862908 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/321c0696-5cc2-4038-8783-5d92da061c2d-home\" (UniqueName: \"kubernetes.io/empty-dir/321c0696-5cc2-4038-8783-5d92da061c2d-home\") pod \"321c0696-5cc2-4038-8783-5d92da061c2d\" (UID: \"321c0696-5cc2-4038-8783-5d92da061c2d\") " Oct 30 17:05:23.863031 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:05:23.862984 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/321c0696-5cc2-4038-8783-5d92da061c2d-model-cache" pod "321c0696-5cc2-4038-8783-5d92da061c2d" (UID: "321c0696-5cc2-4038-8783-5d92da061c2d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:05:23.863125 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:05:23.863102 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/321c0696-5cc2-4038-8783-5d92da061c2d-tmp-dir" pod "321c0696-5cc2-4038-8783-5d92da061c2d" (UID: "321c0696-5cc2-4038-8783-5d92da061c2d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:05:23.863125 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:05:23.863115 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/321c0696-5cc2-4038-8783-5d92da061c2d-home" pod "321c0696-5cc2-4038-8783-5d92da061c2d" (UID: "321c0696-5cc2-4038-8783-5d92da061c2d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:05:23.863299 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:05:23.863142 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/321c0696-5cc2-4038-8783-5d92da061c2d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/321c0696-5cc2-4038-8783-5d92da061c2d-dshm\") pod \"321c0696-5cc2-4038-8783-5d92da061c2d\" (UID: \"321c0696-5cc2-4038-8783-5d92da061c2d\") " Oct 30 17:05:23.864134 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:05:23.864102 5052 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/321c0696-5cc2-4038-8783-5d92da061c2d-tmp-dir\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:05:23.864265 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:05:23.864134 5052 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/321c0696-5cc2-4038-8783-5d92da061c2d-model-cache\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:05:23.864265 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:05:23.864151 5052 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/321c0696-5cc2-4038-8783-5d92da061c2d-home\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:05:23.865108 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:05:23.865069 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/321c0696-5cc2-4038-8783-5d92da061c2d-kube-api-access-njxtd" pod "321c0696-5cc2-4038-8783-5d92da061c2d" (UID: "321c0696-5cc2-4038-8783-5d92da061c2d"). InnerVolumeSpecName "kube-api-access-njxtd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:05:23.865216 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:05:23.865146 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/321c0696-5cc2-4038-8783-5d92da061c2d-dshm" pod "321c0696-5cc2-4038-8783-5d92da061c2d" (UID: "321c0696-5cc2-4038-8783-5d92da061c2d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:05:23.865350 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:05:23.865322 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/321c0696-5cc2-4038-8783-5d92da061c2d-tls-certs" pod "321c0696-5cc2-4038-8783-5d92da061c2d" (UID: "321c0696-5cc2-4038-8783-5d92da061c2d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:05:23.966001 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:05:23.965966 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njxtd\" (UniqueName: \"kubernetes.io/projected/321c0696-5cc2-4038-8783-5d92da061c2d-kube-api-access-njxtd\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:05:23.966001 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:05:23.965998 5052 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/321c0696-5cc2-4038-8783-5d92da061c2d-tls-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:05:23.966001 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:05:23.966009 5052 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/321c0696-5cc2-4038-8783-5d92da061c2d-dshm\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:05:24.621441 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:05:24.621401 5052 generic.go:356] "Generic (PLEG): container finished" podID="321c0696-5cc2-4038-8783-5d92da061c2d" containerID="ac7dc35476d346b2b331b560b82f81313e935c61cb9267acb1b287ccab94b489" exitCode=0 Oct 30 17:05:24.622021 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:05:24.621484 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-9e533f/auth-test-model-9e533f-kserve-6dbc95dc5c-rfqgk" Oct 30 17:05:24.627754 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:05:24.627714 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-9e533f/auth-test-model-9e533f-kserve-6dbc95dc5c-rfqgk" event={"ID":"321c0696-5cc2-4038-8783-5d92da061c2d","Type":"ContainerDied","Data":"ac7dc35476d346b2b331b560b82f81313e935c61cb9267acb1b287ccab94b489"} Oct 30 17:05:24.627754 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:05:24.627754 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-9e533f/auth-test-model-9e533f-kserve-6dbc95dc5c-rfqgk" event={"ID":"321c0696-5cc2-4038-8783-5d92da061c2d","Type":"ContainerDied","Data":"f42411f7f262c6380dd40371a012754f2f8b1495606967d5c93b60a1d9c88fe2"} Oct 30 17:05:24.627982 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:05:24.627771 5052 scope.go:122] "RemoveContainer" containerID="ac7dc35476d346b2b331b560b82f81313e935c61cb9267acb1b287ccab94b489" Oct 30 17:05:24.641180 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:05:24.641160 5052 scope.go:122] "RemoveContainer" containerID="ac7dc35476d346b2b331b560b82f81313e935c61cb9267acb1b287ccab94b489" Oct 30 17:05:24.641444 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:05:24.641424 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac7dc35476d346b2b331b560b82f81313e935c61cb9267acb1b287ccab94b489\": container with ID starting with ac7dc35476d346b2b331b560b82f81313e935c61cb9267acb1b287ccab94b489 not found: ID does not exist" containerID="ac7dc35476d346b2b331b560b82f81313e935c61cb9267acb1b287ccab94b489" Oct 30 17:05:24.641493 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:05:24.641465 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac7dc35476d346b2b331b560b82f81313e935c61cb9267acb1b287ccab94b489"} err="failed to get container status \"ac7dc35476d346b2b331b560b82f81313e935c61cb9267acb1b287ccab94b489\": rpc error: code = NotFound desc = could not find container \"ac7dc35476d346b2b331b560b82f81313e935c61cb9267acb1b287ccab94b489\": container with ID starting with ac7dc35476d346b2b331b560b82f81313e935c61cb9267acb1b287ccab94b489 not found: ID does not exist" Oct 30 17:05:24.681718 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:05:24.681693 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-9e533f/auth-test-model-9e533f-kserve-6dbc95dc5c-rfqgk"] Oct 30 17:05:24.694825 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:05:24.694803 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-9e533f/auth-test-model-9e533f-kserve-6dbc95dc5c-rfqgk"] Oct 30 17:05:26.601569 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:05:26.601537 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="321c0696-5cc2-4038-8783-5d92da061c2d" path="/var/lib/kubelet/pods/321c0696-5cc2-4038-8783-5d92da061c2d/volumes" Oct 30 17:05:39.178670 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:05:39.178633 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-bfecdd/sub-test-model-bfecdd-kserve-d8464b565-47v4j"] Oct 30 17:05:39.179562 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:05:39.179540 5052 cpu_manager.go:414] "containerMap: removing container" podUID="321c0696-5cc2-4038-8783-5d92da061c2d" containerName="main" Oct 30 17:05:39.179562 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:05:39.179562 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="321c0696-5cc2-4038-8783-5d92da061c2d" containerName="main" Oct 30 17:05:39.179730 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:05:39.179712 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="321c0696-5cc2-4038-8783-5d92da061c2d" containerName="main" Oct 30 17:05:39.183812 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:05:39.183789 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-bfecdd/sub-test-model-bfecdd-kserve-d8464b565-47v4j" Oct 30 17:05:39.216302 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:05:39.216272 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-bfecdd\"/\"openshift-service-ca.crt\"" Oct 30 17:05:39.216512 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:05:39.216328 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-bfecdd\"/\"default-dockercfg-d6v5d\"" Oct 30 17:05:39.216512 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:05:39.216391 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-bfecdd\"/\"kube-root-ca.crt\"" Oct 30 17:05:39.216512 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:05:39.216396 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-bfecdd\"/\"sub-test-model-bfecdd-kserve-self-signed-certs\"" Oct 30 17:05:39.220617 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:05:39.220591 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-bfecdd/sub-test-model-bfecdd-kserve-d8464b565-47v4j"] Oct 30 17:05:39.306154 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:05:39.306093 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55216e71-08aa-4b36-912c-696f933beb4b-model-cache\") pod \"sub-test-model-bfecdd-kserve-d8464b565-47v4j\" (UID: \"55216e71-08aa-4b36-912c-696f933beb4b\") " pod="ai-tenant-e2e-shared-a-w0-bfecdd/sub-test-model-bfecdd-kserve-d8464b565-47v4j" Oct 30 17:05:39.306332 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:05:39.306240 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55216e71-08aa-4b36-912c-696f933beb4b-dshm\") pod \"sub-test-model-bfecdd-kserve-d8464b565-47v4j\" (UID: \"55216e71-08aa-4b36-912c-696f933beb4b\") " pod="ai-tenant-e2e-shared-a-w0-bfecdd/sub-test-model-bfecdd-kserve-d8464b565-47v4j" Oct 30 17:05:39.306332 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:05:39.306302 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55216e71-08aa-4b36-912c-696f933beb4b-tls-certs\") pod \"sub-test-model-bfecdd-kserve-d8464b565-47v4j\" (UID: \"55216e71-08aa-4b36-912c-696f933beb4b\") " pod="ai-tenant-e2e-shared-a-w0-bfecdd/sub-test-model-bfecdd-kserve-d8464b565-47v4j" Oct 30 17:05:39.306453 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:05:39.306409 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9b6w\" (UniqueName: \"kubernetes.io/projected/55216e71-08aa-4b36-912c-696f933beb4b-kube-api-access-l9b6w\") pod \"sub-test-model-bfecdd-kserve-d8464b565-47v4j\" (UID: \"55216e71-08aa-4b36-912c-696f933beb4b\") " pod="ai-tenant-e2e-shared-a-w0-bfecdd/sub-test-model-bfecdd-kserve-d8464b565-47v4j" Oct 30 17:05:39.306453 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:05:39.306449 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55216e71-08aa-4b36-912c-696f933beb4b-home\") pod \"sub-test-model-bfecdd-kserve-d8464b565-47v4j\" (UID: \"55216e71-08aa-4b36-912c-696f933beb4b\") " pod="ai-tenant-e2e-shared-a-w0-bfecdd/sub-test-model-bfecdd-kserve-d8464b565-47v4j" Oct 30 17:05:39.306525 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:05:39.306511 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55216e71-08aa-4b36-912c-696f933beb4b-tmp-dir\") pod \"sub-test-model-bfecdd-kserve-d8464b565-47v4j\" (UID: \"55216e71-08aa-4b36-912c-696f933beb4b\") " pod="ai-tenant-e2e-shared-a-w0-bfecdd/sub-test-model-bfecdd-kserve-d8464b565-47v4j" Oct 30 17:05:39.408145 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:05:39.408107 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55216e71-08aa-4b36-912c-696f933beb4b-tls-certs\") pod \"sub-test-model-bfecdd-kserve-d8464b565-47v4j\" (UID: \"55216e71-08aa-4b36-912c-696f933beb4b\") " pod="ai-tenant-e2e-shared-a-w0-bfecdd/sub-test-model-bfecdd-kserve-d8464b565-47v4j" Oct 30 17:05:39.408328 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:05:39.408195 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9b6w\" (UniqueName: \"kubernetes.io/projected/55216e71-08aa-4b36-912c-696f933beb4b-kube-api-access-l9b6w\") pod \"sub-test-model-bfecdd-kserve-d8464b565-47v4j\" (UID: \"55216e71-08aa-4b36-912c-696f933beb4b\") " pod="ai-tenant-e2e-shared-a-w0-bfecdd/sub-test-model-bfecdd-kserve-d8464b565-47v4j" Oct 30 17:05:39.408328 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:05:39.408245 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55216e71-08aa-4b36-912c-696f933beb4b-home\") pod \"sub-test-model-bfecdd-kserve-d8464b565-47v4j\" (UID: \"55216e71-08aa-4b36-912c-696f933beb4b\") " pod="ai-tenant-e2e-shared-a-w0-bfecdd/sub-test-model-bfecdd-kserve-d8464b565-47v4j" Oct 30 17:05:39.408328 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:05:39.408293 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55216e71-08aa-4b36-912c-696f933beb4b-tmp-dir\") pod \"sub-test-model-bfecdd-kserve-d8464b565-47v4j\" (UID: \"55216e71-08aa-4b36-912c-696f933beb4b\") " pod="ai-tenant-e2e-shared-a-w0-bfecdd/sub-test-model-bfecdd-kserve-d8464b565-47v4j" Oct 30 17:05:39.408524 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:05:39.408400 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55216e71-08aa-4b36-912c-696f933beb4b-model-cache\") pod \"sub-test-model-bfecdd-kserve-d8464b565-47v4j\" (UID: \"55216e71-08aa-4b36-912c-696f933beb4b\") " pod="ai-tenant-e2e-shared-a-w0-bfecdd/sub-test-model-bfecdd-kserve-d8464b565-47v4j" Oct 30 17:05:39.408524 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:05:39.408506 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55216e71-08aa-4b36-912c-696f933beb4b-dshm\") pod \"sub-test-model-bfecdd-kserve-d8464b565-47v4j\" (UID: \"55216e71-08aa-4b36-912c-696f933beb4b\") " pod="ai-tenant-e2e-shared-a-w0-bfecdd/sub-test-model-bfecdd-kserve-d8464b565-47v4j" Oct 30 17:05:39.408749 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:05:39.408716 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55216e71-08aa-4b36-912c-696f933beb4b-home\") pod \"sub-test-model-bfecdd-kserve-d8464b565-47v4j\" (UID: \"55216e71-08aa-4b36-912c-696f933beb4b\") " pod="ai-tenant-e2e-shared-a-w0-bfecdd/sub-test-model-bfecdd-kserve-d8464b565-47v4j" Oct 30 17:05:39.408832 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:05:39.408779 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55216e71-08aa-4b36-912c-696f933beb4b-tmp-dir\") pod \"sub-test-model-bfecdd-kserve-d8464b565-47v4j\" (UID: \"55216e71-08aa-4b36-912c-696f933beb4b\") " pod="ai-tenant-e2e-shared-a-w0-bfecdd/sub-test-model-bfecdd-kserve-d8464b565-47v4j" Oct 30 17:05:39.408892 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:05:39.408843 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55216e71-08aa-4b36-912c-696f933beb4b-model-cache\") pod \"sub-test-model-bfecdd-kserve-d8464b565-47v4j\" (UID: \"55216e71-08aa-4b36-912c-696f933beb4b\") " pod="ai-tenant-e2e-shared-a-w0-bfecdd/sub-test-model-bfecdd-kserve-d8464b565-47v4j" Oct 30 17:05:39.410609 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:05:39.410575 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55216e71-08aa-4b36-912c-696f933beb4b-dshm\") pod \"sub-test-model-bfecdd-kserve-d8464b565-47v4j\" (UID: \"55216e71-08aa-4b36-912c-696f933beb4b\") " pod="ai-tenant-e2e-shared-a-w0-bfecdd/sub-test-model-bfecdd-kserve-d8464b565-47v4j" Oct 30 17:05:39.410785 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:05:39.410767 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55216e71-08aa-4b36-912c-696f933beb4b-tls-certs\") pod \"sub-test-model-bfecdd-kserve-d8464b565-47v4j\" (UID: \"55216e71-08aa-4b36-912c-696f933beb4b\") " pod="ai-tenant-e2e-shared-a-w0-bfecdd/sub-test-model-bfecdd-kserve-d8464b565-47v4j" Oct 30 17:05:39.467654 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:05:39.467298 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9b6w\" (UniqueName: \"kubernetes.io/projected/55216e71-08aa-4b36-912c-696f933beb4b-kube-api-access-l9b6w\") pod \"sub-test-model-bfecdd-kserve-d8464b565-47v4j\" (UID: \"55216e71-08aa-4b36-912c-696f933beb4b\") " pod="ai-tenant-e2e-shared-a-w0-bfecdd/sub-test-model-bfecdd-kserve-d8464b565-47v4j" Oct 30 17:05:39.512335 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:05:39.511993 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-bfecdd/sub-test-model-bfecdd-kserve-d8464b565-47v4j" Oct 30 17:05:39.858117 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:05:39.858086 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-bfecdd/sub-test-model-bfecdd-kserve-d8464b565-47v4j"] Oct 30 17:05:39.858762 ip-10-0-131-136 kubenswrapper[5052]: W1030 17:05:39.858726 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod55216e71_08aa_4b36_912c_696f933beb4b.slice/crio-da441f9cade97642529002db07de5b1e7e616e7265695b992da2725eda0e2d83 WatchSource:0}: Error finding container da441f9cade97642529002db07de5b1e7e616e7265695b992da2725eda0e2d83: Status 404 returned error can't find the container with id da441f9cade97642529002db07de5b1e7e616e7265695b992da2725eda0e2d83 Oct 30 17:05:40.133147 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:05:40.131155 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cdwcj"] Oct 30 17:05:40.745080 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:05:40.745045 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-bfecdd/sub-test-model-bfecdd-kserve-d8464b565-47v4j" event={"ID":"55216e71-08aa-4b36-912c-696f933beb4b","Type":"ContainerStarted","Data":"f109efa855489d46514a93cee7023f8cdd868fd25a3363f72635fea8b20de7ec"} Oct 30 17:05:40.745485 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:05:40.745086 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-bfecdd/sub-test-model-bfecdd-kserve-d8464b565-47v4j" event={"ID":"55216e71-08aa-4b36-912c-696f933beb4b","Type":"ContainerStarted","Data":"da441f9cade97642529002db07de5b1e7e616e7265695b992da2725eda0e2d83"} Oct 30 17:05:40.790293 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:05:40.790238 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-bfecdd/sub-test-model-bfecdd-kserve-d8464b565-47v4j" podStartSLOduration=1.7902212469999998 podStartE2EDuration="1.790221247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:05:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:05:40.789502782 +0000 UTC m=+2696.316838017" watchObservedRunningTime="2026-10-30 17:05:40.790221247 +0000 UTC m=+2696.317556470" Oct 30 17:05:41.521605 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:05:41.519681 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-bfecdd/sub-test-model-bfecdd-kserve-d8464b565-47v4j" Oct 30 17:05:42.766393 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:05:42.766345 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d693d0/distinct-2-w6-d693d0-kserve-6498fdc474-dpbjj" podUID="4b98b8ff-72cd-4ddb-a4b8-67078162e611" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:05:44.854296 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:05:44.854255 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-c8db42/sub-test-model-c8db42-kserve-6676499fbc-2wrm4"] Oct 30 17:05:45.585723 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:05:45.585683 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4wrf8_4f0f46f4-5cda-4d67-b6f3-47df06eb844c/console-operator/2.log" Oct 30 17:05:45.587959 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:05:45.587927 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4wrf8_4f0f46f4-5cda-4d67-b6f3-47df06eb844c/console-operator/2.log" Oct 30 17:05:48.841472 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:05:48.841434 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 17:05:49.515167 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:05:49.515131 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-bfecdd/sub-test-model-bfecdd-kserve-d8464b565-47v4j" Oct 30 17:05:49.529059 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:05:49.529026 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-bfecdd/sub-test-model-bfecdd-kserve-d8464b565-47v4j" Oct 30 17:05:49.958537 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:05:49.958503 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-bfecdd/sub-test-model-bfecdd-kserve-d8464b565-47v4j" Oct 30 17:05:51.565887 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:05:51.565851 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-55e83e-5d6c866678-kgrpj"] Oct 30 17:05:51.571962 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:05:51.571942 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-55e83e-5d6c866678-kgrpj" Oct 30 17:05:51.588558 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:05:51.588506 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-55e83e-5d6c866678-kgrpj"] Oct 30 17:05:51.596993 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:05:51.596967 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-55e83e\"" Oct 30 17:05:51.640377 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:05:51.640351 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-55e83e-577c688fdc-nxvxm"] Oct 30 17:05:51.645004 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:05:51.644986 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-55e83e-577c688fdc-nxvxm" Oct 30 17:05:51.663232 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:05:51.663203 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqb5f\" (UniqueName: \"kubernetes.io/projected/3b0ced1c-8a0f-453b-bd03-8f06a02f5f36-kube-api-access-wqb5f\") pod \"payload-processing-e2e-ipp-a-55e83e-5d6c866678-kgrpj\" (UID: \"3b0ced1c-8a0f-453b-bd03-8f06a02f5f36\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-55e83e-5d6c866678-kgrpj" Oct 30 17:05:51.663612 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:05:51.663553 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b0ced1c-8a0f-453b-bd03-8f06a02f5f36-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-55e83e-5d6c866678-kgrpj\" (UID: \"3b0ced1c-8a0f-453b-bd03-8f06a02f5f36\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-55e83e-5d6c866678-kgrpj" Oct 30 17:05:51.665643 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:05:51.665611 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-55e83e-577c688fdc-nxvxm"] Oct 30 17:05:51.665757 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:05:51.665651 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-55e83e-dockercfg-g5pz4\"" Oct 30 17:05:51.792611 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:05:51.786358 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bec23cc5-623a-4c22-84e7-afc66684df13-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-55e83e-577c688fdc-nxvxm\" (UID: \"bec23cc5-623a-4c22-84e7-afc66684df13\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-55e83e-577c688fdc-nxvxm" Oct 30 17:05:51.808925 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:05:51.808890 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqb5f\" (UniqueName: \"kubernetes.io/projected/3b0ced1c-8a0f-453b-bd03-8f06a02f5f36-kube-api-access-wqb5f\") pod \"payload-processing-e2e-ipp-a-55e83e-5d6c866678-kgrpj\" (UID: \"3b0ced1c-8a0f-453b-bd03-8f06a02f5f36\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-55e83e-5d6c866678-kgrpj" Oct 30 17:05:51.809270 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:05:51.809247 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5xks\" (UniqueName: \"kubernetes.io/projected/bec23cc5-623a-4c22-84e7-afc66684df13-kube-api-access-v5xks\") pod \"payload-pre-processing-e2e-ipp-a-55e83e-577c688fdc-nxvxm\" (UID: \"bec23cc5-623a-4c22-84e7-afc66684df13\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-55e83e-577c688fdc-nxvxm" Oct 30 17:05:51.809420 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:05:51.809404 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b0ced1c-8a0f-453b-bd03-8f06a02f5f36-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-55e83e-5d6c866678-kgrpj\" (UID: \"3b0ced1c-8a0f-453b-bd03-8f06a02f5f36\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-55e83e-5d6c866678-kgrpj" Oct 30 17:05:51.846654 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:05:51.840686 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b0ced1c-8a0f-453b-bd03-8f06a02f5f36-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-55e83e-5d6c866678-kgrpj\" (UID: \"3b0ced1c-8a0f-453b-bd03-8f06a02f5f36\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-55e83e-5d6c866678-kgrpj" Oct 30 17:05:51.878889 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:05:51.878754 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqb5f\" (UniqueName: \"kubernetes.io/projected/3b0ced1c-8a0f-453b-bd03-8f06a02f5f36-kube-api-access-wqb5f\") pod \"payload-processing-e2e-ipp-a-55e83e-5d6c866678-kgrpj\" (UID: \"3b0ced1c-8a0f-453b-bd03-8f06a02f5f36\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-55e83e-5d6c866678-kgrpj" Oct 30 17:05:51.905669 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:05:51.890846 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-55e83e-5d6c866678-kgrpj" Oct 30 17:05:51.933531 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:05:51.918732 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bec23cc5-623a-4c22-84e7-afc66684df13-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-55e83e-577c688fdc-nxvxm\" (UID: \"bec23cc5-623a-4c22-84e7-afc66684df13\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-55e83e-577c688fdc-nxvxm" Oct 30 17:05:51.934450 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:05:51.934415 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5xks\" (UniqueName: \"kubernetes.io/projected/bec23cc5-623a-4c22-84e7-afc66684df13-kube-api-access-v5xks\") pod \"payload-pre-processing-e2e-ipp-a-55e83e-577c688fdc-nxvxm\" (UID: \"bec23cc5-623a-4c22-84e7-afc66684df13\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-55e83e-577c688fdc-nxvxm" Oct 30 17:05:51.945820 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:05:51.945791 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bec23cc5-623a-4c22-84e7-afc66684df13-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-55e83e-577c688fdc-nxvxm\" (UID: \"bec23cc5-623a-4c22-84e7-afc66684df13\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-55e83e-577c688fdc-nxvxm" Oct 30 17:05:51.963566 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:05:51.963537 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5xks\" (UniqueName: \"kubernetes.io/projected/bec23cc5-623a-4c22-84e7-afc66684df13-kube-api-access-v5xks\") pod \"payload-pre-processing-e2e-ipp-a-55e83e-577c688fdc-nxvxm\" (UID: \"bec23cc5-623a-4c22-84e7-afc66684df13\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-55e83e-577c688fdc-nxvxm" Oct 30 17:05:52.249006 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:05:52.248974 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-55e83e-5d6c866678-kgrpj"] Oct 30 17:05:52.250200 ip-10-0-131-136 kubenswrapper[5052]: W1030 17:05:52.250164 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b0ced1c_8a0f_453b_bd03_8f06a02f5f36.slice/crio-e0bdc2009704248cd376d4a749e2a425e16c7d7667199a656ed1dd2ae477f267 WatchSource:0}: Error finding container e0bdc2009704248cd376d4a749e2a425e16c7d7667199a656ed1dd2ae477f267: Status 404 returned error can't find the container with id e0bdc2009704248cd376d4a749e2a425e16c7d7667199a656ed1dd2ae477f267 Oct 30 17:05:52.259373 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:05:52.259345 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-55e83e-577c688fdc-nxvxm" Oct 30 17:05:52.583880 ip-10-0-131-136 kubenswrapper[5052]: W1030 17:05:52.583848 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbec23cc5_623a_4c22_84e7_afc66684df13.slice/crio-5be4f64d0e020356a44d973c5e875e44c9fb039b082eb917bc1babd64222ae40 WatchSource:0}: Error finding container 5be4f64d0e020356a44d973c5e875e44c9fb039b082eb917bc1babd64222ae40: Status 404 returned error can't find the container with id 5be4f64d0e020356a44d973c5e875e44c9fb039b082eb917bc1babd64222ae40 Oct 30 17:05:52.584197 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:05:52.584003 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-55e83e-577c688fdc-nxvxm"] Oct 30 17:05:52.973508 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:05:52.973466 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-55e83e-5d6c866678-kgrpj" event={"ID":"3b0ced1c-8a0f-453b-bd03-8f06a02f5f36","Type":"ContainerStarted","Data":"a77f5527cf677b15012dba899fb9689313601cc4143a5f31b49814e4775756fa"} Oct 30 17:05:52.973748 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:05:52.973534 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-55e83e-5d6c866678-kgrpj" event={"ID":"3b0ced1c-8a0f-453b-bd03-8f06a02f5f36","Type":"ContainerStarted","Data":"e0bdc2009704248cd376d4a749e2a425e16c7d7667199a656ed1dd2ae477f267"} Oct 30 17:05:52.973748 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:05:52.973562 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-55e83e-5d6c866678-kgrpj" Oct 30 17:05:52.975187 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:05:52.975152 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-55e83e-577c688fdc-nxvxm" event={"ID":"bec23cc5-623a-4c22-84e7-afc66684df13","Type":"ContainerStarted","Data":"02a263aea2b0e241213df4da6fe08207120a1aeb61dbdcb16d0b9c6d6806175d"} Oct 30 17:05:52.975308 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:05:52.975208 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-55e83e-577c688fdc-nxvxm" event={"ID":"bec23cc5-623a-4c22-84e7-afc66684df13","Type":"ContainerStarted","Data":"5be4f64d0e020356a44d973c5e875e44c9fb039b082eb917bc1babd64222ae40"} Oct 30 17:05:52.975308 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:05:52.975260 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-55e83e-577c688fdc-nxvxm" Oct 30 17:05:53.022436 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:05:53.022393 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-55e83e-5d6c866678-kgrpj" podStartSLOduration=2.022380235 podStartE2EDuration="2.022380235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:05:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:05:53.018175715 +0000 UTC m=+2708.545510945" watchObservedRunningTime="2026-10-30 17:05:53.022380235 +0000 UTC m=+2708.549715494" Oct 30 17:05:53.061182 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:05:53.061137 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-55e83e-577c688fdc-nxvxm" podStartSLOduration=2.061128116 podStartE2EDuration="2.061128116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:05:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:05:53.060500258 +0000 UTC m=+2708.587835478" watchObservedRunningTime="2026-10-30 17:05:53.061128116 +0000 UTC m=+2708.588463335" Oct 30 17:05:59.634269 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:05:59.634227 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b16279-d87cd4586-gmr6p"] Oct 30 17:05:59.894238 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:05:59.894101 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-c8db42/sub-test-model-c8db42-kserve-6676499fbc-2wrm4" podUID="9c1e0fa7-8b1e-430a-b3ff-cef2a4a7bbd1" containerName="main" containerID="cri-o://f478601357926d58a3acffcabf0afdd3760d8f9676fab7bab4bf64f0f27d401b" gracePeriod=45 Oct 30 17:06:00.002813 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:00.002765 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w2-c8db42/sub-test-model-c8db42-kserve-6676499fbc-2wrm4" podUID="9c1e0fa7-8b1e-430a-b3ff-cef2a4a7bbd1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.186:8000/ready\": dial tcp 10.132.0.186:8000: connect: connection refused" Oct 30 17:06:00.096603 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:00.096556 5052 generic.go:356] "Generic (PLEG): container finished" podID="9c1e0fa7-8b1e-430a-b3ff-cef2a4a7bbd1" containerID="f478601357926d58a3acffcabf0afdd3760d8f9676fab7bab4bf64f0f27d401b" exitCode=0 Oct 30 17:06:00.096770 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:00.096620 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-c8db42/sub-test-model-c8db42-kserve-6676499fbc-2wrm4" event={"ID":"9c1e0fa7-8b1e-430a-b3ff-cef2a4a7bbd1","Type":"ContainerDied","Data":"f478601357926d58a3acffcabf0afdd3760d8f9676fab7bab4bf64f0f27d401b"} Oct 30 17:06:00.244354 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:00.244325 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-c8db42/sub-test-model-c8db42-kserve-6676499fbc-2wrm4" Oct 30 17:06:00.336606 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:00.336194 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c1e0fa7-8b1e-430a-b3ff-cef2a4a7bbd1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c1e0fa7-8b1e-430a-b3ff-cef2a4a7bbd1-dshm\") pod \"9c1e0fa7-8b1e-430a-b3ff-cef2a4a7bbd1\" (UID: \"9c1e0fa7-8b1e-430a-b3ff-cef2a4a7bbd1\") " Oct 30 17:06:00.336606 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:00.336261 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c1e0fa7-8b1e-430a-b3ff-cef2a4a7bbd1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c1e0fa7-8b1e-430a-b3ff-cef2a4a7bbd1-tmp-dir\") pod \"9c1e0fa7-8b1e-430a-b3ff-cef2a4a7bbd1\" (UID: \"9c1e0fa7-8b1e-430a-b3ff-cef2a4a7bbd1\") " Oct 30 17:06:00.336606 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:00.336517 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c1e0fa7-8b1e-430a-b3ff-cef2a4a7bbd1-tls-certs\" (UniqueName: \"kubernetes.io/secret/9c1e0fa7-8b1e-430a-b3ff-cef2a4a7bbd1-tls-certs\") pod \"9c1e0fa7-8b1e-430a-b3ff-cef2a4a7bbd1\" (UID: \"9c1e0fa7-8b1e-430a-b3ff-cef2a4a7bbd1\") " Oct 30 17:06:00.336606 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:00.336571 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c1e0fa7-8b1e-430a-b3ff-cef2a4a7bbd1-home\" (UniqueName: \"kubernetes.io/empty-dir/9c1e0fa7-8b1e-430a-b3ff-cef2a4a7bbd1-home\") pod \"9c1e0fa7-8b1e-430a-b3ff-cef2a4a7bbd1\" (UID: \"9c1e0fa7-8b1e-430a-b3ff-cef2a4a7bbd1\") " Oct 30 17:06:00.336606 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:00.336612 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c1e0fa7-8b1e-430a-b3ff-cef2a4a7bbd1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c1e0fa7-8b1e-430a-b3ff-cef2a4a7bbd1-model-cache\") pod \"9c1e0fa7-8b1e-430a-b3ff-cef2a4a7bbd1\" (UID: \"9c1e0fa7-8b1e-430a-b3ff-cef2a4a7bbd1\") " Oct 30 17:06:00.337025 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:00.336671 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c1e0fa7-8b1e-430a-b3ff-cef2a4a7bbd1-kube-api-access-cbhdw\" (UniqueName: \"kubernetes.io/projected/9c1e0fa7-8b1e-430a-b3ff-cef2a4a7bbd1-kube-api-access-cbhdw\") pod \"9c1e0fa7-8b1e-430a-b3ff-cef2a4a7bbd1\" (UID: \"9c1e0fa7-8b1e-430a-b3ff-cef2a4a7bbd1\") " Oct 30 17:06:00.351601 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:00.338674 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c1e0fa7-8b1e-430a-b3ff-cef2a4a7bbd1-model-cache" pod "9c1e0fa7-8b1e-430a-b3ff-cef2a4a7bbd1" (UID: "9c1e0fa7-8b1e-430a-b3ff-cef2a4a7bbd1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:06:00.363599 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:00.354417 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c1e0fa7-8b1e-430a-b3ff-cef2a4a7bbd1-tmp-dir" pod "9c1e0fa7-8b1e-430a-b3ff-cef2a4a7bbd1" (UID: "9c1e0fa7-8b1e-430a-b3ff-cef2a4a7bbd1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:06:00.381097 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:00.370146 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c1e0fa7-8b1e-430a-b3ff-cef2a4a7bbd1-home" pod "9c1e0fa7-8b1e-430a-b3ff-cef2a4a7bbd1" (UID: "9c1e0fa7-8b1e-430a-b3ff-cef2a4a7bbd1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:06:00.398603 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:00.389803 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c1e0fa7-8b1e-430a-b3ff-cef2a4a7bbd1-kube-api-access-cbhdw" pod "9c1e0fa7-8b1e-430a-b3ff-cef2a4a7bbd1" (UID: "9c1e0fa7-8b1e-430a-b3ff-cef2a4a7bbd1"). InnerVolumeSpecName "kube-api-access-cbhdw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:06:00.398603 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:00.396704 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c1e0fa7-8b1e-430a-b3ff-cef2a4a7bbd1-dshm" pod "9c1e0fa7-8b1e-430a-b3ff-cef2a4a7bbd1" (UID: "9c1e0fa7-8b1e-430a-b3ff-cef2a4a7bbd1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:06:00.398959 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:00.398673 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c1e0fa7-8b1e-430a-b3ff-cef2a4a7bbd1-tls-certs" pod "9c1e0fa7-8b1e-430a-b3ff-cef2a4a7bbd1" (UID: "9c1e0fa7-8b1e-430a-b3ff-cef2a4a7bbd1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:06:00.441946 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:00.441854 5052 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c1e0fa7-8b1e-430a-b3ff-cef2a4a7bbd1-home\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:06:00.452294 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:00.452156 5052 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c1e0fa7-8b1e-430a-b3ff-cef2a4a7bbd1-model-cache\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:06:00.452294 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:00.452207 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbhdw\" (UniqueName: \"kubernetes.io/projected/9c1e0fa7-8b1e-430a-b3ff-cef2a4a7bbd1-kube-api-access-cbhdw\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:06:00.452294 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:00.452223 5052 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c1e0fa7-8b1e-430a-b3ff-cef2a4a7bbd1-dshm\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:06:00.452294 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:00.452265 5052 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c1e0fa7-8b1e-430a-b3ff-cef2a4a7bbd1-tmp-dir\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:06:00.452294 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:00.452281 5052 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c1e0fa7-8b1e-430a-b3ff-cef2a4a7bbd1-tls-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:06:01.105188 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:01.105154 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-c8db42/sub-test-model-c8db42-kserve-6676499fbc-2wrm4" event={"ID":"9c1e0fa7-8b1e-430a-b3ff-cef2a4a7bbd1","Type":"ContainerDied","Data":"4772117ce90fbd88cb7137806f79094e2f03800050331ddbc92c2b996f1a55a6"} Oct 30 17:06:01.105188 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:01.105175 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-c8db42/sub-test-model-c8db42-kserve-6676499fbc-2wrm4" Oct 30 17:06:01.105711 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:01.105208 5052 scope.go:122] "RemoveContainer" containerID="f478601357926d58a3acffcabf0afdd3760d8f9676fab7bab4bf64f0f27d401b" Oct 30 17:06:01.164035 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:01.163998 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-c8db42/sub-test-model-c8db42-kserve-6676499fbc-2wrm4"] Oct 30 17:06:01.167643 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:01.167619 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-c8db42/sub-test-model-c8db42-kserve-6676499fbc-2wrm4"] Oct 30 17:06:02.599185 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:02.599147 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c1e0fa7-8b1e-430a-b3ff-cef2a4a7bbd1" path="/var/lib/kubelet/pods/9c1e0fa7-8b1e-430a-b3ff-cef2a4a7bbd1/volumes" Oct 30 17:06:03.992870 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:03.992838 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-55e83e-577c688fdc-nxvxm" Oct 30 17:06:03.993287 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:03.993037 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-55e83e-5d6c866678-kgrpj" Oct 30 17:06:04.635257 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:04.635196 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-b16279-d87cd4586-gmr6p" podUID="49192ccf-145d-47db-b495-46cd4daeb9cf" containerName="payload-processing" containerID="cri-o://33d1f191e869e2df72f712d02e97e83148efa2edd2d4963015d85799dfe08c97" gracePeriod=25 Oct 30 17:06:04.940312 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:04.940287 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b16279-d87cd4586-gmr6p" Oct 30 17:06:05.063609 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:05.060939 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49192ccf-145d-47db-b495-46cd4daeb9cf-kube-api-access-nldgp\" (UniqueName: \"kubernetes.io/projected/49192ccf-145d-47db-b495-46cd4daeb9cf-kube-api-access-nldgp\") pod \"49192ccf-145d-47db-b495-46cd4daeb9cf\" (UID: \"49192ccf-145d-47db-b495-46cd4daeb9cf\") " Oct 30 17:06:05.063609 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:05.061018 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49192ccf-145d-47db-b495-46cd4daeb9cf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49192ccf-145d-47db-b495-46cd4daeb9cf-plugins-config-volume\") pod \"49192ccf-145d-47db-b495-46cd4daeb9cf\" (UID: \"49192ccf-145d-47db-b495-46cd4daeb9cf\") " Oct 30 17:06:05.071130 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:05.070265 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49192ccf-145d-47db-b495-46cd4daeb9cf-plugins-config-volume" pod "49192ccf-145d-47db-b495-46cd4daeb9cf" (UID: "49192ccf-145d-47db-b495-46cd4daeb9cf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:06:05.074768 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:05.074733 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49192ccf-145d-47db-b495-46cd4daeb9cf-kube-api-access-nldgp" pod "49192ccf-145d-47db-b495-46cd4daeb9cf" (UID: "49192ccf-145d-47db-b495-46cd4daeb9cf"). InnerVolumeSpecName "kube-api-access-nldgp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:06:05.147695 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:05.147661 5052 generic.go:356] "Generic (PLEG): container finished" podID="49192ccf-145d-47db-b495-46cd4daeb9cf" containerID="33d1f191e869e2df72f712d02e97e83148efa2edd2d4963015d85799dfe08c97" exitCode=0 Oct 30 17:06:05.147910 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:05.147716 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b16279-d87cd4586-gmr6p" Oct 30 17:06:05.147910 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:05.147751 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b16279-d87cd4586-gmr6p" event={"ID":"49192ccf-145d-47db-b495-46cd4daeb9cf","Type":"ContainerDied","Data":"33d1f191e869e2df72f712d02e97e83148efa2edd2d4963015d85799dfe08c97"} Oct 30 17:06:05.147910 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:05.147806 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b16279-d87cd4586-gmr6p" event={"ID":"49192ccf-145d-47db-b495-46cd4daeb9cf","Type":"ContainerDied","Data":"bb9f34e846e0a0f1894fc27378ee70757bed885acd15b5c4cbf5a0c206490ddb"} Oct 30 17:06:05.147910 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:05.147839 5052 scope.go:122] "RemoveContainer" containerID="33d1f191e869e2df72f712d02e97e83148efa2edd2d4963015d85799dfe08c97" Oct 30 17:06:05.159864 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:05.159844 5052 scope.go:122] "RemoveContainer" containerID="33d1f191e869e2df72f712d02e97e83148efa2edd2d4963015d85799dfe08c97" Oct 30 17:06:05.160132 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:06:05.160110 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33d1f191e869e2df72f712d02e97e83148efa2edd2d4963015d85799dfe08c97\": container with ID starting with 33d1f191e869e2df72f712d02e97e83148efa2edd2d4963015d85799dfe08c97 not found: ID does not exist" containerID="33d1f191e869e2df72f712d02e97e83148efa2edd2d4963015d85799dfe08c97" Oct 30 17:06:05.160180 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:05.160139 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33d1f191e869e2df72f712d02e97e83148efa2edd2d4963015d85799dfe08c97"} err="failed to get container status \"33d1f191e869e2df72f712d02e97e83148efa2edd2d4963015d85799dfe08c97\": rpc error: code = NotFound desc = could not find container \"33d1f191e869e2df72f712d02e97e83148efa2edd2d4963015d85799dfe08c97\": container with ID starting with 33d1f191e869e2df72f712d02e97e83148efa2edd2d4963015d85799dfe08c97 not found: ID does not exist" Oct 30 17:06:05.163076 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:05.163055 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nldgp\" (UniqueName: \"kubernetes.io/projected/49192ccf-145d-47db-b495-46cd4daeb9cf-kube-api-access-nldgp\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:06:05.163148 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:05.163078 5052 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49192ccf-145d-47db-b495-46cd4daeb9cf-plugins-config-volume\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:06:05.205460 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:05.205438 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b16279-d87cd4586-gmr6p"] Oct 30 17:06:05.209703 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:05.209682 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b16279-d87cd4586-gmr6p"] Oct 30 17:06:06.600426 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:06.600391 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49192ccf-145d-47db-b495-46cd4daeb9cf" path="/var/lib/kubelet/pods/49192ccf-145d-47db-b495-46cd4daeb9cf/volumes" Oct 30 17:06:07.114075 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:07.114038 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5cz66"] Oct 30 17:06:07.114301 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:07.114276 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5cz66" podUID="f59239c4-202e-45d5-9942-26fb0410d813" containerName="istio-proxy" containerID="cri-o://5153a71e73a3e6ac24744eac1a5a2deee8a609fcd75de487da50e3064a05cece" gracePeriod=30 Oct 30 17:06:08.385208 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:08.385175 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-hwkgn"] Oct 30 17:06:08.386041 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:08.386022 5052 cpu_manager.go:414] "containerMap: removing container" podUID="49192ccf-145d-47db-b495-46cd4daeb9cf" containerName="payload-processing" Oct 30 17:06:08.386041 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:08.386041 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49192ccf-145d-47db-b495-46cd4daeb9cf" containerName="payload-processing" Oct 30 17:06:08.386159 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:08.386068 5052 cpu_manager.go:414] "containerMap: removing container" podUID="9c1e0fa7-8b1e-430a-b3ff-cef2a4a7bbd1" containerName="main" Oct 30 17:06:08.386159 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:08.386077 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c1e0fa7-8b1e-430a-b3ff-cef2a4a7bbd1" containerName="main" Oct 30 17:06:08.386251 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:08.386239 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="49192ccf-145d-47db-b495-46cd4daeb9cf" containerName="payload-processing" Oct 30 17:06:08.386293 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:08.386255 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c1e0fa7-8b1e-430a-b3ff-cef2a4a7bbd1" containerName="main" Oct 30 17:06:08.390850 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:08.390832 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-hwkgn" Oct 30 17:06:08.412978 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:08.412946 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-hwkgn"] Oct 30 17:06:08.418696 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:08.418662 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96134e17-039f-4ea3-8522-ff54e3ab6797-istio-podinfo\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-hwkgn\" (UID: \"96134e17-039f-4ea3-8522-ff54e3ab6797\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-hwkgn" Oct 30 17:06:08.419172 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:08.418855 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96134e17-039f-4ea3-8522-ff54e3ab6797-istio-token\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-hwkgn\" (UID: \"96134e17-039f-4ea3-8522-ff54e3ab6797\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-hwkgn" Oct 30 17:06:08.419172 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:08.418932 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96134e17-039f-4ea3-8522-ff54e3ab6797-workload-socket\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-hwkgn\" (UID: \"96134e17-039f-4ea3-8522-ff54e3ab6797\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-hwkgn" Oct 30 17:06:08.419172 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:08.419019 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gg4j8\" (UniqueName: \"kubernetes.io/projected/96134e17-039f-4ea3-8522-ff54e3ab6797-kube-api-access-gg4j8\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-hwkgn\" (UID: \"96134e17-039f-4ea3-8522-ff54e3ab6797\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-hwkgn" Oct 30 17:06:08.419172 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:08.419064 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96134e17-039f-4ea3-8522-ff54e3ab6797-workload-certs\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-hwkgn\" (UID: \"96134e17-039f-4ea3-8522-ff54e3ab6797\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-hwkgn" Oct 30 17:06:08.419172 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:08.419130 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96134e17-039f-4ea3-8522-ff54e3ab6797-credential-socket\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-hwkgn\" (UID: \"96134e17-039f-4ea3-8522-ff54e3ab6797\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-hwkgn" Oct 30 17:06:08.419536 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:08.419201 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96134e17-039f-4ea3-8522-ff54e3ab6797-istio-data\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-hwkgn\" (UID: \"96134e17-039f-4ea3-8522-ff54e3ab6797\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-hwkgn" Oct 30 17:06:08.419536 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:08.419261 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96134e17-039f-4ea3-8522-ff54e3ab6797-istio-envoy\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-hwkgn\" (UID: \"96134e17-039f-4ea3-8522-ff54e3ab6797\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-hwkgn" Oct 30 17:06:08.419536 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:08.419286 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96134e17-039f-4ea3-8522-ff54e3ab6797-istiod-ca-cert\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-hwkgn\" (UID: \"96134e17-039f-4ea3-8522-ff54e3ab6797\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-hwkgn" Oct 30 17:06:08.520552 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:08.520517 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96134e17-039f-4ea3-8522-ff54e3ab6797-istio-data\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-hwkgn\" (UID: \"96134e17-039f-4ea3-8522-ff54e3ab6797\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-hwkgn" Oct 30 17:06:08.520798 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:08.520625 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96134e17-039f-4ea3-8522-ff54e3ab6797-istio-envoy\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-hwkgn\" (UID: \"96134e17-039f-4ea3-8522-ff54e3ab6797\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-hwkgn" Oct 30 17:06:08.520798 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:08.520650 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96134e17-039f-4ea3-8522-ff54e3ab6797-istiod-ca-cert\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-hwkgn\" (UID: \"96134e17-039f-4ea3-8522-ff54e3ab6797\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-hwkgn" Oct 30 17:06:08.520798 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:08.520680 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96134e17-039f-4ea3-8522-ff54e3ab6797-istio-podinfo\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-hwkgn\" (UID: \"96134e17-039f-4ea3-8522-ff54e3ab6797\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-hwkgn" Oct 30 17:06:08.520974 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:08.520840 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96134e17-039f-4ea3-8522-ff54e3ab6797-istio-token\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-hwkgn\" (UID: \"96134e17-039f-4ea3-8522-ff54e3ab6797\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-hwkgn" Oct 30 17:06:08.520974 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:08.520910 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96134e17-039f-4ea3-8522-ff54e3ab6797-workload-socket\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-hwkgn\" (UID: \"96134e17-039f-4ea3-8522-ff54e3ab6797\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-hwkgn" Oct 30 17:06:08.520974 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:08.520935 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96134e17-039f-4ea3-8522-ff54e3ab6797-istio-data\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-hwkgn\" (UID: \"96134e17-039f-4ea3-8522-ff54e3ab6797\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-hwkgn" Oct 30 17:06:08.521141 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:08.521011 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gg4j8\" (UniqueName: \"kubernetes.io/projected/96134e17-039f-4ea3-8522-ff54e3ab6797-kube-api-access-gg4j8\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-hwkgn\" (UID: \"96134e17-039f-4ea3-8522-ff54e3ab6797\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-hwkgn" Oct 30 17:06:08.521141 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:08.521054 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96134e17-039f-4ea3-8522-ff54e3ab6797-workload-certs\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-hwkgn\" (UID: \"96134e17-039f-4ea3-8522-ff54e3ab6797\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-hwkgn" Oct 30 17:06:08.521141 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:08.521120 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96134e17-039f-4ea3-8522-ff54e3ab6797-credential-socket\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-hwkgn\" (UID: \"96134e17-039f-4ea3-8522-ff54e3ab6797\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-hwkgn" Oct 30 17:06:08.521309 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:08.521194 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96134e17-039f-4ea3-8522-ff54e3ab6797-workload-socket\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-hwkgn\" (UID: \"96134e17-039f-4ea3-8522-ff54e3ab6797\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-hwkgn" Oct 30 17:06:08.521408 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:08.521382 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96134e17-039f-4ea3-8522-ff54e3ab6797-istiod-ca-cert\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-hwkgn\" (UID: \"96134e17-039f-4ea3-8522-ff54e3ab6797\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-hwkgn" Oct 30 17:06:08.521480 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:08.521408 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96134e17-039f-4ea3-8522-ff54e3ab6797-workload-certs\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-hwkgn\" (UID: \"96134e17-039f-4ea3-8522-ff54e3ab6797\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-hwkgn" Oct 30 17:06:08.521543 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:08.521488 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96134e17-039f-4ea3-8522-ff54e3ab6797-credential-socket\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-hwkgn\" (UID: \"96134e17-039f-4ea3-8522-ff54e3ab6797\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-hwkgn" Oct 30 17:06:08.523539 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:08.523510 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96134e17-039f-4ea3-8522-ff54e3ab6797-istio-envoy\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-hwkgn\" (UID: \"96134e17-039f-4ea3-8522-ff54e3ab6797\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-hwkgn" Oct 30 17:06:08.523672 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:08.523656 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96134e17-039f-4ea3-8522-ff54e3ab6797-istio-podinfo\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-hwkgn\" (UID: \"96134e17-039f-4ea3-8522-ff54e3ab6797\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-hwkgn" Oct 30 17:06:08.545648 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:08.545624 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gg4j8\" (UniqueName: \"kubernetes.io/projected/96134e17-039f-4ea3-8522-ff54e3ab6797-kube-api-access-gg4j8\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-hwkgn\" (UID: \"96134e17-039f-4ea3-8522-ff54e3ab6797\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-hwkgn" Oct 30 17:06:08.545648 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:08.545635 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96134e17-039f-4ea3-8522-ff54e3ab6797-istio-token\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-hwkgn\" (UID: \"96134e17-039f-4ea3-8522-ff54e3ab6797\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-hwkgn" Oct 30 17:06:08.706253 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:08.706216 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-hwkgn" Oct 30 17:06:09.033265 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:09.033233 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-hwkgn"] Oct 30 17:06:09.033850 ip-10-0-131-136 kubenswrapper[5052]: W1030 17:06:09.033826 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96134e17_039f_4ea3_8522_ff54e3ab6797.slice/crio-7076977a1cf44e6e3e985e186e9f234d6260570f5a0271746503b3dce62668d7 WatchSource:0}: Error finding container 7076977a1cf44e6e3e985e186e9f234d6260570f5a0271746503b3dce62668d7: Status 404 returned error can't find the container with id 7076977a1cf44e6e3e985e186e9f234d6260570f5a0271746503b3dce62668d7 Oct 30 17:06:09.036504 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:09.036465 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:06:09.036631 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:09.036548 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:06:09.036631 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:09.036613 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:06:09.183420 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:09.183386 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-hwkgn" event={"ID":"96134e17-039f-4ea3-8522-ff54e3ab6797","Type":"ContainerStarted","Data":"554308610ee1f23add189bb1173774101b5b78e04fe078e83b3df4c8e85488dc"} Oct 30 17:06:09.183678 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:09.183445 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-hwkgn" event={"ID":"96134e17-039f-4ea3-8522-ff54e3ab6797","Type":"ContainerStarted","Data":"7076977a1cf44e6e3e985e186e9f234d6260570f5a0271746503b3dce62668d7"} Oct 30 17:06:09.233512 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:09.233390 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-hwkgn" podStartSLOduration=1.233371779 podStartE2EDuration="1.233371779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:06:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:06:09.230207104 +0000 UTC m=+2724.757542345" watchObservedRunningTime="2026-10-30 17:06:09.233371779 +0000 UTC m=+2724.760706999" Oct 30 17:06:09.706523 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:09.706491 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-hwkgn" Oct 30 17:06:10.708921 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:10.708882 5052 patch_prober.go:29] interesting pod/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-hwkgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body= Oct 30 17:06:10.709338 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:10.708937 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-hwkgn" podUID="96134e17-039f-4ea3-8522-ff54e3ab6797" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" Oct 30 17:06:11.707118 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:11.707083 5052 patch_prober.go:29] interesting pod/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-hwkgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body= Oct 30 17:06:11.707302 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:11.707134 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-hwkgn" podUID="96134e17-039f-4ea3-8522-ff54e3ab6797" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" Oct 30 17:06:12.464314 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:12.464284 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5cz66" Oct 30 17:06:12.500719 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:12.500691 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f59239c4-202e-45d5-9942-26fb0410d813-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f59239c4-202e-45d5-9942-26fb0410d813-workload-socket\") pod \"f59239c4-202e-45d5-9942-26fb0410d813\" (UID: \"f59239c4-202e-45d5-9942-26fb0410d813\") " Oct 30 17:06:12.500842 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:12.500761 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f59239c4-202e-45d5-9942-26fb0410d813-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f59239c4-202e-45d5-9942-26fb0410d813-istiod-ca-cert\") pod \"f59239c4-202e-45d5-9942-26fb0410d813\" (UID: \"f59239c4-202e-45d5-9942-26fb0410d813\") " Oct 30 17:06:12.500842 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:12.500818 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f59239c4-202e-45d5-9942-26fb0410d813-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f59239c4-202e-45d5-9942-26fb0410d813-istio-data\") pod \"f59239c4-202e-45d5-9942-26fb0410d813\" (UID: \"f59239c4-202e-45d5-9942-26fb0410d813\") " Oct 30 17:06:12.500919 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:12.500845 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f59239c4-202e-45d5-9942-26fb0410d813-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f59239c4-202e-45d5-9942-26fb0410d813-istio-envoy\") pod \"f59239c4-202e-45d5-9942-26fb0410d813\" (UID: \"f59239c4-202e-45d5-9942-26fb0410d813\") " Oct 30 17:06:12.500982 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:12.500956 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f59239c4-202e-45d5-9942-26fb0410d813-workload-socket" pod "f59239c4-202e-45d5-9942-26fb0410d813" (UID: "f59239c4-202e-45d5-9942-26fb0410d813"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:06:12.501032 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:12.500971 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f59239c4-202e-45d5-9942-26fb0410d813-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f59239c4-202e-45d5-9942-26fb0410d813-credential-socket\") pod \"f59239c4-202e-45d5-9942-26fb0410d813\" (UID: \"f59239c4-202e-45d5-9942-26fb0410d813\") " Oct 30 17:06:12.501095 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:12.501076 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f59239c4-202e-45d5-9942-26fb0410d813-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f59239c4-202e-45d5-9942-26fb0410d813-istio-podinfo\") pod \"f59239c4-202e-45d5-9942-26fb0410d813\" (UID: \"f59239c4-202e-45d5-9942-26fb0410d813\") " Oct 30 17:06:12.501174 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:12.501159 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f59239c4-202e-45d5-9942-26fb0410d813-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f59239c4-202e-45d5-9942-26fb0410d813-workload-certs\") pod \"f59239c4-202e-45d5-9942-26fb0410d813\" (UID: \"f59239c4-202e-45d5-9942-26fb0410d813\") " Oct 30 17:06:12.501232 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:12.501078 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f59239c4-202e-45d5-9942-26fb0410d813-istio-data" pod "f59239c4-202e-45d5-9942-26fb0410d813" (UID: "f59239c4-202e-45d5-9942-26fb0410d813"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:06:12.501232 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:12.501213 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f59239c4-202e-45d5-9942-26fb0410d813-istiod-ca-cert" pod "f59239c4-202e-45d5-9942-26fb0410d813" (UID: "f59239c4-202e-45d5-9942-26fb0410d813"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:06:12.501350 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:12.501229 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f59239c4-202e-45d5-9942-26fb0410d813-kube-api-access-57xfz\" (UniqueName: \"kubernetes.io/projected/f59239c4-202e-45d5-9942-26fb0410d813-kube-api-access-57xfz\") pod \"f59239c4-202e-45d5-9942-26fb0410d813\" (UID: \"f59239c4-202e-45d5-9942-26fb0410d813\") " Oct 30 17:06:12.501350 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:12.501288 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f59239c4-202e-45d5-9942-26fb0410d813-istio-token\" (UniqueName: \"kubernetes.io/projected/f59239c4-202e-45d5-9942-26fb0410d813-istio-token\") pod \"f59239c4-202e-45d5-9942-26fb0410d813\" (UID: \"f59239c4-202e-45d5-9942-26fb0410d813\") " Oct 30 17:06:12.501350 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:12.501186 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f59239c4-202e-45d5-9942-26fb0410d813-credential-socket" pod "f59239c4-202e-45d5-9942-26fb0410d813" (UID: "f59239c4-202e-45d5-9942-26fb0410d813"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:06:12.501507 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:12.501473 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f59239c4-202e-45d5-9942-26fb0410d813-workload-certs" pod "f59239c4-202e-45d5-9942-26fb0410d813" (UID: "f59239c4-202e-45d5-9942-26fb0410d813"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:06:12.502125 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:12.502102 5052 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f59239c4-202e-45d5-9942-26fb0410d813-credential-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:06:12.502213 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:12.502131 5052 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f59239c4-202e-45d5-9942-26fb0410d813-workload-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:06:12.502213 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:12.502146 5052 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f59239c4-202e-45d5-9942-26fb0410d813-workload-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:06:12.502213 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:12.502158 5052 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f59239c4-202e-45d5-9942-26fb0410d813-istiod-ca-cert\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:06:12.502213 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:12.502171 5052 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f59239c4-202e-45d5-9942-26fb0410d813-istio-data\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:06:12.503184 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:12.503163 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f59239c4-202e-45d5-9942-26fb0410d813-istio-envoy" pod "f59239c4-202e-45d5-9942-26fb0410d813" (UID: "f59239c4-202e-45d5-9942-26fb0410d813"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:06:12.503323 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:12.503304 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f59239c4-202e-45d5-9942-26fb0410d813-istio-podinfo" pod "f59239c4-202e-45d5-9942-26fb0410d813" (UID: "f59239c4-202e-45d5-9942-26fb0410d813"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:06:12.503801 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:12.503777 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f59239c4-202e-45d5-9942-26fb0410d813-kube-api-access-57xfz" pod "f59239c4-202e-45d5-9942-26fb0410d813" (UID: "f59239c4-202e-45d5-9942-26fb0410d813"). InnerVolumeSpecName "kube-api-access-57xfz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:06:12.503893 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:12.503806 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f59239c4-202e-45d5-9942-26fb0410d813-istio-token" pod "f59239c4-202e-45d5-9942-26fb0410d813" (UID: "f59239c4-202e-45d5-9942-26fb0410d813"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:06:12.603628 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:12.603561 5052 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f59239c4-202e-45d5-9942-26fb0410d813-istio-podinfo\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:06:12.603628 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:12.603618 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57xfz\" (UniqueName: \"kubernetes.io/projected/f59239c4-202e-45d5-9942-26fb0410d813-kube-api-access-57xfz\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:06:12.603836 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:12.603635 5052 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f59239c4-202e-45d5-9942-26fb0410d813-istio-token\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:06:12.603836 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:12.603649 5052 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f59239c4-202e-45d5-9942-26fb0410d813-istio-envoy\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:06:12.707269 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:12.707187 5052 patch_prober.go:29] interesting pod/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-hwkgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body= Oct 30 17:06:12.707269 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:12.707247 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-hwkgn" podUID="96134e17-039f-4ea3-8522-ff54e3ab6797" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" Oct 30 17:06:13.252507 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:13.249440 5052 generic.go:356] "Generic (PLEG): container finished" podID="f59239c4-202e-45d5-9942-26fb0410d813" containerID="5153a71e73a3e6ac24744eac1a5a2deee8a609fcd75de487da50e3064a05cece" exitCode=0 Oct 30 17:06:13.252507 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:13.249681 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5cz66" event={"ID":"f59239c4-202e-45d5-9942-26fb0410d813","Type":"ContainerDied","Data":"5153a71e73a3e6ac24744eac1a5a2deee8a609fcd75de487da50e3064a05cece"} Oct 30 17:06:13.252507 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:13.249739 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5cz66" event={"ID":"f59239c4-202e-45d5-9942-26fb0410d813","Type":"ContainerDied","Data":"34fc023fffb89f50d0121b3f0857f2a193e37eb0b350c4158ec4466ba2df3bdf"} Oct 30 17:06:13.252507 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:13.249770 5052 scope.go:122] "RemoveContainer" containerID="5153a71e73a3e6ac24744eac1a5a2deee8a609fcd75de487da50e3064a05cece" Oct 30 17:06:13.252507 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:13.249940 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5cz66" Oct 30 17:06:13.279054 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:13.279033 5052 scope.go:122] "RemoveContainer" containerID="5153a71e73a3e6ac24744eac1a5a2deee8a609fcd75de487da50e3064a05cece" Oct 30 17:06:13.279540 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:06:13.279483 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5153a71e73a3e6ac24744eac1a5a2deee8a609fcd75de487da50e3064a05cece\": container with ID starting with 5153a71e73a3e6ac24744eac1a5a2deee8a609fcd75de487da50e3064a05cece not found: ID does not exist" containerID="5153a71e73a3e6ac24744eac1a5a2deee8a609fcd75de487da50e3064a05cece" Oct 30 17:06:13.279681 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:13.279660 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5153a71e73a3e6ac24744eac1a5a2deee8a609fcd75de487da50e3064a05cece"} err="failed to get container status \"5153a71e73a3e6ac24744eac1a5a2deee8a609fcd75de487da50e3064a05cece\": rpc error: code = NotFound desc = could not find container \"5153a71e73a3e6ac24744eac1a5a2deee8a609fcd75de487da50e3064a05cece\": container with ID starting with 5153a71e73a3e6ac24744eac1a5a2deee8a609fcd75de487da50e3064a05cece not found: ID does not exist" Oct 30 17:06:13.312987 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:13.312961 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5cz66"] Oct 30 17:06:13.319763 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:13.319737 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5cz66"] Oct 30 17:06:13.707257 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:13.706701 5052 patch_prober.go:29] interesting pod/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-hwkgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body= Oct 30 17:06:13.707257 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:13.706762 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-hwkgn" podUID="96134e17-039f-4ea3-8522-ff54e3ab6797" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" Oct 30 17:06:14.599638 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:14.599600 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f59239c4-202e-45d5-9942-26fb0410d813" path="/var/lib/kubelet/pods/f59239c4-202e-45d5-9942-26fb0410d813/volumes" Oct 30 17:06:14.706914 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:14.706880 5052 patch_prober.go:29] interesting pod/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-hwkgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body= Oct 30 17:06:14.707114 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:14.706930 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-hwkgn" podUID="96134e17-039f-4ea3-8522-ff54e3ab6797" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" Oct 30 17:06:15.712023 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:15.711993 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-hwkgn" Oct 30 17:06:15.712465 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:15.712444 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-hwkgn" Oct 30 17:06:15.713150 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:15.713129 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-hwkgn" Oct 30 17:06:18.843593 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:06:18.843543 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 17:06:24.060774 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:24.060741 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-330282-openshift-default-5784687855-s68x6"] Oct 30 17:06:24.061642 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:24.061617 5052 cpu_manager.go:414] "containerMap: removing container" podUID="f59239c4-202e-45d5-9942-26fb0410d813" containerName="istio-proxy" Oct 30 17:06:24.061642 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:24.061640 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f59239c4-202e-45d5-9942-26fb0410d813" containerName="istio-proxy" Oct 30 17:06:24.062060 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:24.061933 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="f59239c4-202e-45d5-9942-26fb0410d813" containerName="istio-proxy" Oct 30 17:06:24.067447 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:24.067428 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-330282-openshift-default-5784687855-s68x6" Oct 30 17:06:24.091097 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:24.090369 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-330282-openshift-default-5784687855-s68x6"] Oct 30 17:06:24.091993 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:24.091907 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-330282-openshift-default-dockercfg-glm6f\"" Oct 30 17:06:24.148240 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:24.148202 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce11f38f-0ae0-4881-aa0f-346a0dd513c4-credential-socket\") pod \"e2e-ipp-a-330282-openshift-default-5784687855-s68x6\" (UID: \"ce11f38f-0ae0-4881-aa0f-346a0dd513c4\") " pod="openshift-ingress/e2e-ipp-a-330282-openshift-default-5784687855-s68x6" Oct 30 17:06:24.148443 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:24.148279 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7v2b9\" (UniqueName: \"kubernetes.io/projected/ce11f38f-0ae0-4881-aa0f-346a0dd513c4-kube-api-access-7v2b9\") pod \"e2e-ipp-a-330282-openshift-default-5784687855-s68x6\" (UID: \"ce11f38f-0ae0-4881-aa0f-346a0dd513c4\") " pod="openshift-ingress/e2e-ipp-a-330282-openshift-default-5784687855-s68x6" Oct 30 17:06:24.148443 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:24.148418 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce11f38f-0ae0-4881-aa0f-346a0dd513c4-istiod-ca-cert\") pod \"e2e-ipp-a-330282-openshift-default-5784687855-s68x6\" (UID: \"ce11f38f-0ae0-4881-aa0f-346a0dd513c4\") " pod="openshift-ingress/e2e-ipp-a-330282-openshift-default-5784687855-s68x6" Oct 30 17:06:24.148684 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:24.148486 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce11f38f-0ae0-4881-aa0f-346a0dd513c4-istio-data\") pod \"e2e-ipp-a-330282-openshift-default-5784687855-s68x6\" (UID: \"ce11f38f-0ae0-4881-aa0f-346a0dd513c4\") " pod="openshift-ingress/e2e-ipp-a-330282-openshift-default-5784687855-s68x6" Oct 30 17:06:24.148684 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:24.148660 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce11f38f-0ae0-4881-aa0f-346a0dd513c4-workload-socket\") pod \"e2e-ipp-a-330282-openshift-default-5784687855-s68x6\" (UID: \"ce11f38f-0ae0-4881-aa0f-346a0dd513c4\") " pod="openshift-ingress/e2e-ipp-a-330282-openshift-default-5784687855-s68x6" Oct 30 17:06:24.148774 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:24.148699 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce11f38f-0ae0-4881-aa0f-346a0dd513c4-istio-podinfo\") pod \"e2e-ipp-a-330282-openshift-default-5784687855-s68x6\" (UID: \"ce11f38f-0ae0-4881-aa0f-346a0dd513c4\") " pod="openshift-ingress/e2e-ipp-a-330282-openshift-default-5784687855-s68x6" Oct 30 17:06:24.148836 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:24.148799 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce11f38f-0ae0-4881-aa0f-346a0dd513c4-istio-envoy\") pod \"e2e-ipp-a-330282-openshift-default-5784687855-s68x6\" (UID: \"ce11f38f-0ae0-4881-aa0f-346a0dd513c4\") " pod="openshift-ingress/e2e-ipp-a-330282-openshift-default-5784687855-s68x6" Oct 30 17:06:24.148884 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:24.148862 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce11f38f-0ae0-4881-aa0f-346a0dd513c4-istio-token\") pod \"e2e-ipp-a-330282-openshift-default-5784687855-s68x6\" (UID: \"ce11f38f-0ae0-4881-aa0f-346a0dd513c4\") " pod="openshift-ingress/e2e-ipp-a-330282-openshift-default-5784687855-s68x6" Oct 30 17:06:24.148968 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:24.148946 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce11f38f-0ae0-4881-aa0f-346a0dd513c4-workload-certs\") pod \"e2e-ipp-a-330282-openshift-default-5784687855-s68x6\" (UID: \"ce11f38f-0ae0-4881-aa0f-346a0dd513c4\") " pod="openshift-ingress/e2e-ipp-a-330282-openshift-default-5784687855-s68x6" Oct 30 17:06:24.252857 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:24.252795 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce11f38f-0ae0-4881-aa0f-346a0dd513c4-credential-socket\") pod \"e2e-ipp-a-330282-openshift-default-5784687855-s68x6\" (UID: \"ce11f38f-0ae0-4881-aa0f-346a0dd513c4\") " pod="openshift-ingress/e2e-ipp-a-330282-openshift-default-5784687855-s68x6" Oct 30 17:06:24.262838 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:24.262800 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7v2b9\" (UniqueName: \"kubernetes.io/projected/ce11f38f-0ae0-4881-aa0f-346a0dd513c4-kube-api-access-7v2b9\") pod \"e2e-ipp-a-330282-openshift-default-5784687855-s68x6\" (UID: \"ce11f38f-0ae0-4881-aa0f-346a0dd513c4\") " pod="openshift-ingress/e2e-ipp-a-330282-openshift-default-5784687855-s68x6" Oct 30 17:06:24.263156 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:24.263136 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce11f38f-0ae0-4881-aa0f-346a0dd513c4-istiod-ca-cert\") pod \"e2e-ipp-a-330282-openshift-default-5784687855-s68x6\" (UID: \"ce11f38f-0ae0-4881-aa0f-346a0dd513c4\") " pod="openshift-ingress/e2e-ipp-a-330282-openshift-default-5784687855-s68x6" Oct 30 17:06:24.263324 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:24.263308 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce11f38f-0ae0-4881-aa0f-346a0dd513c4-istio-data\") pod \"e2e-ipp-a-330282-openshift-default-5784687855-s68x6\" (UID: \"ce11f38f-0ae0-4881-aa0f-346a0dd513c4\") " pod="openshift-ingress/e2e-ipp-a-330282-openshift-default-5784687855-s68x6" Oct 30 17:06:24.283614 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:24.263539 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce11f38f-0ae0-4881-aa0f-346a0dd513c4-workload-socket\") pod \"e2e-ipp-a-330282-openshift-default-5784687855-s68x6\" (UID: \"ce11f38f-0ae0-4881-aa0f-346a0dd513c4\") " pod="openshift-ingress/e2e-ipp-a-330282-openshift-default-5784687855-s68x6" Oct 30 17:06:24.284052 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:24.254053 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce11f38f-0ae0-4881-aa0f-346a0dd513c4-credential-socket\") pod \"e2e-ipp-a-330282-openshift-default-5784687855-s68x6\" (UID: \"ce11f38f-0ae0-4881-aa0f-346a0dd513c4\") " pod="openshift-ingress/e2e-ipp-a-330282-openshift-default-5784687855-s68x6" Oct 30 17:06:24.284289 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:24.284263 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce11f38f-0ae0-4881-aa0f-346a0dd513c4-istio-podinfo\") pod \"e2e-ipp-a-330282-openshift-default-5784687855-s68x6\" (UID: \"ce11f38f-0ae0-4881-aa0f-346a0dd513c4\") " pod="openshift-ingress/e2e-ipp-a-330282-openshift-default-5784687855-s68x6" Oct 30 17:06:24.284655 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:24.284633 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce11f38f-0ae0-4881-aa0f-346a0dd513c4-istio-envoy\") pod \"e2e-ipp-a-330282-openshift-default-5784687855-s68x6\" (UID: \"ce11f38f-0ae0-4881-aa0f-346a0dd513c4\") " pod="openshift-ingress/e2e-ipp-a-330282-openshift-default-5784687855-s68x6" Oct 30 17:06:24.284905 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:24.284888 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce11f38f-0ae0-4881-aa0f-346a0dd513c4-istio-token\") pod \"e2e-ipp-a-330282-openshift-default-5784687855-s68x6\" (UID: \"ce11f38f-0ae0-4881-aa0f-346a0dd513c4\") " pod="openshift-ingress/e2e-ipp-a-330282-openshift-default-5784687855-s68x6" Oct 30 17:06:24.285402 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:24.285380 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce11f38f-0ae0-4881-aa0f-346a0dd513c4-workload-certs\") pod \"e2e-ipp-a-330282-openshift-default-5784687855-s68x6\" (UID: \"ce11f38f-0ae0-4881-aa0f-346a0dd513c4\") " pod="openshift-ingress/e2e-ipp-a-330282-openshift-default-5784687855-s68x6" Oct 30 17:06:24.302789 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:24.301341 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce11f38f-0ae0-4881-aa0f-346a0dd513c4-istiod-ca-cert\") pod \"e2e-ipp-a-330282-openshift-default-5784687855-s68x6\" (UID: \"ce11f38f-0ae0-4881-aa0f-346a0dd513c4\") " pod="openshift-ingress/e2e-ipp-a-330282-openshift-default-5784687855-s68x6" Oct 30 17:06:24.315521 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:24.308885 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce11f38f-0ae0-4881-aa0f-346a0dd513c4-workload-socket\") pod \"e2e-ipp-a-330282-openshift-default-5784687855-s68x6\" (UID: \"ce11f38f-0ae0-4881-aa0f-346a0dd513c4\") " pod="openshift-ingress/e2e-ipp-a-330282-openshift-default-5784687855-s68x6" Oct 30 17:06:24.315521 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:24.309040 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce11f38f-0ae0-4881-aa0f-346a0dd513c4-istio-data\") pod \"e2e-ipp-a-330282-openshift-default-5784687855-s68x6\" (UID: \"ce11f38f-0ae0-4881-aa0f-346a0dd513c4\") " pod="openshift-ingress/e2e-ipp-a-330282-openshift-default-5784687855-s68x6" Oct 30 17:06:24.316097 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:24.315834 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce11f38f-0ae0-4881-aa0f-346a0dd513c4-workload-certs\") pod \"e2e-ipp-a-330282-openshift-default-5784687855-s68x6\" (UID: \"ce11f38f-0ae0-4881-aa0f-346a0dd513c4\") " pod="openshift-ingress/e2e-ipp-a-330282-openshift-default-5784687855-s68x6" Oct 30 17:06:24.320423 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:24.320392 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce11f38f-0ae0-4881-aa0f-346a0dd513c4-istio-envoy\") pod \"e2e-ipp-a-330282-openshift-default-5784687855-s68x6\" (UID: \"ce11f38f-0ae0-4881-aa0f-346a0dd513c4\") " pod="openshift-ingress/e2e-ipp-a-330282-openshift-default-5784687855-s68x6" Oct 30 17:06:24.358096 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:24.358013 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce11f38f-0ae0-4881-aa0f-346a0dd513c4-istio-podinfo\") pod \"e2e-ipp-a-330282-openshift-default-5784687855-s68x6\" (UID: \"ce11f38f-0ae0-4881-aa0f-346a0dd513c4\") " pod="openshift-ingress/e2e-ipp-a-330282-openshift-default-5784687855-s68x6" Oct 30 17:06:24.359836 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:24.359692 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce11f38f-0ae0-4881-aa0f-346a0dd513c4-istio-token\") pod \"e2e-ipp-a-330282-openshift-default-5784687855-s68x6\" (UID: \"ce11f38f-0ae0-4881-aa0f-346a0dd513c4\") " pod="openshift-ingress/e2e-ipp-a-330282-openshift-default-5784687855-s68x6" Oct 30 17:06:24.360098 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:24.360079 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7v2b9\" (UniqueName: \"kubernetes.io/projected/ce11f38f-0ae0-4881-aa0f-346a0dd513c4-kube-api-access-7v2b9\") pod \"e2e-ipp-a-330282-openshift-default-5784687855-s68x6\" (UID: \"ce11f38f-0ae0-4881-aa0f-346a0dd513c4\") " pod="openshift-ingress/e2e-ipp-a-330282-openshift-default-5784687855-s68x6" Oct 30 17:06:24.411680 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:24.411635 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-330282-openshift-default-5784687855-s68x6" Oct 30 17:06:24.786466 ip-10-0-131-136 kubenswrapper[5052]: W1030 17:06:24.786434 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce11f38f_0ae0_4881_aa0f_346a0dd513c4.slice/crio-2deda4d44343e1cf5c40a5775bacd0137dec6c3757a86c32285abb8f562cbb36 WatchSource:0}: Error finding container 2deda4d44343e1cf5c40a5775bacd0137dec6c3757a86c32285abb8f562cbb36: Status 404 returned error can't find the container with id 2deda4d44343e1cf5c40a5775bacd0137dec6c3757a86c32285abb8f562cbb36 Oct 30 17:06:24.787460 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:24.787364 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-330282-openshift-default-5784687855-s68x6"] Oct 30 17:06:24.789944 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:24.789914 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:06:24.790034 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:24.789978 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:06:24.790034 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:24.790010 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:06:25.410643 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:25.410609 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-330282-openshift-default-5784687855-s68x6" event={"ID":"ce11f38f-0ae0-4881-aa0f-346a0dd513c4","Type":"ContainerStarted","Data":"2e348d887319104ed3f97de52fb4effecebdb81eab193bcd2a22ac7b0b96a325"} Oct 30 17:06:25.411018 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:25.410661 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-330282-openshift-default-5784687855-s68x6" event={"ID":"ce11f38f-0ae0-4881-aa0f-346a0dd513c4","Type":"ContainerStarted","Data":"2deda4d44343e1cf5c40a5775bacd0137dec6c3757a86c32285abb8f562cbb36"} Oct 30 17:06:25.412953 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:25.412936 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-330282-openshift-default-5784687855-s68x6" Oct 30 17:06:25.419112 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:25.419090 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-330282-openshift-default-5784687855-s68x6" Oct 30 17:06:25.460776 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:25.460714 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-330282-openshift-default-5784687855-s68x6" podStartSLOduration=1.46070218 podStartE2EDuration="1.46070218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:06:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:06:25.457975628 +0000 UTC m=+2740.985310846" watchObservedRunningTime="2026-10-30 17:06:25.46070218 +0000 UTC m=+2740.988037467" Oct 30 17:06:26.420004 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:26.419969 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-330282-openshift-default-5784687855-s68x6" Oct 30 17:06:26.420914 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:26.420892 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-330282-openshift-default-5784687855-s68x6" Oct 30 17:06:30.209365 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:30.209327 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-330282-b5d7c8c9f-6xbhn"] Oct 30 17:06:30.214142 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:30.214118 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-330282-b5d7c8c9f-6xbhn" Oct 30 17:06:30.235329 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:30.235293 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-330282-b5d7c8c9f-6xbhn"] Oct 30 17:06:30.244454 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:30.244418 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-330282-dockercfg-nkcpr\"" Oct 30 17:06:30.244454 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:30.244436 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-330282\"" Oct 30 17:06:30.402605 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:30.341054 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5691fdc-cd71-42ca-9eb7-98d431c101cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-330282-b5d7c8c9f-6xbhn\" (UID: \"d5691fdc-cd71-42ca-9eb7-98d431c101cc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-330282-b5d7c8c9f-6xbhn" Oct 30 17:06:30.402605 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:30.341380 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8vnt\" (UniqueName: \"kubernetes.io/projected/d5691fdc-cd71-42ca-9eb7-98d431c101cc-kube-api-access-v8vnt\") pod \"payload-pre-processing-e2e-ipp-a-330282-b5d7c8c9f-6xbhn\" (UID: \"d5691fdc-cd71-42ca-9eb7-98d431c101cc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-330282-b5d7c8c9f-6xbhn" Oct 30 17:06:30.451086 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:30.451042 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8vnt\" (UniqueName: \"kubernetes.io/projected/d5691fdc-cd71-42ca-9eb7-98d431c101cc-kube-api-access-v8vnt\") pod \"payload-pre-processing-e2e-ipp-a-330282-b5d7c8c9f-6xbhn\" (UID: \"d5691fdc-cd71-42ca-9eb7-98d431c101cc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-330282-b5d7c8c9f-6xbhn" Oct 30 17:06:30.451291 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:30.451221 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5691fdc-cd71-42ca-9eb7-98d431c101cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-330282-b5d7c8c9f-6xbhn\" (UID: \"d5691fdc-cd71-42ca-9eb7-98d431c101cc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-330282-b5d7c8c9f-6xbhn" Oct 30 17:06:30.453113 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:30.452284 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5691fdc-cd71-42ca-9eb7-98d431c101cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-330282-b5d7c8c9f-6xbhn\" (UID: \"d5691fdc-cd71-42ca-9eb7-98d431c101cc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-330282-b5d7c8c9f-6xbhn" Oct 30 17:06:30.492976 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:30.492696 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8vnt\" (UniqueName: \"kubernetes.io/projected/d5691fdc-cd71-42ca-9eb7-98d431c101cc-kube-api-access-v8vnt\") pod \"payload-pre-processing-e2e-ipp-a-330282-b5d7c8c9f-6xbhn\" (UID: \"d5691fdc-cd71-42ca-9eb7-98d431c101cc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-330282-b5d7c8c9f-6xbhn" Oct 30 17:06:30.529560 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:30.529529 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-330282-b5d7c8c9f-6xbhn" Oct 30 17:06:30.842440 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:30.842408 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-330282-b5d7c8c9f-6xbhn"] Oct 30 17:06:30.843519 ip-10-0-131-136 kubenswrapper[5052]: W1030 17:06:30.843494 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5691fdc_cd71_42ca_9eb7_98d431c101cc.slice/crio-13b6680add58f288f7c50d1831f11e516b85c46067fe024ccc8fe57c163985ed WatchSource:0}: Error finding container 13b6680add58f288f7c50d1831f11e516b85c46067fe024ccc8fe57c163985ed: Status 404 returned error can't find the container with id 13b6680add58f288f7c50d1831f11e516b85c46067fe024ccc8fe57c163985ed Oct 30 17:06:31.544751 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:31.544708 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-330282-b5d7c8c9f-6xbhn" event={"ID":"d5691fdc-cd71-42ca-9eb7-98d431c101cc","Type":"ContainerStarted","Data":"27d0c0e2a8d4e3d65db6ffa8d644dcc22862934883dfdbafa5e650fa20b67696"} Oct 30 17:06:31.545189 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:31.544778 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-330282-b5d7c8c9f-6xbhn" event={"ID":"d5691fdc-cd71-42ca-9eb7-98d431c101cc","Type":"ContainerStarted","Data":"13b6680add58f288f7c50d1831f11e516b85c46067fe024ccc8fe57c163985ed"} Oct 30 17:06:31.545189 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:31.544869 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-330282-b5d7c8c9f-6xbhn" Oct 30 17:06:31.590409 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:31.590354 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-330282-b5d7c8c9f-6xbhn" podStartSLOduration=1.5903404559999998 podStartE2EDuration="1.590340456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:06:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:06:31.587566311 +0000 UTC m=+2747.114901531" watchObservedRunningTime="2026-10-30 17:06:31.590340456 +0000 UTC m=+2747.117675674" Oct 30 17:06:42.555829 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:42.555795 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-330282-b5d7c8c9f-6xbhn" Oct 30 17:06:44.216552 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:44.216509 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w6-830ba9/auth-test-model-830ba9-kserve-7cb5656f77-vshw4"] Oct 30 17:06:48.842253 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:06:48.842212 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 17:06:57.368963 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:57.368930 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-e1e830-openshift-default-5949c998b5-8zw7j"] Oct 30 17:06:57.375572 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:57.375549 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-e1e830-openshift-default-5949c998b5-8zw7j" Oct 30 17:06:57.405482 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:57.405452 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-e1e830-openshift-default-dockercfg-nxqw9\"" Oct 30 17:06:57.435638 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:57.435563 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-e1e830-openshift-default-5949c998b5-8zw7j"] Oct 30 17:06:57.526737 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:57.526691 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72mlx\" (UniqueName: \"kubernetes.io/projected/91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a-kube-api-access-72mlx\") pod \"e2e-ipp-b-e1e830-openshift-default-5949c998b5-8zw7j\" (UID: \"91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a\") " pod="openshift-ingress/e2e-ipp-b-e1e830-openshift-default-5949c998b5-8zw7j" Oct 30 17:06:57.527008 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:57.526989 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a-istio-envoy\") pod \"e2e-ipp-b-e1e830-openshift-default-5949c998b5-8zw7j\" (UID: \"91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a\") " pod="openshift-ingress/e2e-ipp-b-e1e830-openshift-default-5949c998b5-8zw7j" Oct 30 17:06:57.573604 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:57.527191 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a-istio-token\") pod \"e2e-ipp-b-e1e830-openshift-default-5949c998b5-8zw7j\" (UID: \"91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a\") " pod="openshift-ingress/e2e-ipp-b-e1e830-openshift-default-5949c998b5-8zw7j" Oct 30 17:06:57.610444 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:57.610400 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a-istio-podinfo\") pod \"e2e-ipp-b-e1e830-openshift-default-5949c998b5-8zw7j\" (UID: \"91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a\") " pod="openshift-ingress/e2e-ipp-b-e1e830-openshift-default-5949c998b5-8zw7j" Oct 30 17:06:57.620111 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:57.620021 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a-workload-socket\") pod \"e2e-ipp-b-e1e830-openshift-default-5949c998b5-8zw7j\" (UID: \"91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a\") " pod="openshift-ingress/e2e-ipp-b-e1e830-openshift-default-5949c998b5-8zw7j" Oct 30 17:06:57.652602 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:57.652242 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a-istio-data\") pod \"e2e-ipp-b-e1e830-openshift-default-5949c998b5-8zw7j\" (UID: \"91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a\") " pod="openshift-ingress/e2e-ipp-b-e1e830-openshift-default-5949c998b5-8zw7j" Oct 30 17:06:57.652602 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:57.652418 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a-istiod-ca-cert\") pod \"e2e-ipp-b-e1e830-openshift-default-5949c998b5-8zw7j\" (UID: \"91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a\") " pod="openshift-ingress/e2e-ipp-b-e1e830-openshift-default-5949c998b5-8zw7j" Oct 30 17:06:57.652925 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:57.652683 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a-credential-socket\") pod \"e2e-ipp-b-e1e830-openshift-default-5949c998b5-8zw7j\" (UID: \"91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a\") " pod="openshift-ingress/e2e-ipp-b-e1e830-openshift-default-5949c998b5-8zw7j" Oct 30 17:06:57.652925 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:57.652728 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a-workload-certs\") pod \"e2e-ipp-b-e1e830-openshift-default-5949c998b5-8zw7j\" (UID: \"91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a\") " pod="openshift-ingress/e2e-ipp-b-e1e830-openshift-default-5949c998b5-8zw7j" Oct 30 17:06:57.755216 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:57.755176 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a-workload-socket\") pod \"e2e-ipp-b-e1e830-openshift-default-5949c998b5-8zw7j\" (UID: \"91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a\") " pod="openshift-ingress/e2e-ipp-b-e1e830-openshift-default-5949c998b5-8zw7j" Oct 30 17:06:57.755405 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:57.755241 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a-istio-data\") pod \"e2e-ipp-b-e1e830-openshift-default-5949c998b5-8zw7j\" (UID: \"91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a\") " pod="openshift-ingress/e2e-ipp-b-e1e830-openshift-default-5949c998b5-8zw7j" Oct 30 17:06:57.755405 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:57.755275 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a-istiod-ca-cert\") pod \"e2e-ipp-b-e1e830-openshift-default-5949c998b5-8zw7j\" (UID: \"91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a\") " pod="openshift-ingress/e2e-ipp-b-e1e830-openshift-default-5949c998b5-8zw7j" Oct 30 17:06:57.755405 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:57.755300 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a-credential-socket\") pod \"e2e-ipp-b-e1e830-openshift-default-5949c998b5-8zw7j\" (UID: \"91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a\") " pod="openshift-ingress/e2e-ipp-b-e1e830-openshift-default-5949c998b5-8zw7j" Oct 30 17:06:57.755405 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:57.755330 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a-workload-certs\") pod \"e2e-ipp-b-e1e830-openshift-default-5949c998b5-8zw7j\" (UID: \"91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a\") " pod="openshift-ingress/e2e-ipp-b-e1e830-openshift-default-5949c998b5-8zw7j" Oct 30 17:06:57.755706 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:57.755468 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72mlx\" (UniqueName: \"kubernetes.io/projected/91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a-kube-api-access-72mlx\") pod \"e2e-ipp-b-e1e830-openshift-default-5949c998b5-8zw7j\" (UID: \"91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a\") " pod="openshift-ingress/e2e-ipp-b-e1e830-openshift-default-5949c998b5-8zw7j" Oct 30 17:06:57.755706 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:57.755510 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a-istio-envoy\") pod \"e2e-ipp-b-e1e830-openshift-default-5949c998b5-8zw7j\" (UID: \"91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a\") " pod="openshift-ingress/e2e-ipp-b-e1e830-openshift-default-5949c998b5-8zw7j" Oct 30 17:06:57.755706 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:57.755594 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a-istio-token\") pod \"e2e-ipp-b-e1e830-openshift-default-5949c998b5-8zw7j\" (UID: \"91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a\") " pod="openshift-ingress/e2e-ipp-b-e1e830-openshift-default-5949c998b5-8zw7j" Oct 30 17:06:57.755706 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:57.755641 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a-istio-podinfo\") pod \"e2e-ipp-b-e1e830-openshift-default-5949c998b5-8zw7j\" (UID: \"91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a\") " pod="openshift-ingress/e2e-ipp-b-e1e830-openshift-default-5949c998b5-8zw7j" Oct 30 17:06:57.755706 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:57.755648 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a-workload-socket\") pod \"e2e-ipp-b-e1e830-openshift-default-5949c998b5-8zw7j\" (UID: \"91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a\") " pod="openshift-ingress/e2e-ipp-b-e1e830-openshift-default-5949c998b5-8zw7j" Oct 30 17:06:57.755978 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:57.755715 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a-istio-data\") pod \"e2e-ipp-b-e1e830-openshift-default-5949c998b5-8zw7j\" (UID: \"91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a\") " pod="openshift-ingress/e2e-ipp-b-e1e830-openshift-default-5949c998b5-8zw7j" Oct 30 17:06:57.755978 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:57.755750 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a-credential-socket\") pod \"e2e-ipp-b-e1e830-openshift-default-5949c998b5-8zw7j\" (UID: \"91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a\") " pod="openshift-ingress/e2e-ipp-b-e1e830-openshift-default-5949c998b5-8zw7j" Oct 30 17:06:57.755978 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:57.755859 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a-workload-certs\") pod \"e2e-ipp-b-e1e830-openshift-default-5949c998b5-8zw7j\" (UID: \"91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a\") " pod="openshift-ingress/e2e-ipp-b-e1e830-openshift-default-5949c998b5-8zw7j" Oct 30 17:06:57.756163 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:57.756140 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a-istiod-ca-cert\") pod \"e2e-ipp-b-e1e830-openshift-default-5949c998b5-8zw7j\" (UID: \"91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a\") " pod="openshift-ingress/e2e-ipp-b-e1e830-openshift-default-5949c998b5-8zw7j" Oct 30 17:06:57.757989 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:57.757965 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a-istio-envoy\") pod \"e2e-ipp-b-e1e830-openshift-default-5949c998b5-8zw7j\" (UID: \"91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a\") " pod="openshift-ingress/e2e-ipp-b-e1e830-openshift-default-5949c998b5-8zw7j" Oct 30 17:06:57.758511 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:57.758493 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a-istio-podinfo\") pod \"e2e-ipp-b-e1e830-openshift-default-5949c998b5-8zw7j\" (UID: \"91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a\") " pod="openshift-ingress/e2e-ipp-b-e1e830-openshift-default-5949c998b5-8zw7j" Oct 30 17:06:57.795313 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:57.795237 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72mlx\" (UniqueName: \"kubernetes.io/projected/91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a-kube-api-access-72mlx\") pod \"e2e-ipp-b-e1e830-openshift-default-5949c998b5-8zw7j\" (UID: \"91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a\") " pod="openshift-ingress/e2e-ipp-b-e1e830-openshift-default-5949c998b5-8zw7j" Oct 30 17:06:57.808605 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:57.807546 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a-istio-token\") pod \"e2e-ipp-b-e1e830-openshift-default-5949c998b5-8zw7j\" (UID: \"91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a\") " pod="openshift-ingress/e2e-ipp-b-e1e830-openshift-default-5949c998b5-8zw7j" Oct 30 17:06:58.000849 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:58.000807 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-e1e830-openshift-default-5949c998b5-8zw7j" Oct 30 17:06:58.285653 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:58.284774 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cdwcj"] Oct 30 17:06:58.393604 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:58.389628 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-e1e830-openshift-default-5949c998b5-8zw7j"] Oct 30 17:06:58.404604 ip-10-0-131-136 kubenswrapper[5052]: W1030 17:06:58.402122 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91887ee1_8aeb_46eb_aefa_a11fe7dd7f9a.slice/crio-15304aaaceecda0be1665ba4e52406b5d55973080bd144092cf37f926b1acefe WatchSource:0}: Error finding container 15304aaaceecda0be1665ba4e52406b5d55973080bd144092cf37f926b1acefe: Status 404 returned error can't find the container with id 15304aaaceecda0be1665ba4e52406b5d55973080bd144092cf37f926b1acefe Oct 30 17:06:58.435360 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:58.435309 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:06:58.436845 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:58.436813 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:06:58.437034 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:58.437015 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:06:58.830508 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:58.830393 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e1e830-openshift-default-5949c998b5-8zw7j" event={"ID":"91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a","Type":"ContainerStarted","Data":"607b3811a04d0437b68e538055a6316e57a386c63ffe58f35f134c1cf8009053"} Oct 30 17:06:58.830508 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:58.830463 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e1e830-openshift-default-5949c998b5-8zw7j" event={"ID":"91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a","Type":"ContainerStarted","Data":"15304aaaceecda0be1665ba4e52406b5d55973080bd144092cf37f926b1acefe"} Oct 30 17:06:58.881526 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:58.881477 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-e1e830-openshift-default-5949c998b5-8zw7j" podStartSLOduration=1.881459015 podStartE2EDuration="1.881459015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:06:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:06:58.877285637 +0000 UTC m=+2774.404620876" watchObservedRunningTime="2026-10-30 17:06:58.881459015 +0000 UTC m=+2774.408794233" Oct 30 17:06:59.002964 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:59.002930 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-e1e830-openshift-default-5949c998b5-8zw7j" Oct 30 17:06:59.047503 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:59.045155 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-e1e830-openshift-default-5949c998b5-8zw7j" Oct 30 17:06:59.241686 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:59.241643 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w6-830ba9/auth-test-model-830ba9-kserve-7cb5656f77-vshw4" podUID="dafdc56a-7a9f-4ed2-b4ab-bf4e25f04437" containerName="main" containerID="cri-o://7c744aee30d2410c8d1b97543c85976cceecbbe6af2c157bdc67ca94a0815e8e" gracePeriod=45 Oct 30 17:06:59.258661 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:59.258623 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w6-830ba9/auth-test-model-830ba9-kserve-7cb5656f77-vshw4" podUID="dafdc56a-7a9f-4ed2-b4ab-bf4e25f04437" containerName="main" probeResult="failure" output="Get \"https://10.132.0.188:8000/ready\": dial tcp 10.132.0.188:8000: connect: connection refused" Oct 30 17:06:59.538512 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:59.538484 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-830ba9/auth-test-model-830ba9-kserve-7cb5656f77-vshw4" Oct 30 17:06:59.602716 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:59.602681 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dafdc56a-7a9f-4ed2-b4ab-bf4e25f04437-tls-certs\" (UniqueName: \"kubernetes.io/secret/dafdc56a-7a9f-4ed2-b4ab-bf4e25f04437-tls-certs\") pod \"dafdc56a-7a9f-4ed2-b4ab-bf4e25f04437\" (UID: \"dafdc56a-7a9f-4ed2-b4ab-bf4e25f04437\") " Oct 30 17:06:59.602896 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:59.602786 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dafdc56a-7a9f-4ed2-b4ab-bf4e25f04437-dshm\" (UniqueName: \"kubernetes.io/empty-dir/dafdc56a-7a9f-4ed2-b4ab-bf4e25f04437-dshm\") pod \"dafdc56a-7a9f-4ed2-b4ab-bf4e25f04437\" (UID: \"dafdc56a-7a9f-4ed2-b4ab-bf4e25f04437\") " Oct 30 17:06:59.602896 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:59.602846 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dafdc56a-7a9f-4ed2-b4ab-bf4e25f04437-home\" (UniqueName: \"kubernetes.io/empty-dir/dafdc56a-7a9f-4ed2-b4ab-bf4e25f04437-home\") pod \"dafdc56a-7a9f-4ed2-b4ab-bf4e25f04437\" (UID: \"dafdc56a-7a9f-4ed2-b4ab-bf4e25f04437\") " Oct 30 17:06:59.603109 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:59.603073 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dafdc56a-7a9f-4ed2-b4ab-bf4e25f04437-kube-api-access-5vvfn\" (UniqueName: \"kubernetes.io/projected/dafdc56a-7a9f-4ed2-b4ab-bf4e25f04437-kube-api-access-5vvfn\") pod \"dafdc56a-7a9f-4ed2-b4ab-bf4e25f04437\" (UID: \"dafdc56a-7a9f-4ed2-b4ab-bf4e25f04437\") " Oct 30 17:06:59.603186 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:59.603158 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dafdc56a-7a9f-4ed2-b4ab-bf4e25f04437-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dafdc56a-7a9f-4ed2-b4ab-bf4e25f04437-tmp-dir\") pod \"dafdc56a-7a9f-4ed2-b4ab-bf4e25f04437\" (UID: \"dafdc56a-7a9f-4ed2-b4ab-bf4e25f04437\") " Oct 30 17:06:59.603246 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:59.603191 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dafdc56a-7a9f-4ed2-b4ab-bf4e25f04437-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dafdc56a-7a9f-4ed2-b4ab-bf4e25f04437-model-cache\") pod \"dafdc56a-7a9f-4ed2-b4ab-bf4e25f04437\" (UID: \"dafdc56a-7a9f-4ed2-b4ab-bf4e25f04437\") " Oct 30 17:06:59.603709 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:59.603645 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dafdc56a-7a9f-4ed2-b4ab-bf4e25f04437-tmp-dir" pod "dafdc56a-7a9f-4ed2-b4ab-bf4e25f04437" (UID: "dafdc56a-7a9f-4ed2-b4ab-bf4e25f04437"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:06:59.603709 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:59.603667 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dafdc56a-7a9f-4ed2-b4ab-bf4e25f04437-home" pod "dafdc56a-7a9f-4ed2-b4ab-bf4e25f04437" (UID: "dafdc56a-7a9f-4ed2-b4ab-bf4e25f04437"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:06:59.603926 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:59.603839 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dafdc56a-7a9f-4ed2-b4ab-bf4e25f04437-model-cache" pod "dafdc56a-7a9f-4ed2-b4ab-bf4e25f04437" (UID: "dafdc56a-7a9f-4ed2-b4ab-bf4e25f04437"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:06:59.604033 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:59.604010 5052 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dafdc56a-7a9f-4ed2-b4ab-bf4e25f04437-tmp-dir\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:06:59.604096 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:59.604036 5052 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dafdc56a-7a9f-4ed2-b4ab-bf4e25f04437-model-cache\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:06:59.604096 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:59.604049 5052 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dafdc56a-7a9f-4ed2-b4ab-bf4e25f04437-home\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:06:59.605861 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:59.605831 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dafdc56a-7a9f-4ed2-b4ab-bf4e25f04437-dshm" pod "dafdc56a-7a9f-4ed2-b4ab-bf4e25f04437" (UID: "dafdc56a-7a9f-4ed2-b4ab-bf4e25f04437"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:06:59.606292 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:59.606266 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dafdc56a-7a9f-4ed2-b4ab-bf4e25f04437-kube-api-access-5vvfn" pod "dafdc56a-7a9f-4ed2-b4ab-bf4e25f04437" (UID: "dafdc56a-7a9f-4ed2-b4ab-bf4e25f04437"). InnerVolumeSpecName "kube-api-access-5vvfn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:06:59.606817 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:59.606784 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dafdc56a-7a9f-4ed2-b4ab-bf4e25f04437-tls-certs" pod "dafdc56a-7a9f-4ed2-b4ab-bf4e25f04437" (UID: "dafdc56a-7a9f-4ed2-b4ab-bf4e25f04437"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:06:59.706507 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:59.706479 5052 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dafdc56a-7a9f-4ed2-b4ab-bf4e25f04437-dshm\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:06:59.706701 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:59.706544 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vvfn\" (UniqueName: \"kubernetes.io/projected/dafdc56a-7a9f-4ed2-b4ab-bf4e25f04437-kube-api-access-5vvfn\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:06:59.706701 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:59.706570 5052 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dafdc56a-7a9f-4ed2-b4ab-bf4e25f04437-tls-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:06:59.885662 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:59.878868 5052 generic.go:356] "Generic (PLEG): container finished" podID="dafdc56a-7a9f-4ed2-b4ab-bf4e25f04437" containerID="7c744aee30d2410c8d1b97543c85976cceecbbe6af2c157bdc67ca94a0815e8e" exitCode=0 Oct 30 17:06:59.885662 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:59.880449 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-830ba9/auth-test-model-830ba9-kserve-7cb5656f77-vshw4" Oct 30 17:06:59.885662 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:59.880739 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-830ba9/auth-test-model-830ba9-kserve-7cb5656f77-vshw4" event={"ID":"dafdc56a-7a9f-4ed2-b4ab-bf4e25f04437","Type":"ContainerDied","Data":"7c744aee30d2410c8d1b97543c85976cceecbbe6af2c157bdc67ca94a0815e8e"} Oct 30 17:06:59.885662 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:59.880815 5052 scope.go:122] "RemoveContainer" containerID="7c744aee30d2410c8d1b97543c85976cceecbbe6af2c157bdc67ca94a0815e8e" Oct 30 17:06:59.885662 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:59.884954 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-e1e830-openshift-default-5949c998b5-8zw7j" Oct 30 17:06:59.885662 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:59.885174 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-e1e830-openshift-default-5949c998b5-8zw7j" Oct 30 17:06:59.885662 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:06:59.885256 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-830ba9/auth-test-model-830ba9-kserve-7cb5656f77-vshw4" event={"ID":"dafdc56a-7a9f-4ed2-b4ab-bf4e25f04437","Type":"ContainerDied","Data":"01ef032675d126119ae152229e8a11f10919c5e61a97c9174e0747a86c5d9c46"} Oct 30 17:07:00.009531 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:00.009199 5052 scope.go:122] "RemoveContainer" containerID="7c744aee30d2410c8d1b97543c85976cceecbbe6af2c157bdc67ca94a0815e8e" Oct 30 17:07:00.010188 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:07:00.010076 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c744aee30d2410c8d1b97543c85976cceecbbe6af2c157bdc67ca94a0815e8e\": container with ID starting with 7c744aee30d2410c8d1b97543c85976cceecbbe6af2c157bdc67ca94a0815e8e not found: ID does not exist" containerID="7c744aee30d2410c8d1b97543c85976cceecbbe6af2c157bdc67ca94a0815e8e" Oct 30 17:07:00.010188 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:00.010118 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c744aee30d2410c8d1b97543c85976cceecbbe6af2c157bdc67ca94a0815e8e"} err="failed to get container status \"7c744aee30d2410c8d1b97543c85976cceecbbe6af2c157bdc67ca94a0815e8e\": rpc error: code = NotFound desc = could not find container \"7c744aee30d2410c8d1b97543c85976cceecbbe6af2c157bdc67ca94a0815e8e\": container with ID starting with 7c744aee30d2410c8d1b97543c85976cceecbbe6af2c157bdc67ca94a0815e8e not found: ID does not exist" Oct 30 17:07:00.206611 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:00.199657 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w6-830ba9/auth-test-model-830ba9-kserve-7cb5656f77-vshw4"] Oct 30 17:07:00.235610 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:00.222402 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w6-830ba9/auth-test-model-830ba9-kserve-7cb5656f77-vshw4"] Oct 30 17:07:00.651490 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:00.651054 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dafdc56a-7a9f-4ed2-b4ab-bf4e25f04437" path="/var/lib/kubelet/pods/dafdc56a-7a9f-4ed2-b4ab-bf4e25f04437/volumes" Oct 30 17:07:04.497075 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:04.497032 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e1e830-6dff758855-k2x7x"] Oct 30 17:07:04.498083 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:04.498061 5052 cpu_manager.go:414] "containerMap: removing container" podUID="dafdc56a-7a9f-4ed2-b4ab-bf4e25f04437" containerName="main" Oct 30 17:07:04.498083 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:04.498084 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dafdc56a-7a9f-4ed2-b4ab-bf4e25f04437" containerName="main" Oct 30 17:07:04.498275 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:04.498260 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="dafdc56a-7a9f-4ed2-b4ab-bf4e25f04437" containerName="main" Oct 30 17:07:04.503702 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:04.503680 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e1e830-6dff758855-k2x7x" Oct 30 17:07:04.521061 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:04.521031 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e1e830-6dff758855-k2x7x"] Oct 30 17:07:04.525336 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:04.525306 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3860a637-7f6c-4b17-ba89-a2f039d081df-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-e1e830-6dff758855-k2x7x\" (UID: \"3860a637-7f6c-4b17-ba89-a2f039d081df\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e1e830-6dff758855-k2x7x" Oct 30 17:07:04.525800 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:04.525611 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6v8s\" (UniqueName: \"kubernetes.io/projected/3860a637-7f6c-4b17-ba89-a2f039d081df-kube-api-access-k6v8s\") pod \"payload-pre-processing-e2e-ipp-b-e1e830-6dff758855-k2x7x\" (UID: \"3860a637-7f6c-4b17-ba89-a2f039d081df\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e1e830-6dff758855-k2x7x" Oct 30 17:07:04.528650 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:04.528616 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-e1e830\"" Oct 30 17:07:04.560249 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:04.560218 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e1e830-5ff5d8674b-clctx"] Oct 30 17:07:04.565121 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:04.565099 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e1e830-5ff5d8674b-clctx" Oct 30 17:07:04.586567 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:04.586533 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e1e830-5ff5d8674b-clctx"] Oct 30 17:07:04.617715 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:04.612976 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-e1e830-dockercfg-vrmhl\"" Oct 30 17:07:04.631674 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:04.630898 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3860a637-7f6c-4b17-ba89-a2f039d081df-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-e1e830-6dff758855-k2x7x\" (UID: \"3860a637-7f6c-4b17-ba89-a2f039d081df\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e1e830-6dff758855-k2x7x" Oct 30 17:07:04.631674 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:04.631157 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8vwb\" (UniqueName: \"kubernetes.io/projected/ef9859ce-d7fd-47cb-b62a-a6c8d605c188-kube-api-access-d8vwb\") pod \"payload-processing-e2e-ipp-b-e1e830-5ff5d8674b-clctx\" (UID: \"ef9859ce-d7fd-47cb-b62a-a6c8d605c188\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e1e830-5ff5d8674b-clctx" Oct 30 17:07:04.631674 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:04.631197 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef9859ce-d7fd-47cb-b62a-a6c8d605c188-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-e1e830-5ff5d8674b-clctx\" (UID: \"ef9859ce-d7fd-47cb-b62a-a6c8d605c188\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e1e830-5ff5d8674b-clctx" Oct 30 17:07:04.631674 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:04.631294 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6v8s\" (UniqueName: \"kubernetes.io/projected/3860a637-7f6c-4b17-ba89-a2f039d081df-kube-api-access-k6v8s\") pod \"payload-pre-processing-e2e-ipp-b-e1e830-6dff758855-k2x7x\" (UID: \"3860a637-7f6c-4b17-ba89-a2f039d081df\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e1e830-6dff758855-k2x7x" Oct 30 17:07:04.687911 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:04.640660 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3860a637-7f6c-4b17-ba89-a2f039d081df-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-e1e830-6dff758855-k2x7x\" (UID: \"3860a637-7f6c-4b17-ba89-a2f039d081df\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e1e830-6dff758855-k2x7x" Oct 30 17:07:04.727610 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:04.720314 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6v8s\" (UniqueName: \"kubernetes.io/projected/3860a637-7f6c-4b17-ba89-a2f039d081df-kube-api-access-k6v8s\") pod \"payload-pre-processing-e2e-ipp-b-e1e830-6dff758855-k2x7x\" (UID: \"3860a637-7f6c-4b17-ba89-a2f039d081df\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e1e830-6dff758855-k2x7x" Oct 30 17:07:04.751650 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:04.745927 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8vwb\" (UniqueName: \"kubernetes.io/projected/ef9859ce-d7fd-47cb-b62a-a6c8d605c188-kube-api-access-d8vwb\") pod \"payload-processing-e2e-ipp-b-e1e830-5ff5d8674b-clctx\" (UID: \"ef9859ce-d7fd-47cb-b62a-a6c8d605c188\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e1e830-5ff5d8674b-clctx" Oct 30 17:07:04.751650 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:04.745999 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef9859ce-d7fd-47cb-b62a-a6c8d605c188-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-e1e830-5ff5d8674b-clctx\" (UID: \"ef9859ce-d7fd-47cb-b62a-a6c8d605c188\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e1e830-5ff5d8674b-clctx" Oct 30 17:07:04.751650 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:04.747243 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef9859ce-d7fd-47cb-b62a-a6c8d605c188-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-e1e830-5ff5d8674b-clctx\" (UID: \"ef9859ce-d7fd-47cb-b62a-a6c8d605c188\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e1e830-5ff5d8674b-clctx" Oct 30 17:07:04.831610 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:04.819276 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e1e830-6dff758855-k2x7x" Oct 30 17:07:04.873897 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:04.867713 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8vwb\" (UniqueName: \"kubernetes.io/projected/ef9859ce-d7fd-47cb-b62a-a6c8d605c188-kube-api-access-d8vwb\") pod \"payload-processing-e2e-ipp-b-e1e830-5ff5d8674b-clctx\" (UID: \"ef9859ce-d7fd-47cb-b62a-a6c8d605c188\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e1e830-5ff5d8674b-clctx" Oct 30 17:07:04.895744 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:04.895454 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e1e830-5ff5d8674b-clctx" Oct 30 17:07:05.234356 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:05.234326 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e1e830-6dff758855-k2x7x"] Oct 30 17:07:05.234493 ip-10-0-131-136 kubenswrapper[5052]: W1030 17:07:05.234467 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3860a637_7f6c_4b17_ba89_a2f039d081df.slice/crio-39579a4e13f10cbff29d3954044b178cd139396c4b2a988e995770b6f35e7cbf WatchSource:0}: Error finding container 39579a4e13f10cbff29d3954044b178cd139396c4b2a988e995770b6f35e7cbf: Status 404 returned error can't find the container with id 39579a4e13f10cbff29d3954044b178cd139396c4b2a988e995770b6f35e7cbf Oct 30 17:07:05.248191 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:05.248163 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e1e830-5ff5d8674b-clctx"] Oct 30 17:07:05.248725 ip-10-0-131-136 kubenswrapper[5052]: W1030 17:07:05.248699 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef9859ce_d7fd_47cb_b62a_a6c8d605c188.slice/crio-27ffc4a156b734480cfa326e04114c1427b656b6f1b827e366d237e03c1859c8 WatchSource:0}: Error finding container 27ffc4a156b734480cfa326e04114c1427b656b6f1b827e366d237e03c1859c8: Status 404 returned error can't find the container with id 27ffc4a156b734480cfa326e04114c1427b656b6f1b827e366d237e03c1859c8 Oct 30 17:07:05.930060 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:05.930018 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e1e830-6dff758855-k2x7x" event={"ID":"3860a637-7f6c-4b17-ba89-a2f039d081df","Type":"ContainerStarted","Data":"c1d2ee99a7a77087cfb20b8bf09781e83ad0f81bcbc83f3e396df65328d96fe3"} Oct 30 17:07:05.930576 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:05.930125 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e1e830-6dff758855-k2x7x" event={"ID":"3860a637-7f6c-4b17-ba89-a2f039d081df","Type":"ContainerStarted","Data":"39579a4e13f10cbff29d3954044b178cd139396c4b2a988e995770b6f35e7cbf"} Oct 30 17:07:05.930576 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:05.930265 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e1e830-6dff758855-k2x7x" Oct 30 17:07:05.936313 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:05.936284 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e1e830-5ff5d8674b-clctx" event={"ID":"ef9859ce-d7fd-47cb-b62a-a6c8d605c188","Type":"ContainerStarted","Data":"d1d19569d03bdb504a7d3cf065ffc9bc53c1fdb3879883be093d74cb8f783b80"} Oct 30 17:07:05.936479 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:05.936330 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e1e830-5ff5d8674b-clctx" event={"ID":"ef9859ce-d7fd-47cb-b62a-a6c8d605c188","Type":"ContainerStarted","Data":"27ffc4a156b734480cfa326e04114c1427b656b6f1b827e366d237e03c1859c8"} Oct 30 17:07:05.936479 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:05.936380 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-e1e830-5ff5d8674b-clctx" Oct 30 17:07:05.980056 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:05.980006 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e1e830-6dff758855-k2x7x" podStartSLOduration=1.979991811 podStartE2EDuration="1.979991811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:07:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:07:05.977095501 +0000 UTC m=+2781.504430722" watchObservedRunningTime="2026-10-30 17:07:05.979991811 +0000 UTC m=+2781.507327034" Oct 30 17:07:06.022885 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:06.022834 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-e1e830-5ff5d8674b-clctx" podStartSLOduration=2.022816792 podStartE2EDuration="2.022816792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:07:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:07:06.019054019 +0000 UTC m=+2781.546389238" watchObservedRunningTime="2026-10-30 17:07:06.022816792 +0000 UTC m=+2781.550152039" Oct 30 17:07:09.436303 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:09.436269 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w6-e5bfb0/auth-test-model-e5bfb0-kserve-8599ccd689-kdckc"] Oct 30 17:07:16.947884 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:16.947852 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e1e830-6dff758855-k2x7x" Oct 30 17:07:16.950512 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:16.948256 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-e1e830-5ff5d8674b-clctx" Oct 30 17:07:18.845706 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:07:18.845673 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 17:07:20.377801 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:20.377767 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-830ba9-5995465bff-gb57g"] Oct 30 17:07:21.577574 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:21.577529 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-55e83e-5d6c866678-kgrpj"] Oct 30 17:07:21.609854 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:21.609821 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-55e83e-577c688fdc-nxvxm"] Oct 30 17:07:24.463084 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:24.463036 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w6-e5bfb0/auth-test-model-e5bfb0-kserve-8599ccd689-kdckc" podUID="64fa47f5-0840-429e-9dd6-2a4a0a20ce57" containerName="main" containerID="cri-o://4fd42315b563a7869560037df1845f10783f52712c3d75380928f6c46dbd3894" gracePeriod=45 Oct 30 17:07:24.621221 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:24.621177 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w6-e5bfb0/auth-test-model-e5bfb0-kserve-8599ccd689-kdckc" podUID="64fa47f5-0840-429e-9dd6-2a4a0a20ce57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.189:8000/ready\": dial tcp 10.132.0.189:8000: connect: connection refused" Oct 30 17:07:24.768956 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:24.768922 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-e5bfb0/auth-test-model-e5bfb0-kserve-8599ccd689-kdckc" Oct 30 17:07:24.870565 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:24.870531 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64fa47f5-0840-429e-9dd6-2a4a0a20ce57-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64fa47f5-0840-429e-9dd6-2a4a0a20ce57-tmp-dir\") pod \"64fa47f5-0840-429e-9dd6-2a4a0a20ce57\" (UID: \"64fa47f5-0840-429e-9dd6-2a4a0a20ce57\") " Oct 30 17:07:24.870565 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:24.870591 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64fa47f5-0840-429e-9dd6-2a4a0a20ce57-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64fa47f5-0840-429e-9dd6-2a4a0a20ce57-model-cache\") pod \"64fa47f5-0840-429e-9dd6-2a4a0a20ce57\" (UID: \"64fa47f5-0840-429e-9dd6-2a4a0a20ce57\") " Oct 30 17:07:24.870879 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:24.870760 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64fa47f5-0840-429e-9dd6-2a4a0a20ce57-kube-api-access-8jvr9\" (UniqueName: \"kubernetes.io/projected/64fa47f5-0840-429e-9dd6-2a4a0a20ce57-kube-api-access-8jvr9\") pod \"64fa47f5-0840-429e-9dd6-2a4a0a20ce57\" (UID: \"64fa47f5-0840-429e-9dd6-2a4a0a20ce57\") " Oct 30 17:07:24.870879 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:24.870871 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64fa47f5-0840-429e-9dd6-2a4a0a20ce57-tls-certs\" (UniqueName: \"kubernetes.io/secret/64fa47f5-0840-429e-9dd6-2a4a0a20ce57-tls-certs\") pod \"64fa47f5-0840-429e-9dd6-2a4a0a20ce57\" (UID: \"64fa47f5-0840-429e-9dd6-2a4a0a20ce57\") " Oct 30 17:07:24.871001 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:24.870859 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64fa47f5-0840-429e-9dd6-2a4a0a20ce57-tmp-dir" pod "64fa47f5-0840-429e-9dd6-2a4a0a20ce57" (UID: "64fa47f5-0840-429e-9dd6-2a4a0a20ce57"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:07:24.871001 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:24.870936 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64fa47f5-0840-429e-9dd6-2a4a0a20ce57-model-cache" pod "64fa47f5-0840-429e-9dd6-2a4a0a20ce57" (UID: "64fa47f5-0840-429e-9dd6-2a4a0a20ce57"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:07:24.871001 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:24.870959 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64fa47f5-0840-429e-9dd6-2a4a0a20ce57-home\" (UniqueName: \"kubernetes.io/empty-dir/64fa47f5-0840-429e-9dd6-2a4a0a20ce57-home\") pod \"64fa47f5-0840-429e-9dd6-2a4a0a20ce57\" (UID: \"64fa47f5-0840-429e-9dd6-2a4a0a20ce57\") " Oct 30 17:07:24.871167 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:24.871027 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64fa47f5-0840-429e-9dd6-2a4a0a20ce57-dshm\" (UniqueName: \"kubernetes.io/empty-dir/64fa47f5-0840-429e-9dd6-2a4a0a20ce57-dshm\") pod \"64fa47f5-0840-429e-9dd6-2a4a0a20ce57\" (UID: \"64fa47f5-0840-429e-9dd6-2a4a0a20ce57\") " Oct 30 17:07:24.871228 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:24.871160 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64fa47f5-0840-429e-9dd6-2a4a0a20ce57-home" pod "64fa47f5-0840-429e-9dd6-2a4a0a20ce57" (UID: "64fa47f5-0840-429e-9dd6-2a4a0a20ce57"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:07:24.871948 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:24.871918 5052 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64fa47f5-0840-429e-9dd6-2a4a0a20ce57-home\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:07:24.872081 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:24.871950 5052 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64fa47f5-0840-429e-9dd6-2a4a0a20ce57-tmp-dir\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:07:24.872081 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:24.871967 5052 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64fa47f5-0840-429e-9dd6-2a4a0a20ce57-model-cache\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:07:24.873308 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:24.873279 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64fa47f5-0840-429e-9dd6-2a4a0a20ce57-tls-certs" pod "64fa47f5-0840-429e-9dd6-2a4a0a20ce57" (UID: "64fa47f5-0840-429e-9dd6-2a4a0a20ce57"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:07:24.873536 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:24.873512 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64fa47f5-0840-429e-9dd6-2a4a0a20ce57-kube-api-access-8jvr9" pod "64fa47f5-0840-429e-9dd6-2a4a0a20ce57" (UID: "64fa47f5-0840-429e-9dd6-2a4a0a20ce57"). InnerVolumeSpecName "kube-api-access-8jvr9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:07:24.873804 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:24.873786 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64fa47f5-0840-429e-9dd6-2a4a0a20ce57-dshm" pod "64fa47f5-0840-429e-9dd6-2a4a0a20ce57" (UID: "64fa47f5-0840-429e-9dd6-2a4a0a20ce57"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:07:24.973943 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:24.973909 5052 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64fa47f5-0840-429e-9dd6-2a4a0a20ce57-dshm\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:07:24.973943 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:24.973944 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jvr9\" (UniqueName: \"kubernetes.io/projected/64fa47f5-0840-429e-9dd6-2a4a0a20ce57-kube-api-access-8jvr9\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:07:24.974153 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:24.973960 5052 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64fa47f5-0840-429e-9dd6-2a4a0a20ce57-tls-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:07:25.093547 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:25.093464 5052 generic.go:356] "Generic (PLEG): container finished" podID="64fa47f5-0840-429e-9dd6-2a4a0a20ce57" containerID="4fd42315b563a7869560037df1845f10783f52712c3d75380928f6c46dbd3894" exitCode=0 Oct 30 17:07:25.093547 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:25.093529 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-e5bfb0/auth-test-model-e5bfb0-kserve-8599ccd689-kdckc" Oct 30 17:07:25.093763 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:25.093554 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-e5bfb0/auth-test-model-e5bfb0-kserve-8599ccd689-kdckc" event={"ID":"64fa47f5-0840-429e-9dd6-2a4a0a20ce57","Type":"ContainerDied","Data":"4fd42315b563a7869560037df1845f10783f52712c3d75380928f6c46dbd3894"} Oct 30 17:07:25.093763 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:25.093617 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-e5bfb0/auth-test-model-e5bfb0-kserve-8599ccd689-kdckc" event={"ID":"64fa47f5-0840-429e-9dd6-2a4a0a20ce57","Type":"ContainerDied","Data":"1438a9707802e06c08930d50bcc15eff78e2a6b21d00b5568cd2cc16a2bf22ca"} Oct 30 17:07:25.093763 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:25.093639 5052 scope.go:122] "RemoveContainer" containerID="4fd42315b563a7869560037df1845f10783f52712c3d75380928f6c46dbd3894" Oct 30 17:07:25.104961 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:25.104942 5052 scope.go:122] "RemoveContainer" containerID="4fd42315b563a7869560037df1845f10783f52712c3d75380928f6c46dbd3894" Oct 30 17:07:25.105250 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:07:25.105223 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fd42315b563a7869560037df1845f10783f52712c3d75380928f6c46dbd3894\": container with ID starting with 4fd42315b563a7869560037df1845f10783f52712c3d75380928f6c46dbd3894 not found: ID does not exist" containerID="4fd42315b563a7869560037df1845f10783f52712c3d75380928f6c46dbd3894" Oct 30 17:07:25.105296 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:25.105249 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fd42315b563a7869560037df1845f10783f52712c3d75380928f6c46dbd3894"} err="failed to get container status \"4fd42315b563a7869560037df1845f10783f52712c3d75380928f6c46dbd3894\": rpc error: code = NotFound desc = could not find container \"4fd42315b563a7869560037df1845f10783f52712c3d75380928f6c46dbd3894\": container with ID starting with 4fd42315b563a7869560037df1845f10783f52712c3d75380928f6c46dbd3894 not found: ID does not exist" Oct 30 17:07:25.150660 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:25.150628 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w6-e5bfb0/auth-test-model-e5bfb0-kserve-8599ccd689-kdckc"] Oct 30 17:07:25.163667 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:25.163644 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w6-e5bfb0/auth-test-model-e5bfb0-kserve-8599ccd689-kdckc"] Oct 30 17:07:25.379339 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:25.379233 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-830ba9-5995465bff-gb57g" podUID="5f25133c-8cae-4395-807d-8b77508a9285" containerName="payload-pre-processing" containerID="cri-o://c988b975c2c4e83f36d5f5bcdd16c6fa65f8d932ba95d2ee3ab129807bec5d1f" gracePeriod=25 Oct 30 17:07:26.578964 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:26.578923 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-55e83e-5d6c866678-kgrpj" podUID="3b0ced1c-8a0f-453b-bd03-8f06a02f5f36" containerName="payload-processing" containerID="cri-o://a77f5527cf677b15012dba899fb9689313601cc4143a5f31b49814e4775756fa" gracePeriod=25 Oct 30 17:07:26.602047 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:26.602017 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64fa47f5-0840-429e-9dd6-2a4a0a20ce57" path="/var/lib/kubelet/pods/64fa47f5-0840-429e-9dd6-2a4a0a20ce57/volumes" Oct 30 17:07:26.610273 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:26.610230 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-55e83e-577c688fdc-nxvxm" podUID="bec23cc5-623a-4c22-84e7-afc66684df13" containerName="payload-pre-processing" containerID="cri-o://02a263aea2b0e241213df4da6fe08207120a1aeb61dbdcb16d0b9c6d6806175d" gracePeriod=25 Oct 30 17:07:26.912441 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:26.912406 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-55e83e-5d6c866678-kgrpj" Oct 30 17:07:26.976979 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:26.976941 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-55e83e-577c688fdc-nxvxm" Oct 30 17:07:27.006797 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:27.006761 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b0ced1c-8a0f-453b-bd03-8f06a02f5f36-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b0ced1c-8a0f-453b-bd03-8f06a02f5f36-plugins-config-volume\") pod \"3b0ced1c-8a0f-453b-bd03-8f06a02f5f36\" (UID: \"3b0ced1c-8a0f-453b-bd03-8f06a02f5f36\") " Oct 30 17:07:27.007234 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:27.007199 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b0ced1c-8a0f-453b-bd03-8f06a02f5f36-plugins-config-volume" pod "3b0ced1c-8a0f-453b-bd03-8f06a02f5f36" (UID: "3b0ced1c-8a0f-453b-bd03-8f06a02f5f36"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:07:27.007426 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:27.007397 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b0ced1c-8a0f-453b-bd03-8f06a02f5f36-kube-api-access-wqb5f\" (UniqueName: \"kubernetes.io/projected/3b0ced1c-8a0f-453b-bd03-8f06a02f5f36-kube-api-access-wqb5f\") pod \"3b0ced1c-8a0f-453b-bd03-8f06a02f5f36\" (UID: \"3b0ced1c-8a0f-453b-bd03-8f06a02f5f36\") " Oct 30 17:07:27.008151 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:27.008120 5052 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b0ced1c-8a0f-453b-bd03-8f06a02f5f36-plugins-config-volume\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:07:27.010520 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:27.010484 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b0ced1c-8a0f-453b-bd03-8f06a02f5f36-kube-api-access-wqb5f" pod "3b0ced1c-8a0f-453b-bd03-8f06a02f5f36" (UID: "3b0ced1c-8a0f-453b-bd03-8f06a02f5f36"). InnerVolumeSpecName "kube-api-access-wqb5f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:07:27.109524 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:27.109436 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bec23cc5-623a-4c22-84e7-afc66684df13-kube-api-access-v5xks\" (UniqueName: \"kubernetes.io/projected/bec23cc5-623a-4c22-84e7-afc66684df13-kube-api-access-v5xks\") pod \"bec23cc5-623a-4c22-84e7-afc66684df13\" (UID: \"bec23cc5-623a-4c22-84e7-afc66684df13\") " Oct 30 17:07:27.109831 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:27.109809 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bec23cc5-623a-4c22-84e7-afc66684df13-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bec23cc5-623a-4c22-84e7-afc66684df13-plugins-config-volume\") pod \"bec23cc5-623a-4c22-84e7-afc66684df13\" (UID: \"bec23cc5-623a-4c22-84e7-afc66684df13\") " Oct 30 17:07:27.110223 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:27.110199 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bec23cc5-623a-4c22-84e7-afc66684df13-plugins-config-volume" pod "bec23cc5-623a-4c22-84e7-afc66684df13" (UID: "bec23cc5-623a-4c22-84e7-afc66684df13"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:07:27.110494 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:27.110469 5052 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bec23cc5-623a-4c22-84e7-afc66684df13-plugins-config-volume\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:07:27.110647 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:27.110498 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqb5f\" (UniqueName: \"kubernetes.io/projected/3b0ced1c-8a0f-453b-bd03-8f06a02f5f36-kube-api-access-wqb5f\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:07:27.110980 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:27.110952 5052 generic.go:356] "Generic (PLEG): container finished" podID="3b0ced1c-8a0f-453b-bd03-8f06a02f5f36" containerID="a77f5527cf677b15012dba899fb9689313601cc4143a5f31b49814e4775756fa" exitCode=0 Oct 30 17:07:27.111066 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:27.111006 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-55e83e-5d6c866678-kgrpj" Oct 30 17:07:27.111066 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:27.111041 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-55e83e-5d6c866678-kgrpj" event={"ID":"3b0ced1c-8a0f-453b-bd03-8f06a02f5f36","Type":"ContainerDied","Data":"a77f5527cf677b15012dba899fb9689313601cc4143a5f31b49814e4775756fa"} Oct 30 17:07:27.111150 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:27.111116 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-55e83e-5d6c866678-kgrpj" event={"ID":"3b0ced1c-8a0f-453b-bd03-8f06a02f5f36","Type":"ContainerDied","Data":"e0bdc2009704248cd376d4a749e2a425e16c7d7667199a656ed1dd2ae477f267"} Oct 30 17:07:27.111205 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:27.111147 5052 scope.go:122] "RemoveContainer" containerID="a77f5527cf677b15012dba899fb9689313601cc4143a5f31b49814e4775756fa" Oct 30 17:07:27.112082 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:27.112062 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bec23cc5-623a-4c22-84e7-afc66684df13-kube-api-access-v5xks" pod "bec23cc5-623a-4c22-84e7-afc66684df13" (UID: "bec23cc5-623a-4c22-84e7-afc66684df13"). InnerVolumeSpecName "kube-api-access-v5xks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:07:27.112738 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:27.112708 5052 generic.go:356] "Generic (PLEG): container finished" podID="bec23cc5-623a-4c22-84e7-afc66684df13" containerID="02a263aea2b0e241213df4da6fe08207120a1aeb61dbdcb16d0b9c6d6806175d" exitCode=0 Oct 30 17:07:27.112853 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:27.112766 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-55e83e-577c688fdc-nxvxm" Oct 30 17:07:27.112853 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:27.112779 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-55e83e-577c688fdc-nxvxm" event={"ID":"bec23cc5-623a-4c22-84e7-afc66684df13","Type":"ContainerDied","Data":"02a263aea2b0e241213df4da6fe08207120a1aeb61dbdcb16d0b9c6d6806175d"} Oct 30 17:07:27.112853 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:27.112824 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-55e83e-577c688fdc-nxvxm" event={"ID":"bec23cc5-623a-4c22-84e7-afc66684df13","Type":"ContainerDied","Data":"5be4f64d0e020356a44d973c5e875e44c9fb039b082eb917bc1babd64222ae40"} Oct 30 17:07:27.122866 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:27.122843 5052 scope.go:122] "RemoveContainer" containerID="a77f5527cf677b15012dba899fb9689313601cc4143a5f31b49814e4775756fa" Oct 30 17:07:27.123178 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:07:27.123157 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a77f5527cf677b15012dba899fb9689313601cc4143a5f31b49814e4775756fa\": container with ID starting with a77f5527cf677b15012dba899fb9689313601cc4143a5f31b49814e4775756fa not found: ID does not exist" containerID="a77f5527cf677b15012dba899fb9689313601cc4143a5f31b49814e4775756fa" Oct 30 17:07:27.123264 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:27.123201 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a77f5527cf677b15012dba899fb9689313601cc4143a5f31b49814e4775756fa"} err="failed to get container status \"a77f5527cf677b15012dba899fb9689313601cc4143a5f31b49814e4775756fa\": rpc error: code = NotFound desc = could not find container \"a77f5527cf677b15012dba899fb9689313601cc4143a5f31b49814e4775756fa\": container with ID starting with a77f5527cf677b15012dba899fb9689313601cc4143a5f31b49814e4775756fa not found: ID does not exist" Oct 30 17:07:27.123264 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:27.123220 5052 scope.go:122] "RemoveContainer" containerID="02a263aea2b0e241213df4da6fe08207120a1aeb61dbdcb16d0b9c6d6806175d" Oct 30 17:07:27.135217 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:27.135193 5052 scope.go:122] "RemoveContainer" containerID="02a263aea2b0e241213df4da6fe08207120a1aeb61dbdcb16d0b9c6d6806175d" Oct 30 17:07:27.135625 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:07:27.135576 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02a263aea2b0e241213df4da6fe08207120a1aeb61dbdcb16d0b9c6d6806175d\": container with ID starting with 02a263aea2b0e241213df4da6fe08207120a1aeb61dbdcb16d0b9c6d6806175d not found: ID does not exist" containerID="02a263aea2b0e241213df4da6fe08207120a1aeb61dbdcb16d0b9c6d6806175d" Oct 30 17:07:27.135755 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:27.135631 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02a263aea2b0e241213df4da6fe08207120a1aeb61dbdcb16d0b9c6d6806175d"} err="failed to get container status \"02a263aea2b0e241213df4da6fe08207120a1aeb61dbdcb16d0b9c6d6806175d\": rpc error: code = NotFound desc = could not find container \"02a263aea2b0e241213df4da6fe08207120a1aeb61dbdcb16d0b9c6d6806175d\": container with ID starting with 02a263aea2b0e241213df4da6fe08207120a1aeb61dbdcb16d0b9c6d6806175d not found: ID does not exist" Oct 30 17:07:27.167680 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:27.167602 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-55e83e-5d6c866678-kgrpj"] Oct 30 17:07:27.171901 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:27.171872 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-55e83e-5d6c866678-kgrpj"] Oct 30 17:07:27.209144 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:27.209111 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-55e83e-577c688fdc-nxvxm"] Oct 30 17:07:27.211978 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:27.211946 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5xks\" (UniqueName: \"kubernetes.io/projected/bec23cc5-623a-4c22-84e7-afc66684df13-kube-api-access-v5xks\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:07:27.216875 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:27.216850 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-55e83e-577c688fdc-nxvxm"] Oct 30 17:07:28.599738 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:28.599710 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b0ced1c-8a0f-453b-bd03-8f06a02f5f36" path="/var/lib/kubelet/pods/3b0ced1c-8a0f-453b-bd03-8f06a02f5f36/volumes" Oct 30 17:07:28.600213 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:28.600198 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bec23cc5-623a-4c22-84e7-afc66684df13" path="/var/lib/kubelet/pods/bec23cc5-623a-4c22-84e7-afc66684df13/volumes" Oct 30 17:07:32.893112 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:32.893067 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-830ba9-5995465bff-gb57g" podUID="5f25133c-8cae-4395-807d-8b77508a9285" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:07:39.563240 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:39.563205 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-6jzqj"] Oct 30 17:07:39.564062 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:39.564040 5052 cpu_manager.go:414] "containerMap: removing container" podUID="bec23cc5-623a-4c22-84e7-afc66684df13" containerName="payload-pre-processing" Oct 30 17:07:39.564062 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:39.564061 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bec23cc5-623a-4c22-84e7-afc66684df13" containerName="payload-pre-processing" Oct 30 17:07:39.564258 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:39.564083 5052 cpu_manager.go:414] "containerMap: removing container" podUID="3b0ced1c-8a0f-453b-bd03-8f06a02f5f36" containerName="payload-processing" Oct 30 17:07:39.564258 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:39.564091 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b0ced1c-8a0f-453b-bd03-8f06a02f5f36" containerName="payload-processing" Oct 30 17:07:39.564258 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:39.564129 5052 cpu_manager.go:414] "containerMap: removing container" podUID="64fa47f5-0840-429e-9dd6-2a4a0a20ce57" containerName="main" Oct 30 17:07:39.564258 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:39.564136 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64fa47f5-0840-429e-9dd6-2a4a0a20ce57" containerName="main" Oct 30 17:07:39.564526 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:39.564285 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="bec23cc5-623a-4c22-84e7-afc66684df13" containerName="payload-pre-processing" Oct 30 17:07:39.564526 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:39.564295 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b0ced1c-8a0f-453b-bd03-8f06a02f5f36" containerName="payload-processing" Oct 30 17:07:39.564526 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:39.564307 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="64fa47f5-0840-429e-9dd6-2a4a0a20ce57" containerName="main" Oct 30 17:07:39.569556 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:39.569530 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-6jzqj" Oct 30 17:07:39.585842 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:39.585814 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-e4048c-openshift-default-dockercfg-ck5hk\"" Oct 30 17:07:39.605768 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:39.605738 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-6jzqj"] Oct 30 17:07:39.645838 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:39.645798 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwrx9\" (UniqueName: \"kubernetes.io/projected/1fb4a801-fd82-4036-ba89-79fdf32d3be3-kube-api-access-lwrx9\") pod \"e2e-ipp-a-e4048c-openshift-default-5c764747c9-6jzqj\" (UID: \"1fb4a801-fd82-4036-ba89-79fdf32d3be3\") " pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-6jzqj" Oct 30 17:07:39.646019 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:39.645913 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fb4a801-fd82-4036-ba89-79fdf32d3be3-credential-socket\") pod \"e2e-ipp-a-e4048c-openshift-default-5c764747c9-6jzqj\" (UID: \"1fb4a801-fd82-4036-ba89-79fdf32d3be3\") " pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-6jzqj" Oct 30 17:07:39.646019 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:39.645955 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fb4a801-fd82-4036-ba89-79fdf32d3be3-istio-podinfo\") pod \"e2e-ipp-a-e4048c-openshift-default-5c764747c9-6jzqj\" (UID: \"1fb4a801-fd82-4036-ba89-79fdf32d3be3\") " pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-6jzqj" Oct 30 17:07:39.646019 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:39.645994 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fb4a801-fd82-4036-ba89-79fdf32d3be3-workload-certs\") pod \"e2e-ipp-a-e4048c-openshift-default-5c764747c9-6jzqj\" (UID: \"1fb4a801-fd82-4036-ba89-79fdf32d3be3\") " pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-6jzqj" Oct 30 17:07:39.646198 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:39.646037 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fb4a801-fd82-4036-ba89-79fdf32d3be3-istiod-ca-cert\") pod \"e2e-ipp-a-e4048c-openshift-default-5c764747c9-6jzqj\" (UID: \"1fb4a801-fd82-4036-ba89-79fdf32d3be3\") " pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-6jzqj" Oct 30 17:07:39.646198 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:39.646104 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fb4a801-fd82-4036-ba89-79fdf32d3be3-istio-data\") pod \"e2e-ipp-a-e4048c-openshift-default-5c764747c9-6jzqj\" (UID: \"1fb4a801-fd82-4036-ba89-79fdf32d3be3\") " pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-6jzqj" Oct 30 17:07:39.646198 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:39.646147 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fb4a801-fd82-4036-ba89-79fdf32d3be3-istio-envoy\") pod \"e2e-ipp-a-e4048c-openshift-default-5c764747c9-6jzqj\" (UID: \"1fb4a801-fd82-4036-ba89-79fdf32d3be3\") " pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-6jzqj" Oct 30 17:07:39.646373 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:39.646195 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fb4a801-fd82-4036-ba89-79fdf32d3be3-istio-token\") pod \"e2e-ipp-a-e4048c-openshift-default-5c764747c9-6jzqj\" (UID: \"1fb4a801-fd82-4036-ba89-79fdf32d3be3\") " pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-6jzqj" Oct 30 17:07:39.646373 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:39.646251 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fb4a801-fd82-4036-ba89-79fdf32d3be3-workload-socket\") pod \"e2e-ipp-a-e4048c-openshift-default-5c764747c9-6jzqj\" (UID: \"1fb4a801-fd82-4036-ba89-79fdf32d3be3\") " pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-6jzqj" Oct 30 17:07:39.762602 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:39.755843 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fb4a801-fd82-4036-ba89-79fdf32d3be3-istiod-ca-cert\") pod \"e2e-ipp-a-e4048c-openshift-default-5c764747c9-6jzqj\" (UID: \"1fb4a801-fd82-4036-ba89-79fdf32d3be3\") " pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-6jzqj" Oct 30 17:07:39.762602 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:39.755962 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fb4a801-fd82-4036-ba89-79fdf32d3be3-istio-data\") pod \"e2e-ipp-a-e4048c-openshift-default-5c764747c9-6jzqj\" (UID: \"1fb4a801-fd82-4036-ba89-79fdf32d3be3\") " pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-6jzqj" Oct 30 17:07:39.762602 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:39.756006 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fb4a801-fd82-4036-ba89-79fdf32d3be3-istio-envoy\") pod \"e2e-ipp-a-e4048c-openshift-default-5c764747c9-6jzqj\" (UID: \"1fb4a801-fd82-4036-ba89-79fdf32d3be3\") " pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-6jzqj" Oct 30 17:07:39.762602 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:39.756049 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fb4a801-fd82-4036-ba89-79fdf32d3be3-istio-token\") pod \"e2e-ipp-a-e4048c-openshift-default-5c764747c9-6jzqj\" (UID: \"1fb4a801-fd82-4036-ba89-79fdf32d3be3\") " pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-6jzqj" Oct 30 17:07:39.762602 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:39.756116 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fb4a801-fd82-4036-ba89-79fdf32d3be3-workload-socket\") pod \"e2e-ipp-a-e4048c-openshift-default-5c764747c9-6jzqj\" (UID: \"1fb4a801-fd82-4036-ba89-79fdf32d3be3\") " pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-6jzqj" Oct 30 17:07:39.762602 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:39.756413 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwrx9\" (UniqueName: \"kubernetes.io/projected/1fb4a801-fd82-4036-ba89-79fdf32d3be3-kube-api-access-lwrx9\") pod \"e2e-ipp-a-e4048c-openshift-default-5c764747c9-6jzqj\" (UID: \"1fb4a801-fd82-4036-ba89-79fdf32d3be3\") " pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-6jzqj" Oct 30 17:07:39.762602 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:39.756499 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fb4a801-fd82-4036-ba89-79fdf32d3be3-credential-socket\") pod \"e2e-ipp-a-e4048c-openshift-default-5c764747c9-6jzqj\" (UID: \"1fb4a801-fd82-4036-ba89-79fdf32d3be3\") " pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-6jzqj" Oct 30 17:07:39.762602 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:39.756537 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fb4a801-fd82-4036-ba89-79fdf32d3be3-istio-podinfo\") pod \"e2e-ipp-a-e4048c-openshift-default-5c764747c9-6jzqj\" (UID: \"1fb4a801-fd82-4036-ba89-79fdf32d3be3\") " pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-6jzqj" Oct 30 17:07:39.762602 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:39.756592 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fb4a801-fd82-4036-ba89-79fdf32d3be3-workload-certs\") pod \"e2e-ipp-a-e4048c-openshift-default-5c764747c9-6jzqj\" (UID: \"1fb4a801-fd82-4036-ba89-79fdf32d3be3\") " pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-6jzqj" Oct 30 17:07:39.762602 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:39.757843 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fb4a801-fd82-4036-ba89-79fdf32d3be3-istiod-ca-cert\") pod \"e2e-ipp-a-e4048c-openshift-default-5c764747c9-6jzqj\" (UID: \"1fb4a801-fd82-4036-ba89-79fdf32d3be3\") " pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-6jzqj" Oct 30 17:07:39.762602 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:39.758092 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fb4a801-fd82-4036-ba89-79fdf32d3be3-istio-data\") pod \"e2e-ipp-a-e4048c-openshift-default-5c764747c9-6jzqj\" (UID: \"1fb4a801-fd82-4036-ba89-79fdf32d3be3\") " pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-6jzqj" Oct 30 17:07:39.784621 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:39.772625 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fb4a801-fd82-4036-ba89-79fdf32d3be3-workload-socket\") pod \"e2e-ipp-a-e4048c-openshift-default-5c764747c9-6jzqj\" (UID: \"1fb4a801-fd82-4036-ba89-79fdf32d3be3\") " pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-6jzqj" Oct 30 17:07:39.784621 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:39.773391 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fb4a801-fd82-4036-ba89-79fdf32d3be3-credential-socket\") pod \"e2e-ipp-a-e4048c-openshift-default-5c764747c9-6jzqj\" (UID: \"1fb4a801-fd82-4036-ba89-79fdf32d3be3\") " pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-6jzqj" Oct 30 17:07:39.786469 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:39.786405 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fb4a801-fd82-4036-ba89-79fdf32d3be3-workload-certs\") pod \"e2e-ipp-a-e4048c-openshift-default-5c764747c9-6jzqj\" (UID: \"1fb4a801-fd82-4036-ba89-79fdf32d3be3\") " pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-6jzqj" Oct 30 17:07:39.816442 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:39.816321 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fb4a801-fd82-4036-ba89-79fdf32d3be3-istio-podinfo\") pod \"e2e-ipp-a-e4048c-openshift-default-5c764747c9-6jzqj\" (UID: \"1fb4a801-fd82-4036-ba89-79fdf32d3be3\") " pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-6jzqj" Oct 30 17:07:39.818323 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:39.818280 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fb4a801-fd82-4036-ba89-79fdf32d3be3-istio-envoy\") pod \"e2e-ipp-a-e4048c-openshift-default-5c764747c9-6jzqj\" (UID: \"1fb4a801-fd82-4036-ba89-79fdf32d3be3\") " pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-6jzqj" Oct 30 17:07:39.823312 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:39.823283 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwrx9\" (UniqueName: \"kubernetes.io/projected/1fb4a801-fd82-4036-ba89-79fdf32d3be3-kube-api-access-lwrx9\") pod \"e2e-ipp-a-e4048c-openshift-default-5c764747c9-6jzqj\" (UID: \"1fb4a801-fd82-4036-ba89-79fdf32d3be3\") " pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-6jzqj" Oct 30 17:07:39.824447 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:39.824383 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fb4a801-fd82-4036-ba89-79fdf32d3be3-istio-token\") pod \"e2e-ipp-a-e4048c-openshift-default-5c764747c9-6jzqj\" (UID: \"1fb4a801-fd82-4036-ba89-79fdf32d3be3\") " pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-6jzqj" Oct 30 17:07:39.920335 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:39.913207 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-6jzqj" Oct 30 17:07:40.284621 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:40.284549 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-6jzqj"] Oct 30 17:07:40.284912 ip-10-0-131-136 kubenswrapper[5052]: W1030 17:07:40.284885 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fb4a801_fd82_4036_ba89_79fdf32d3be3.slice/crio-73fb543e4645885a1ddf6207dcef913129be63e23f30da9fe57d905c7f899baf WatchSource:0}: Error finding container 73fb543e4645885a1ddf6207dcef913129be63e23f30da9fe57d905c7f899baf: Status 404 returned error can't find the container with id 73fb543e4645885a1ddf6207dcef913129be63e23f30da9fe57d905c7f899baf Oct 30 17:07:40.288002 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:40.287967 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:07:40.288101 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:40.288037 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:07:40.288101 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:40.288069 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:07:41.236816 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:41.236775 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-6jzqj" event={"ID":"1fb4a801-fd82-4036-ba89-79fdf32d3be3","Type":"ContainerStarted","Data":"8ecd743ed8132bc3c3146a7ea6e84cb5048848ec8e3982af6e37ab90b57a97a0"} Oct 30 17:07:41.237252 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:41.236843 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-6jzqj" event={"ID":"1fb4a801-fd82-4036-ba89-79fdf32d3be3","Type":"ContainerStarted","Data":"73fb543e4645885a1ddf6207dcef913129be63e23f30da9fe57d905c7f899baf"} Oct 30 17:07:41.299682 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:41.299618 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-6jzqj" podStartSLOduration=2.299601426 podStartE2EDuration="2.299601426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:07:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:07:41.297193284 +0000 UTC m=+2816.824528529" watchObservedRunningTime="2026-10-30 17:07:41.299601426 +0000 UTC m=+2816.826936644" Oct 30 17:07:41.914893 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:41.914812 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-6jzqj" Oct 30 17:07:41.920772 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:41.920743 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-6jzqj" Oct 30 17:07:42.244481 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:42.244398 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-6jzqj" Oct 30 17:07:42.245311 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:42.245289 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-6jzqj" Oct 30 17:07:42.888295 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:42.888259 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-830ba9-5995465bff-gb57g" podUID="5f25133c-8cae-4395-807d-8b77508a9285" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:07:45.394911 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:45.394880 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e4048c-d969759b4-c9knx"] Oct 30 17:07:45.399490 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:45.399469 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e4048c-d969759b4-c9knx" Oct 30 17:07:45.417094 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:45.417071 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e4048c-d969759b4-c9knx"] Oct 30 17:07:45.425924 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:45.425898 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-e4048c\"" Oct 30 17:07:45.456308 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:45.456277 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e4048c-5765848847-mg8pn"] Oct 30 17:07:45.461843 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:45.461818 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e4048c-5765848847-mg8pn" Oct 30 17:07:45.483833 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:45.483809 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-e4048c-dockercfg-sqgkn\"" Oct 30 17:07:45.489639 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:45.489617 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e4048c-5765848847-mg8pn"] Oct 30 17:07:45.573001 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:45.562047 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vn4gt\" (UniqueName: \"kubernetes.io/projected/95b7602a-63c6-4dd8-9f9f-198a1866ce43-kube-api-access-vn4gt\") pod \"payload-processing-e2e-ipp-a-e4048c-d969759b4-c9knx\" (UID: \"95b7602a-63c6-4dd8-9f9f-198a1866ce43\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e4048c-d969759b4-c9knx" Oct 30 17:07:45.573001 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:45.562218 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-js6vs\" (UniqueName: \"kubernetes.io/projected/8decf675-2bb4-431f-9ab6-da639afd4f55-kube-api-access-js6vs\") pod \"payload-pre-processing-e2e-ipp-a-e4048c-5765848847-mg8pn\" (UID: \"8decf675-2bb4-431f-9ab6-da639afd4f55\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e4048c-5765848847-mg8pn" Oct 30 17:07:45.573001 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:45.562246 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95b7602a-63c6-4dd8-9f9f-198a1866ce43-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e4048c-d969759b4-c9knx\" (UID: \"95b7602a-63c6-4dd8-9f9f-198a1866ce43\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e4048c-d969759b4-c9knx" Oct 30 17:07:45.573001 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:45.562403 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8decf675-2bb4-431f-9ab6-da639afd4f55-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e4048c-5765848847-mg8pn\" (UID: \"8decf675-2bb4-431f-9ab6-da639afd4f55\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e4048c-5765848847-mg8pn" Oct 30 17:07:45.670644 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:45.667745 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vn4gt\" (UniqueName: \"kubernetes.io/projected/95b7602a-63c6-4dd8-9f9f-198a1866ce43-kube-api-access-vn4gt\") pod \"payload-processing-e2e-ipp-a-e4048c-d969759b4-c9knx\" (UID: \"95b7602a-63c6-4dd8-9f9f-198a1866ce43\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e4048c-d969759b4-c9knx" Oct 30 17:07:45.670644 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:45.667837 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-js6vs\" (UniqueName: \"kubernetes.io/projected/8decf675-2bb4-431f-9ab6-da639afd4f55-kube-api-access-js6vs\") pod \"payload-pre-processing-e2e-ipp-a-e4048c-5765848847-mg8pn\" (UID: \"8decf675-2bb4-431f-9ab6-da639afd4f55\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e4048c-5765848847-mg8pn" Oct 30 17:07:45.670644 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:45.667873 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95b7602a-63c6-4dd8-9f9f-198a1866ce43-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e4048c-d969759b4-c9knx\" (UID: \"95b7602a-63c6-4dd8-9f9f-198a1866ce43\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e4048c-d969759b4-c9knx" Oct 30 17:07:45.670644 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:45.667961 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8decf675-2bb4-431f-9ab6-da639afd4f55-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e4048c-5765848847-mg8pn\" (UID: \"8decf675-2bb4-431f-9ab6-da639afd4f55\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e4048c-5765848847-mg8pn" Oct 30 17:07:45.670644 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:45.669046 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8decf675-2bb4-431f-9ab6-da639afd4f55-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e4048c-5765848847-mg8pn\" (UID: \"8decf675-2bb4-431f-9ab6-da639afd4f55\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e4048c-5765848847-mg8pn" Oct 30 17:07:45.760608 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:45.758954 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95b7602a-63c6-4dd8-9f9f-198a1866ce43-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e4048c-d969759b4-c9knx\" (UID: \"95b7602a-63c6-4dd8-9f9f-198a1866ce43\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e4048c-d969759b4-c9knx" Oct 30 17:07:45.812944 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:45.811483 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vn4gt\" (UniqueName: \"kubernetes.io/projected/95b7602a-63c6-4dd8-9f9f-198a1866ce43-kube-api-access-vn4gt\") pod \"payload-processing-e2e-ipp-a-e4048c-d969759b4-c9knx\" (UID: \"95b7602a-63c6-4dd8-9f9f-198a1866ce43\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e4048c-d969759b4-c9knx" Oct 30 17:07:45.832566 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:45.815894 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-js6vs\" (UniqueName: \"kubernetes.io/projected/8decf675-2bb4-431f-9ab6-da639afd4f55-kube-api-access-js6vs\") pod \"payload-pre-processing-e2e-ipp-a-e4048c-5765848847-mg8pn\" (UID: \"8decf675-2bb4-431f-9ab6-da639afd4f55\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e4048c-5765848847-mg8pn" Oct 30 17:07:45.922567 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:45.922538 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-830ba9-5995465bff-gb57g" Oct 30 17:07:46.011362 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:46.011328 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f25133c-8cae-4395-807d-8b77508a9285-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f25133c-8cae-4395-807d-8b77508a9285-plugins-config-volume\") pod \"5f25133c-8cae-4395-807d-8b77508a9285\" (UID: \"5f25133c-8cae-4395-807d-8b77508a9285\") " Oct 30 17:07:46.011523 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:46.011423 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f25133c-8cae-4395-807d-8b77508a9285-kube-api-access-vnjrj\" (UniqueName: \"kubernetes.io/projected/5f25133c-8cae-4395-807d-8b77508a9285-kube-api-access-vnjrj\") pod \"5f25133c-8cae-4395-807d-8b77508a9285\" (UID: \"5f25133c-8cae-4395-807d-8b77508a9285\") " Oct 30 17:07:46.011735 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:46.011712 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f25133c-8cae-4395-807d-8b77508a9285-plugins-config-volume" pod "5f25133c-8cae-4395-807d-8b77508a9285" (UID: "5f25133c-8cae-4395-807d-8b77508a9285"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:07:46.012112 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:46.012086 5052 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f25133c-8cae-4395-807d-8b77508a9285-plugins-config-volume\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:07:46.013721 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:46.013695 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f25133c-8cae-4395-807d-8b77508a9285-kube-api-access-vnjrj" pod "5f25133c-8cae-4395-807d-8b77508a9285" (UID: "5f25133c-8cae-4395-807d-8b77508a9285"). InnerVolumeSpecName "kube-api-access-vnjrj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:07:46.070877 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:46.070844 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e4048c-d969759b4-c9knx" Oct 30 17:07:46.080095 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:46.080071 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e4048c-5765848847-mg8pn" Oct 30 17:07:46.114302 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:46.114264 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnjrj\" (UniqueName: \"kubernetes.io/projected/5f25133c-8cae-4395-807d-8b77508a9285-kube-api-access-vnjrj\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:07:46.281704 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:46.281662 5052 generic.go:356] "Generic (PLEG): container finished" podID="5f25133c-8cae-4395-807d-8b77508a9285" containerID="c988b975c2c4e83f36d5f5bcdd16c6fa65f8d932ba95d2ee3ab129807bec5d1f" exitCode=0 Oct 30 17:07:46.281886 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:46.281715 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-830ba9-5995465bff-gb57g" Oct 30 17:07:46.281886 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:46.281745 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-830ba9-5995465bff-gb57g" event={"ID":"5f25133c-8cae-4395-807d-8b77508a9285","Type":"ContainerDied","Data":"c988b975c2c4e83f36d5f5bcdd16c6fa65f8d932ba95d2ee3ab129807bec5d1f"} Oct 30 17:07:46.281886 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:46.281804 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-830ba9-5995465bff-gb57g" event={"ID":"5f25133c-8cae-4395-807d-8b77508a9285","Type":"ContainerDied","Data":"6d598f77ab9200db7fcf058ae00606bbad7cc5a52367fa3d7e47e28565e3dee7"} Oct 30 17:07:46.281886 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:46.281834 5052 scope.go:122] "RemoveContainer" containerID="c988b975c2c4e83f36d5f5bcdd16c6fa65f8d932ba95d2ee3ab129807bec5d1f" Oct 30 17:07:46.293287 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:46.293264 5052 scope.go:122] "RemoveContainer" containerID="c988b975c2c4e83f36d5f5bcdd16c6fa65f8d932ba95d2ee3ab129807bec5d1f" Oct 30 17:07:46.293531 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:07:46.293512 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c988b975c2c4e83f36d5f5bcdd16c6fa65f8d932ba95d2ee3ab129807bec5d1f\": container with ID starting with c988b975c2c4e83f36d5f5bcdd16c6fa65f8d932ba95d2ee3ab129807bec5d1f not found: ID does not exist" containerID="c988b975c2c4e83f36d5f5bcdd16c6fa65f8d932ba95d2ee3ab129807bec5d1f" Oct 30 17:07:46.293595 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:46.293539 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c988b975c2c4e83f36d5f5bcdd16c6fa65f8d932ba95d2ee3ab129807bec5d1f"} err="failed to get container status \"c988b975c2c4e83f36d5f5bcdd16c6fa65f8d932ba95d2ee3ab129807bec5d1f\": rpc error: code = NotFound desc = could not find container \"c988b975c2c4e83f36d5f5bcdd16c6fa65f8d932ba95d2ee3ab129807bec5d1f\": container with ID starting with c988b975c2c4e83f36d5f5bcdd16c6fa65f8d932ba95d2ee3ab129807bec5d1f not found: ID does not exist" Oct 30 17:07:46.338849 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:46.338817 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-830ba9-5995465bff-gb57g"] Oct 30 17:07:46.354398 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:46.354363 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-830ba9-5995465bff-gb57g"] Oct 30 17:07:46.383519 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:46.383487 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e4048c-d969759b4-c9knx"] Oct 30 17:07:46.384084 ip-10-0-131-136 kubenswrapper[5052]: W1030 17:07:46.384060 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95b7602a_63c6_4dd8_9f9f_198a1866ce43.slice/crio-e794a5c6350ca56177871cfa4712848ff4282359d2e598b38215cd4ac889a22e WatchSource:0}: Error finding container e794a5c6350ca56177871cfa4712848ff4282359d2e598b38215cd4ac889a22e: Status 404 returned error can't find the container with id e794a5c6350ca56177871cfa4712848ff4282359d2e598b38215cd4ac889a22e Oct 30 17:07:46.396770 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:46.396745 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e4048c-5765848847-mg8pn"] Oct 30 17:07:46.399382 ip-10-0-131-136 kubenswrapper[5052]: W1030 17:07:46.399358 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8decf675_2bb4_431f_9ab6_da639afd4f55.slice/crio-7280e380a795d765f2d349f87ce3b4621f9a1349fef47869b11c0dfeac9b5668 WatchSource:0}: Error finding container 7280e380a795d765f2d349f87ce3b4621f9a1349fef47869b11c0dfeac9b5668: Status 404 returned error can't find the container with id 7280e380a795d765f2d349f87ce3b4621f9a1349fef47869b11c0dfeac9b5668 Oct 30 17:07:46.599382 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:46.599349 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f25133c-8cae-4395-807d-8b77508a9285" path="/var/lib/kubelet/pods/5f25133c-8cae-4395-807d-8b77508a9285/volumes" Oct 30 17:07:47.293928 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:47.293882 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e4048c-5765848847-mg8pn" event={"ID":"8decf675-2bb4-431f-9ab6-da639afd4f55","Type":"ContainerStarted","Data":"4913bc65d1350df1ba0477c59d333376921e4ac9be591f82a22b50fec687a004"} Oct 30 17:07:47.294176 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:47.293971 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e4048c-5765848847-mg8pn" event={"ID":"8decf675-2bb4-431f-9ab6-da639afd4f55","Type":"ContainerStarted","Data":"7280e380a795d765f2d349f87ce3b4621f9a1349fef47869b11c0dfeac9b5668"} Oct 30 17:07:47.294312 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:47.294288 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e4048c-5765848847-mg8pn" Oct 30 17:07:47.297474 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:47.297443 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-e4048c-d969759b4-c9knx" Oct 30 17:07:47.297627 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:47.297519 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e4048c-d969759b4-c9knx" event={"ID":"95b7602a-63c6-4dd8-9f9f-198a1866ce43","Type":"ContainerStarted","Data":"28d5dccc444218f7a3652ee04d2dd340dcfac5e68069543fefc5ced6a4799d35"} Oct 30 17:07:47.297627 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:47.297557 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e4048c-d969759b4-c9knx" event={"ID":"95b7602a-63c6-4dd8-9f9f-198a1866ce43","Type":"ContainerStarted","Data":"e794a5c6350ca56177871cfa4712848ff4282359d2e598b38215cd4ac889a22e"} Oct 30 17:07:47.338758 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:47.338694 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e4048c-5765848847-mg8pn" podStartSLOduration=2.338676128 podStartE2EDuration="2.338676128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:07:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:07:47.33572233 +0000 UTC m=+2822.863057587" watchObservedRunningTime="2026-10-30 17:07:47.338676128 +0000 UTC m=+2822.866011347" Oct 30 17:07:47.378296 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:47.378242 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-e4048c-d969759b4-c9knx" podStartSLOduration=2.3782297310000002 podStartE2EDuration="2.378229731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:07:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:07:47.377972594 +0000 UTC m=+2822.905307840" watchObservedRunningTime="2026-10-30 17:07:47.378229731 +0000 UTC m=+2822.905564949" Oct 30 17:07:48.844111 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:07:48.844065 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 17:07:50.766998 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:50.766966 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-830ba9-546f46dd46-vm797"] Oct 30 17:07:50.767972 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:50.767915 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-830ba9-546f46dd46-vm797" podUID="37471512-e011-4938-9a3d-296d60601d0b" containerName="maas-api" containerID="cri-o://c7915883e8b1da982bbf64f21749e70602a5f50ad1c81e116bbb1b286e0af39e" gracePeriod=30 Oct 30 17:07:51.086827 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:51.086785 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-830ba9-546f46dd46-vm797" Oct 30 17:07:51.151762 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:51.151729 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37471512-e011-4938-9a3d-296d60601d0b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/37471512-e011-4938-9a3d-296d60601d0b-metrics-certs\") pod \"37471512-e011-4938-9a3d-296d60601d0b\" (UID: \"37471512-e011-4938-9a3d-296d60601d0b\") " Oct 30 17:07:51.151963 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:51.151805 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37471512-e011-4938-9a3d-296d60601d0b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37471512-e011-4938-9a3d-296d60601d0b-openshift-service-ca\") pod \"37471512-e011-4938-9a3d-296d60601d0b\" (UID: \"37471512-e011-4938-9a3d-296d60601d0b\") " Oct 30 17:07:51.151963 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:51.151951 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37471512-e011-4938-9a3d-296d60601d0b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37471512-e011-4938-9a3d-296d60601d0b-maas-api-tls\") pod \"37471512-e011-4938-9a3d-296d60601d0b\" (UID: \"37471512-e011-4938-9a3d-296d60601d0b\") " Oct 30 17:07:51.152118 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:51.152102 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37471512-e011-4938-9a3d-296d60601d0b-kube-api-access-mhvsw\" (UniqueName: \"kubernetes.io/projected/37471512-e011-4938-9a3d-296d60601d0b-kube-api-access-mhvsw\") pod \"37471512-e011-4938-9a3d-296d60601d0b\" (UID: \"37471512-e011-4938-9a3d-296d60601d0b\") " Oct 30 17:07:51.159106 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:51.158979 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37471512-e011-4938-9a3d-296d60601d0b-metrics-certs" pod "37471512-e011-4938-9a3d-296d60601d0b" (UID: "37471512-e011-4938-9a3d-296d60601d0b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:07:51.159106 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:51.159058 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37471512-e011-4938-9a3d-296d60601d0b-maas-api-tls" pod "37471512-e011-4938-9a3d-296d60601d0b" (UID: "37471512-e011-4938-9a3d-296d60601d0b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:07:51.159489 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:51.159143 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37471512-e011-4938-9a3d-296d60601d0b-kube-api-access-mhvsw" pod "37471512-e011-4938-9a3d-296d60601d0b" (UID: "37471512-e011-4938-9a3d-296d60601d0b"). InnerVolumeSpecName "kube-api-access-mhvsw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:07:51.174702 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:51.164494 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37471512-e011-4938-9a3d-296d60601d0b-openshift-service-ca" pod "37471512-e011-4938-9a3d-296d60601d0b" (UID: "37471512-e011-4938-9a3d-296d60601d0b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:07:51.281632 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:51.263926 5052 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37471512-e011-4938-9a3d-296d60601d0b-metrics-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:07:51.281632 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:51.263969 5052 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37471512-e011-4938-9a3d-296d60601d0b-openshift-service-ca\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:07:51.281632 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:51.263984 5052 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37471512-e011-4938-9a3d-296d60601d0b-maas-api-tls\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:07:51.281632 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:51.263999 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhvsw\" (UniqueName: \"kubernetes.io/projected/37471512-e011-4938-9a3d-296d60601d0b-kube-api-access-mhvsw\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:07:51.484620 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:51.484205 5052 generic.go:356] "Generic (PLEG): container finished" podID="37471512-e011-4938-9a3d-296d60601d0b" containerID="c7915883e8b1da982bbf64f21749e70602a5f50ad1c81e116bbb1b286e0af39e" exitCode=0 Oct 30 17:07:51.484620 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:51.484257 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-830ba9-546f46dd46-vm797" event={"ID":"37471512-e011-4938-9a3d-296d60601d0b","Type":"ContainerDied","Data":"c7915883e8b1da982bbf64f21749e70602a5f50ad1c81e116bbb1b286e0af39e"} Oct 30 17:07:51.484620 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:51.484344 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-830ba9-546f46dd46-vm797" event={"ID":"37471512-e011-4938-9a3d-296d60601d0b","Type":"ContainerDied","Data":"d07b90e64fc9cfb09cff005b40f5564b05d5d4066c8439fc1b02747adbabd92d"} Oct 30 17:07:51.484620 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:51.484377 5052 scope.go:122] "RemoveContainer" containerID="c7915883e8b1da982bbf64f21749e70602a5f50ad1c81e116bbb1b286e0af39e" Oct 30 17:07:51.484620 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:51.484556 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-830ba9-546f46dd46-vm797" Oct 30 17:07:51.565151 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:51.563537 5052 scope.go:122] "RemoveContainer" containerID="c7915883e8b1da982bbf64f21749e70602a5f50ad1c81e116bbb1b286e0af39e" Oct 30 17:07:51.566103 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:07:51.565935 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7915883e8b1da982bbf64f21749e70602a5f50ad1c81e116bbb1b286e0af39e\": container with ID starting with c7915883e8b1da982bbf64f21749e70602a5f50ad1c81e116bbb1b286e0af39e not found: ID does not exist" containerID="c7915883e8b1da982bbf64f21749e70602a5f50ad1c81e116bbb1b286e0af39e" Oct 30 17:07:51.566103 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:51.565975 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7915883e8b1da982bbf64f21749e70602a5f50ad1c81e116bbb1b286e0af39e"} err="failed to get container status \"c7915883e8b1da982bbf64f21749e70602a5f50ad1c81e116bbb1b286e0af39e\": rpc error: code = NotFound desc = could not find container \"c7915883e8b1da982bbf64f21749e70602a5f50ad1c81e116bbb1b286e0af39e\": container with ID starting with c7915883e8b1da982bbf64f21749e70602a5f50ad1c81e116bbb1b286e0af39e not found: ID does not exist" Oct 30 17:07:51.584884 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:51.584808 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-830ba9-546f46dd46-vm797"] Oct 30 17:07:51.594906 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:51.594860 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-830ba9-546f46dd46-vm797"] Oct 30 17:07:52.650469 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:52.650428 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37471512-e011-4938-9a3d-296d60601d0b" path="/var/lib/kubelet/pods/37471512-e011-4938-9a3d-296d60601d0b/volumes" Oct 30 17:07:55.912533 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:55.912495 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-4ac125-openshift-default-6569d9988c-4zscj"] Oct 30 17:07:55.913685 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:55.913665 5052 cpu_manager.go:414] "containerMap: removing container" podUID="5f25133c-8cae-4395-807d-8b77508a9285" containerName="payload-pre-processing" Oct 30 17:07:55.913761 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:55.913687 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f25133c-8cae-4395-807d-8b77508a9285" containerName="payload-pre-processing" Oct 30 17:07:55.913761 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:55.913724 5052 cpu_manager.go:414] "containerMap: removing container" podUID="37471512-e011-4938-9a3d-296d60601d0b" containerName="maas-api" Oct 30 17:07:55.913761 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:55.913732 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37471512-e011-4938-9a3d-296d60601d0b" containerName="maas-api" Oct 30 17:07:55.913906 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:55.913894 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f25133c-8cae-4395-807d-8b77508a9285" containerName="payload-pre-processing" Oct 30 17:07:55.913954 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:55.913912 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="37471512-e011-4938-9a3d-296d60601d0b" containerName="maas-api" Oct 30 17:07:55.919671 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:55.919648 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-4ac125-openshift-default-6569d9988c-4zscj"] Oct 30 17:07:55.919803 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:55.919733 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-4ac125-openshift-default-6569d9988c-4zscj" Oct 30 17:07:55.956871 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:55.937888 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-4ac125-openshift-default-dockercfg-2jf42\"" Oct 30 17:07:56.118045 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:56.118000 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4be3cd3f-d7d6-40ec-ad72-8012cfbdb946-istio-podinfo\") pod \"e2e-ipp-a-4ac125-openshift-default-6569d9988c-4zscj\" (UID: \"4be3cd3f-d7d6-40ec-ad72-8012cfbdb946\") " pod="openshift-ingress/e2e-ipp-a-4ac125-openshift-default-6569d9988c-4zscj" Oct 30 17:07:56.118267 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:56.118120 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4be3cd3f-d7d6-40ec-ad72-8012cfbdb946-credential-socket\") pod \"e2e-ipp-a-4ac125-openshift-default-6569d9988c-4zscj\" (UID: \"4be3cd3f-d7d6-40ec-ad72-8012cfbdb946\") " pod="openshift-ingress/e2e-ipp-a-4ac125-openshift-default-6569d9988c-4zscj" Oct 30 17:07:56.118267 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:56.118174 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4be3cd3f-d7d6-40ec-ad72-8012cfbdb946-istio-token\") pod \"e2e-ipp-a-4ac125-openshift-default-6569d9988c-4zscj\" (UID: \"4be3cd3f-d7d6-40ec-ad72-8012cfbdb946\") " pod="openshift-ingress/e2e-ipp-a-4ac125-openshift-default-6569d9988c-4zscj" Oct 30 17:07:56.118403 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:56.118291 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4be3cd3f-d7d6-40ec-ad72-8012cfbdb946-istiod-ca-cert\") pod \"e2e-ipp-a-4ac125-openshift-default-6569d9988c-4zscj\" (UID: \"4be3cd3f-d7d6-40ec-ad72-8012cfbdb946\") " pod="openshift-ingress/e2e-ipp-a-4ac125-openshift-default-6569d9988c-4zscj" Oct 30 17:07:56.118403 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:56.118359 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4be3cd3f-d7d6-40ec-ad72-8012cfbdb946-istio-data\") pod \"e2e-ipp-a-4ac125-openshift-default-6569d9988c-4zscj\" (UID: \"4be3cd3f-d7d6-40ec-ad72-8012cfbdb946\") " pod="openshift-ingress/e2e-ipp-a-4ac125-openshift-default-6569d9988c-4zscj" Oct 30 17:07:56.118518 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:56.118493 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4be3cd3f-d7d6-40ec-ad72-8012cfbdb946-workload-socket\") pod \"e2e-ipp-a-4ac125-openshift-default-6569d9988c-4zscj\" (UID: \"4be3cd3f-d7d6-40ec-ad72-8012cfbdb946\") " pod="openshift-ingress/e2e-ipp-a-4ac125-openshift-default-6569d9988c-4zscj" Oct 30 17:07:56.118639 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:56.118626 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4be3cd3f-d7d6-40ec-ad72-8012cfbdb946-workload-certs\") pod \"e2e-ipp-a-4ac125-openshift-default-6569d9988c-4zscj\" (UID: \"4be3cd3f-d7d6-40ec-ad72-8012cfbdb946\") " pod="openshift-ingress/e2e-ipp-a-4ac125-openshift-default-6569d9988c-4zscj" Oct 30 17:07:56.118703 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:56.118646 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4be3cd3f-d7d6-40ec-ad72-8012cfbdb946-istio-envoy\") pod \"e2e-ipp-a-4ac125-openshift-default-6569d9988c-4zscj\" (UID: \"4be3cd3f-d7d6-40ec-ad72-8012cfbdb946\") " pod="openshift-ingress/e2e-ipp-a-4ac125-openshift-default-6569d9988c-4zscj" Oct 30 17:07:56.118703 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:56.118661 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzdww\" (UniqueName: \"kubernetes.io/projected/4be3cd3f-d7d6-40ec-ad72-8012cfbdb946-kube-api-access-jzdww\") pod \"e2e-ipp-a-4ac125-openshift-default-6569d9988c-4zscj\" (UID: \"4be3cd3f-d7d6-40ec-ad72-8012cfbdb946\") " pod="openshift-ingress/e2e-ipp-a-4ac125-openshift-default-6569d9988c-4zscj" Oct 30 17:07:56.219825 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:56.219733 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4be3cd3f-d7d6-40ec-ad72-8012cfbdb946-workload-certs\") pod \"e2e-ipp-a-4ac125-openshift-default-6569d9988c-4zscj\" (UID: \"4be3cd3f-d7d6-40ec-ad72-8012cfbdb946\") " pod="openshift-ingress/e2e-ipp-a-4ac125-openshift-default-6569d9988c-4zscj" Oct 30 17:07:56.219825 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:56.219778 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4be3cd3f-d7d6-40ec-ad72-8012cfbdb946-istio-envoy\") pod \"e2e-ipp-a-4ac125-openshift-default-6569d9988c-4zscj\" (UID: \"4be3cd3f-d7d6-40ec-ad72-8012cfbdb946\") " pod="openshift-ingress/e2e-ipp-a-4ac125-openshift-default-6569d9988c-4zscj" Oct 30 17:07:56.219825 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:56.219795 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzdww\" (UniqueName: \"kubernetes.io/projected/4be3cd3f-d7d6-40ec-ad72-8012cfbdb946-kube-api-access-jzdww\") pod \"e2e-ipp-a-4ac125-openshift-default-6569d9988c-4zscj\" (UID: \"4be3cd3f-d7d6-40ec-ad72-8012cfbdb946\") " pod="openshift-ingress/e2e-ipp-a-4ac125-openshift-default-6569d9988c-4zscj" Oct 30 17:07:56.220128 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:56.219842 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4be3cd3f-d7d6-40ec-ad72-8012cfbdb946-istio-podinfo\") pod \"e2e-ipp-a-4ac125-openshift-default-6569d9988c-4zscj\" (UID: \"4be3cd3f-d7d6-40ec-ad72-8012cfbdb946\") " pod="openshift-ingress/e2e-ipp-a-4ac125-openshift-default-6569d9988c-4zscj" Oct 30 17:07:56.220128 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:56.219983 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4be3cd3f-d7d6-40ec-ad72-8012cfbdb946-credential-socket\") pod \"e2e-ipp-a-4ac125-openshift-default-6569d9988c-4zscj\" (UID: \"4be3cd3f-d7d6-40ec-ad72-8012cfbdb946\") " pod="openshift-ingress/e2e-ipp-a-4ac125-openshift-default-6569d9988c-4zscj" Oct 30 17:07:56.220128 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:56.220028 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4be3cd3f-d7d6-40ec-ad72-8012cfbdb946-istio-token\") pod \"e2e-ipp-a-4ac125-openshift-default-6569d9988c-4zscj\" (UID: \"4be3cd3f-d7d6-40ec-ad72-8012cfbdb946\") " pod="openshift-ingress/e2e-ipp-a-4ac125-openshift-default-6569d9988c-4zscj" Oct 30 17:07:56.220128 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:56.220120 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4be3cd3f-d7d6-40ec-ad72-8012cfbdb946-workload-certs\") pod \"e2e-ipp-a-4ac125-openshift-default-6569d9988c-4zscj\" (UID: \"4be3cd3f-d7d6-40ec-ad72-8012cfbdb946\") " pod="openshift-ingress/e2e-ipp-a-4ac125-openshift-default-6569d9988c-4zscj" Oct 30 17:07:56.220366 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:56.220132 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4be3cd3f-d7d6-40ec-ad72-8012cfbdb946-istiod-ca-cert\") pod \"e2e-ipp-a-4ac125-openshift-default-6569d9988c-4zscj\" (UID: \"4be3cd3f-d7d6-40ec-ad72-8012cfbdb946\") " pod="openshift-ingress/e2e-ipp-a-4ac125-openshift-default-6569d9988c-4zscj" Oct 30 17:07:56.220366 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:56.220219 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4be3cd3f-d7d6-40ec-ad72-8012cfbdb946-istio-data\") pod \"e2e-ipp-a-4ac125-openshift-default-6569d9988c-4zscj\" (UID: \"4be3cd3f-d7d6-40ec-ad72-8012cfbdb946\") " pod="openshift-ingress/e2e-ipp-a-4ac125-openshift-default-6569d9988c-4zscj" Oct 30 17:07:56.220366 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:56.220318 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4be3cd3f-d7d6-40ec-ad72-8012cfbdb946-credential-socket\") pod \"e2e-ipp-a-4ac125-openshift-default-6569d9988c-4zscj\" (UID: \"4be3cd3f-d7d6-40ec-ad72-8012cfbdb946\") " pod="openshift-ingress/e2e-ipp-a-4ac125-openshift-default-6569d9988c-4zscj" Oct 30 17:07:56.220366 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:56.220357 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4be3cd3f-d7d6-40ec-ad72-8012cfbdb946-workload-socket\") pod \"e2e-ipp-a-4ac125-openshift-default-6569d9988c-4zscj\" (UID: \"4be3cd3f-d7d6-40ec-ad72-8012cfbdb946\") " pod="openshift-ingress/e2e-ipp-a-4ac125-openshift-default-6569d9988c-4zscj" Oct 30 17:07:56.220625 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:56.220592 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4be3cd3f-d7d6-40ec-ad72-8012cfbdb946-istio-data\") pod \"e2e-ipp-a-4ac125-openshift-default-6569d9988c-4zscj\" (UID: \"4be3cd3f-d7d6-40ec-ad72-8012cfbdb946\") " pod="openshift-ingress/e2e-ipp-a-4ac125-openshift-default-6569d9988c-4zscj" Oct 30 17:07:56.220708 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:56.220644 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4be3cd3f-d7d6-40ec-ad72-8012cfbdb946-workload-socket\") pod \"e2e-ipp-a-4ac125-openshift-default-6569d9988c-4zscj\" (UID: \"4be3cd3f-d7d6-40ec-ad72-8012cfbdb946\") " pod="openshift-ingress/e2e-ipp-a-4ac125-openshift-default-6569d9988c-4zscj" Oct 30 17:07:56.220817 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:56.220798 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4be3cd3f-d7d6-40ec-ad72-8012cfbdb946-istiod-ca-cert\") pod \"e2e-ipp-a-4ac125-openshift-default-6569d9988c-4zscj\" (UID: \"4be3cd3f-d7d6-40ec-ad72-8012cfbdb946\") " pod="openshift-ingress/e2e-ipp-a-4ac125-openshift-default-6569d9988c-4zscj" Oct 30 17:07:56.222300 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:56.222263 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4be3cd3f-d7d6-40ec-ad72-8012cfbdb946-istio-envoy\") pod \"e2e-ipp-a-4ac125-openshift-default-6569d9988c-4zscj\" (UID: \"4be3cd3f-d7d6-40ec-ad72-8012cfbdb946\") " pod="openshift-ingress/e2e-ipp-a-4ac125-openshift-default-6569d9988c-4zscj" Oct 30 17:07:56.222602 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:56.222570 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4be3cd3f-d7d6-40ec-ad72-8012cfbdb946-istio-podinfo\") pod \"e2e-ipp-a-4ac125-openshift-default-6569d9988c-4zscj\" (UID: \"4be3cd3f-d7d6-40ec-ad72-8012cfbdb946\") " pod="openshift-ingress/e2e-ipp-a-4ac125-openshift-default-6569d9988c-4zscj" Oct 30 17:07:56.280603 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:56.273792 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzdww\" (UniqueName: \"kubernetes.io/projected/4be3cd3f-d7d6-40ec-ad72-8012cfbdb946-kube-api-access-jzdww\") pod \"e2e-ipp-a-4ac125-openshift-default-6569d9988c-4zscj\" (UID: \"4be3cd3f-d7d6-40ec-ad72-8012cfbdb946\") " pod="openshift-ingress/e2e-ipp-a-4ac125-openshift-default-6569d9988c-4zscj" Oct 30 17:07:56.306726 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:56.301279 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4be3cd3f-d7d6-40ec-ad72-8012cfbdb946-istio-token\") pod \"e2e-ipp-a-4ac125-openshift-default-6569d9988c-4zscj\" (UID: \"4be3cd3f-d7d6-40ec-ad72-8012cfbdb946\") " pod="openshift-ingress/e2e-ipp-a-4ac125-openshift-default-6569d9988c-4zscj" Oct 30 17:07:56.557083 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:56.556979 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-4ac125-openshift-default-6569d9988c-4zscj" Oct 30 17:07:56.890946 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:56.890914 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-4ac125-openshift-default-6569d9988c-4zscj"] Oct 30 17:07:56.891505 ip-10-0-131-136 kubenswrapper[5052]: W1030 17:07:56.891477 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4be3cd3f_d7d6_40ec_ad72_8012cfbdb946.slice/crio-12a9f5d398806e9ec2ad61c84597181b7b4d04605d069ff2484c90dfb0643cdb WatchSource:0}: Error finding container 12a9f5d398806e9ec2ad61c84597181b7b4d04605d069ff2484c90dfb0643cdb: Status 404 returned error can't find the container with id 12a9f5d398806e9ec2ad61c84597181b7b4d04605d069ff2484c90dfb0643cdb Oct 30 17:07:56.894462 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:56.894430 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:07:56.894559 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:56.894496 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:07:56.894559 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:56.894525 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:07:56.917542 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:56.917511 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4ac125-openshift-default-6569d9988c-4zscj" event={"ID":"4be3cd3f-d7d6-40ec-ad72-8012cfbdb946","Type":"ContainerStarted","Data":"12a9f5d398806e9ec2ad61c84597181b7b4d04605d069ff2484c90dfb0643cdb"} Oct 30 17:07:57.926944 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:57.926903 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4ac125-openshift-default-6569d9988c-4zscj" event={"ID":"4be3cd3f-d7d6-40ec-ad72-8012cfbdb946","Type":"ContainerStarted","Data":"81768e865fda8052bd1932d84fde94c0d357a5da7e2b3420fecccefb935956fc"} Oct 30 17:07:57.990075 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:57.990024 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-4ac125-openshift-default-6569d9988c-4zscj" podStartSLOduration=2.990011144 podStartE2EDuration="2.990011144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:07:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:07:57.989442275 +0000 UTC m=+2833.516777495" watchObservedRunningTime="2026-10-30 17:07:57.990011144 +0000 UTC m=+2833.517346362" Oct 30 17:07:58.308346 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:58.308316 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e4048c-5765848847-mg8pn" Oct 30 17:07:58.572387 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:58.569931 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-4ac125-openshift-default-6569d9988c-4zscj" Oct 30 17:07:58.628660 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:58.628371 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-4ac125-openshift-default-6569d9988c-4zscj" Oct 30 17:07:59.057606 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:59.019699 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-4ac125-openshift-default-6569d9988c-4zscj" Oct 30 17:07:59.057606 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:59.039893 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-4ac125-openshift-default-6569d9988c-4zscj" Oct 30 17:07:59.314958 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:07:59.314877 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-e4048c-d969759b4-c9knx" Oct 30 17:08:02.226529 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:02.226489 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4ac125-9dcf56996-v4vjr"] Oct 30 17:08:02.248909 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:02.248875 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4ac125-9dcf56996-v4vjr"] Oct 30 17:08:02.249099 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:02.248976 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4ac125-9dcf56996-v4vjr" Oct 30 17:08:02.278622 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:02.278571 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-4ac125-dockercfg-bwtw8\"" Oct 30 17:08:02.291854 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:02.291803 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-4ac125\"" Oct 30 17:08:02.303696 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:02.303662 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4ac125-84b8c9c875-7gtbm"] Oct 30 17:08:02.339122 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:02.339076 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4ac125-84b8c9c875-7gtbm"] Oct 30 17:08:02.341633 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:02.341607 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ac125-84b8c9c875-7gtbm" Oct 30 17:08:02.431477 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:02.429792 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxbdd\" (UniqueName: \"kubernetes.io/projected/d0efd4aa-b83a-48d8-868e-be7d351899e2-kube-api-access-cxbdd\") pod \"payload-processing-e2e-ipp-a-4ac125-9dcf56996-v4vjr\" (UID: \"d0efd4aa-b83a-48d8-868e-be7d351899e2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4ac125-9dcf56996-v4vjr" Oct 30 17:08:02.431477 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:02.429883 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0efd4aa-b83a-48d8-868e-be7d351899e2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-4ac125-9dcf56996-v4vjr\" (UID: \"d0efd4aa-b83a-48d8-868e-be7d351899e2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4ac125-9dcf56996-v4vjr" Oct 30 17:08:02.431477 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:02.429927 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/565bcd81-6055-4578-ae1a-628829ed7163-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-4ac125-84b8c9c875-7gtbm\" (UID: \"565bcd81-6055-4578-ae1a-628829ed7163\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ac125-84b8c9c875-7gtbm" Oct 30 17:08:02.431477 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:02.430062 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqd8x\" (UniqueName: \"kubernetes.io/projected/565bcd81-6055-4578-ae1a-628829ed7163-kube-api-access-pqd8x\") pod \"payload-pre-processing-e2e-ipp-a-4ac125-84b8c9c875-7gtbm\" (UID: \"565bcd81-6055-4578-ae1a-628829ed7163\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ac125-84b8c9c875-7gtbm" Oct 30 17:08:02.596741 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:02.594005 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxbdd\" (UniqueName: \"kubernetes.io/projected/d0efd4aa-b83a-48d8-868e-be7d351899e2-kube-api-access-cxbdd\") pod \"payload-processing-e2e-ipp-a-4ac125-9dcf56996-v4vjr\" (UID: \"d0efd4aa-b83a-48d8-868e-be7d351899e2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4ac125-9dcf56996-v4vjr" Oct 30 17:08:02.596741 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:02.594113 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0efd4aa-b83a-48d8-868e-be7d351899e2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-4ac125-9dcf56996-v4vjr\" (UID: \"d0efd4aa-b83a-48d8-868e-be7d351899e2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4ac125-9dcf56996-v4vjr" Oct 30 17:08:02.596741 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:02.594165 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/565bcd81-6055-4578-ae1a-628829ed7163-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-4ac125-84b8c9c875-7gtbm\" (UID: \"565bcd81-6055-4578-ae1a-628829ed7163\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ac125-84b8c9c875-7gtbm" Oct 30 17:08:02.596741 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:02.594321 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqd8x\" (UniqueName: \"kubernetes.io/projected/565bcd81-6055-4578-ae1a-628829ed7163-kube-api-access-pqd8x\") pod \"payload-pre-processing-e2e-ipp-a-4ac125-84b8c9c875-7gtbm\" (UID: \"565bcd81-6055-4578-ae1a-628829ed7163\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ac125-84b8c9c875-7gtbm" Oct 30 17:08:02.596741 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:02.595801 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0efd4aa-b83a-48d8-868e-be7d351899e2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-4ac125-9dcf56996-v4vjr\" (UID: \"d0efd4aa-b83a-48d8-868e-be7d351899e2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4ac125-9dcf56996-v4vjr" Oct 30 17:08:02.597425 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:02.596873 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/565bcd81-6055-4578-ae1a-628829ed7163-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-4ac125-84b8c9c875-7gtbm\" (UID: \"565bcd81-6055-4578-ae1a-628829ed7163\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ac125-84b8c9c875-7gtbm" Oct 30 17:08:02.640974 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:02.640909 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxbdd\" (UniqueName: \"kubernetes.io/projected/d0efd4aa-b83a-48d8-868e-be7d351899e2-kube-api-access-cxbdd\") pod \"payload-processing-e2e-ipp-a-4ac125-9dcf56996-v4vjr\" (UID: \"d0efd4aa-b83a-48d8-868e-be7d351899e2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4ac125-9dcf56996-v4vjr" Oct 30 17:08:02.645052 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:02.645019 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqd8x\" (UniqueName: \"kubernetes.io/projected/565bcd81-6055-4578-ae1a-628829ed7163-kube-api-access-pqd8x\") pod \"payload-pre-processing-e2e-ipp-a-4ac125-84b8c9c875-7gtbm\" (UID: \"565bcd81-6055-4578-ae1a-628829ed7163\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ac125-84b8c9c875-7gtbm" Oct 30 17:08:02.653950 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:02.653893 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ac125-84b8c9c875-7gtbm" Oct 30 17:08:02.884824 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:02.884728 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4ac125-9dcf56996-v4vjr" Oct 30 17:08:02.988059 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:02.988027 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4ac125-84b8c9c875-7gtbm"] Oct 30 17:08:02.995175 ip-10-0-131-136 kubenswrapper[5052]: W1030 17:08:02.995145 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod565bcd81_6055_4578_ae1a_628829ed7163.slice/crio-07bc9cfabc08e97a7655bd241a7b7b8b786efb42fc6c20d6e6d1fdb4c9bf06a4 WatchSource:0}: Error finding container 07bc9cfabc08e97a7655bd241a7b7b8b786efb42fc6c20d6e6d1fdb4c9bf06a4: Status 404 returned error can't find the container with id 07bc9cfabc08e97a7655bd241a7b7b8b786efb42fc6c20d6e6d1fdb4c9bf06a4 Oct 30 17:08:03.057226 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:03.057176 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ac125-84b8c9c875-7gtbm" event={"ID":"565bcd81-6055-4578-ae1a-628829ed7163","Type":"ContainerStarted","Data":"07bc9cfabc08e97a7655bd241a7b7b8b786efb42fc6c20d6e6d1fdb4c9bf06a4"} Oct 30 17:08:03.240259 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:03.240230 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4ac125-9dcf56996-v4vjr"] Oct 30 17:08:04.066334 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:04.066299 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ac125-84b8c9c875-7gtbm" event={"ID":"565bcd81-6055-4578-ae1a-628829ed7163","Type":"ContainerStarted","Data":"24edfa1ba57b5fa857c286285a96152769e94bba56816ccd75f7974cecf01420"} Oct 30 17:08:04.066670 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:04.066558 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ac125-84b8c9c875-7gtbm" Oct 30 17:08:04.068331 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:04.068300 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4ac125-9dcf56996-v4vjr" event={"ID":"d0efd4aa-b83a-48d8-868e-be7d351899e2","Type":"ContainerStarted","Data":"1ebf48a099ae97701dcbd8b9380c62a74822c1d0bfd47804d55764f4ebfd9abd"} Oct 30 17:08:04.068452 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:04.068353 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4ac125-9dcf56996-v4vjr" event={"ID":"d0efd4aa-b83a-48d8-868e-be7d351899e2","Type":"ContainerStarted","Data":"6aa69ba44ab2a49249ce945980f95ea261c2b41241ea8a1d5b48d45b77179a95"} Oct 30 17:08:04.068452 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:04.068381 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-4ac125-9dcf56996-v4vjr" Oct 30 17:08:04.113243 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:04.113193 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ac125-84b8c9c875-7gtbm" podStartSLOduration=2.113180436 podStartE2EDuration="2.113180436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:08:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:08:04.108668876 +0000 UTC m=+2839.636004116" watchObservedRunningTime="2026-10-30 17:08:04.113180436 +0000 UTC m=+2839.640515679" Oct 30 17:08:04.153901 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:04.153845 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-4ac125-9dcf56996-v4vjr" podStartSLOduration=2.153831717 podStartE2EDuration="2.153831717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:08:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:08:04.150287644 +0000 UTC m=+2839.677622865" watchObservedRunningTime="2026-10-30 17:08:04.153831717 +0000 UTC m=+2839.681166935" Oct 30 17:08:04.914488 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:04.914451 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-330282-b5d7c8c9f-6xbhn"] Oct 30 17:08:08.200346 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:08.200251 5052 generic.go:356] "Generic (PLEG): container finished" podID="ce11f38f-0ae0-4881-aa0f-346a0dd513c4" containerID="2e348d887319104ed3f97de52fb4effecebdb81eab193bcd2a22ac7b0b96a325" exitCode=0 Oct 30 17:08:08.200346 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:08.200322 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-330282-openshift-default-5784687855-s68x6" event={"ID":"ce11f38f-0ae0-4881-aa0f-346a0dd513c4","Type":"ContainerDied","Data":"2e348d887319104ed3f97de52fb4effecebdb81eab193bcd2a22ac7b0b96a325"} Oct 30 17:08:08.200939 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:08.200789 5052 scope.go:122] "RemoveContainer" containerID="2e348d887319104ed3f97de52fb4effecebdb81eab193bcd2a22ac7b0b96a325" Oct 30 17:08:08.202780 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:08.202751 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:08:08.202869 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:08.202840 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:08:08.202917 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:08.202877 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:08:09.210216 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:09.210177 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-330282-openshift-default-5784687855-s68x6" event={"ID":"ce11f38f-0ae0-4881-aa0f-346a0dd513c4","Type":"ContainerStarted","Data":"b09cf2b2468112f122ac51f5ee4d6dc72dbe40920fc903d4c8c2a3b30abb5c8b"} Oct 30 17:08:09.412277 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:09.412240 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-330282-openshift-default-5784687855-s68x6" Oct 30 17:08:09.413385 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:09.413360 5052 patch_prober.go:29] interesting pod/e2e-ipp-a-330282-openshift-default-5784687855-s68x6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body= Oct 30 17:08:09.413524 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:09.413410 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-330282-openshift-default-5784687855-s68x6" podUID="ce11f38f-0ae0-4881-aa0f-346a0dd513c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" Oct 30 17:08:09.799034 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:09.798997 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-e5bfb0-6f84945dc8-s9w5w"] Oct 30 17:08:09.854736 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:09.854700 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-e5bfb0-86457d6567-v7zt5"] Oct 30 17:08:10.100652 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:10.081572 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-330282-b5d7c8c9f-6xbhn" podUID="d5691fdc-cd71-42ca-9eb7-98d431c101cc" containerName="payload-pre-processing" containerID="cri-o://27d0c0e2a8d4e3d65db6ffa8d644dcc22862934883dfdbafa5e650fa20b67696" gracePeriod=25 Oct 30 17:08:10.432605 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:10.432036 5052 patch_prober.go:29] interesting pod/e2e-ipp-a-330282-openshift-default-5784687855-s68x6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body= Oct 30 17:08:10.432605 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:10.432102 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-330282-openshift-default-5784687855-s68x6" podUID="ce11f38f-0ae0-4881-aa0f-346a0dd513c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" Oct 30 17:08:10.694573 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:10.694519 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-330282-b5d7c8c9f-6xbhn" Oct 30 17:08:10.860415 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:10.860374 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5691fdc-cd71-42ca-9eb7-98d431c101cc-kube-api-access-v8vnt\" (UniqueName: \"kubernetes.io/projected/d5691fdc-cd71-42ca-9eb7-98d431c101cc-kube-api-access-v8vnt\") pod \"d5691fdc-cd71-42ca-9eb7-98d431c101cc\" (UID: \"d5691fdc-cd71-42ca-9eb7-98d431c101cc\") " Oct 30 17:08:10.860684 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:10.860668 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5691fdc-cd71-42ca-9eb7-98d431c101cc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5691fdc-cd71-42ca-9eb7-98d431c101cc-plugins-config-volume\") pod \"d5691fdc-cd71-42ca-9eb7-98d431c101cc\" (UID: \"d5691fdc-cd71-42ca-9eb7-98d431c101cc\") " Oct 30 17:08:10.861062 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:10.861033 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5691fdc-cd71-42ca-9eb7-98d431c101cc-plugins-config-volume" pod "d5691fdc-cd71-42ca-9eb7-98d431c101cc" (UID: "d5691fdc-cd71-42ca-9eb7-98d431c101cc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:08:10.861231 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:10.861214 5052 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5691fdc-cd71-42ca-9eb7-98d431c101cc-plugins-config-volume\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:08:10.862843 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:10.862814 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5691fdc-cd71-42ca-9eb7-98d431c101cc-kube-api-access-v8vnt" pod "d5691fdc-cd71-42ca-9eb7-98d431c101cc" (UID: "d5691fdc-cd71-42ca-9eb7-98d431c101cc"). InnerVolumeSpecName "kube-api-access-v8vnt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:08:10.978684 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:10.973952 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8vnt\" (UniqueName: \"kubernetes.io/projected/d5691fdc-cd71-42ca-9eb7-98d431c101cc-kube-api-access-v8vnt\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:08:11.246661 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:11.232236 5052 generic.go:356] "Generic (PLEG): container finished" podID="d5691fdc-cd71-42ca-9eb7-98d431c101cc" containerID="27d0c0e2a8d4e3d65db6ffa8d644dcc22862934883dfdbafa5e650fa20b67696" exitCode=0 Oct 30 17:08:11.246661 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:11.232512 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-330282-b5d7c8c9f-6xbhn" event={"ID":"d5691fdc-cd71-42ca-9eb7-98d431c101cc","Type":"ContainerDied","Data":"27d0c0e2a8d4e3d65db6ffa8d644dcc22862934883dfdbafa5e650fa20b67696"} Oct 30 17:08:11.246661 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:11.232621 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-330282-b5d7c8c9f-6xbhn" event={"ID":"d5691fdc-cd71-42ca-9eb7-98d431c101cc","Type":"ContainerDied","Data":"13b6680add58f288f7c50d1831f11e516b85c46067fe024ccc8fe57c163985ed"} Oct 30 17:08:11.246661 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:11.232680 5052 scope.go:122] "RemoveContainer" containerID="27d0c0e2a8d4e3d65db6ffa8d644dcc22862934883dfdbafa5e650fa20b67696" Oct 30 17:08:11.246661 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:11.232858 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-330282-b5d7c8c9f-6xbhn" Oct 30 17:08:11.267137 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:11.267115 5052 scope.go:122] "RemoveContainer" containerID="27d0c0e2a8d4e3d65db6ffa8d644dcc22862934883dfdbafa5e650fa20b67696" Oct 30 17:08:11.267436 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:08:11.267413 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27d0c0e2a8d4e3d65db6ffa8d644dcc22862934883dfdbafa5e650fa20b67696\": container with ID starting with 27d0c0e2a8d4e3d65db6ffa8d644dcc22862934883dfdbafa5e650fa20b67696 not found: ID does not exist" containerID="27d0c0e2a8d4e3d65db6ffa8d644dcc22862934883dfdbafa5e650fa20b67696" Oct 30 17:08:11.267510 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:11.267449 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27d0c0e2a8d4e3d65db6ffa8d644dcc22862934883dfdbafa5e650fa20b67696"} err="failed to get container status \"27d0c0e2a8d4e3d65db6ffa8d644dcc22862934883dfdbafa5e650fa20b67696\": rpc error: code = NotFound desc = could not find container \"27d0c0e2a8d4e3d65db6ffa8d644dcc22862934883dfdbafa5e650fa20b67696\": container with ID starting with 27d0c0e2a8d4e3d65db6ffa8d644dcc22862934883dfdbafa5e650fa20b67696 not found: ID does not exist" Oct 30 17:08:11.316383 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:11.316353 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-330282-b5d7c8c9f-6xbhn"] Oct 30 17:08:11.320057 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:11.320036 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-330282-b5d7c8c9f-6xbhn"] Oct 30 17:08:11.412810 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:11.412777 5052 patch_prober.go:29] interesting pod/e2e-ipp-a-330282-openshift-default-5784687855-s68x6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body= Oct 30 17:08:11.412991 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:11.412832 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-330282-openshift-default-5784687855-s68x6" podUID="ce11f38f-0ae0-4881-aa0f-346a0dd513c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" Oct 30 17:08:11.419865 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:11.419845 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-330282-openshift-default-5784687855-s68x6" Oct 30 17:08:12.412501 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:12.412468 5052 patch_prober.go:29] interesting pod/e2e-ipp-a-330282-openshift-default-5784687855-s68x6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body= Oct 30 17:08:12.412934 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:12.412536 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-330282-openshift-default-5784687855-s68x6" podUID="ce11f38f-0ae0-4881-aa0f-346a0dd513c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" Oct 30 17:08:12.600124 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:12.600089 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5691fdc-cd71-42ca-9eb7-98d431c101cc" path="/var/lib/kubelet/pods/d5691fdc-cd71-42ca-9eb7-98d431c101cc/volumes" Oct 30 17:08:13.412570 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:13.412538 5052 patch_prober.go:29] interesting pod/e2e-ipp-a-330282-openshift-default-5784687855-s68x6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body= Oct 30 17:08:13.412998 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:13.412613 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-330282-openshift-default-5784687855-s68x6" podUID="ce11f38f-0ae0-4881-aa0f-346a0dd513c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" Oct 30 17:08:14.413138 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:14.413096 5052 patch_prober.go:29] interesting pod/e2e-ipp-a-330282-openshift-default-5784687855-s68x6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body= Oct 30 17:08:14.413632 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:14.413170 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-330282-openshift-default-5784687855-s68x6" podUID="ce11f38f-0ae0-4881-aa0f-346a0dd513c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" Oct 30 17:08:14.799490 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:14.799391 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-e5bfb0-6f84945dc8-s9w5w" podUID="731de2b2-ab6b-499b-a12e-21b7a550ea86" containerName="payload-processing" containerID="cri-o://9b128d66989128941673582c034464e89983bf1d1394b7b2a189515dc22f85b0" gracePeriod=25 Oct 30 17:08:14.856052 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:14.855952 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-e5bfb0-86457d6567-v7zt5" podUID="72b342d0-d3cc-43c3-8013-ef235a934f3b" containerName="payload-pre-processing" containerID="cri-o://6f8a74af7c300932373f5973ace89ca43d6806bc45da9f4a264beeffecd593be" gracePeriod=25 Oct 30 17:08:15.079741 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:15.079662 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ac125-84b8c9c875-7gtbm" Oct 30 17:08:15.079741 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:15.079728 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-4ac125-9dcf56996-v4vjr" Oct 30 17:08:15.417514 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:15.417483 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-330282-openshift-default-5784687855-s68x6" Oct 30 17:08:15.418784 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:15.418766 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-330282-openshift-default-5784687855-s68x6" Oct 30 17:08:16.858172 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:16.858130 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-e5bfb0-86457d6567-v7zt5" podUID="72b342d0-d3cc-43c3-8013-ef235a934f3b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:08:16.858671 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:16.858380 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-e5bfb0-6f84945dc8-s9w5w" podUID="731de2b2-ab6b-499b-a12e-21b7a550ea86" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:08:18.841510 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:08:18.841471 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 17:08:23.332803 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:23.332769 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e5bfb0-8486ccf4c4-kqwsv"] Oct 30 17:08:23.333497 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:23.333037 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e5bfb0-8486ccf4c4-kqwsv" podUID="0562b1aa-d0da-4e5e-ba02-2042715e7b35" containerName="maas-api" containerID="cri-o://35196dbf053d6615411723a65ad229bd92c2de53f9603b2988032fd58936133a" gracePeriod=30 Oct 30 17:08:23.860896 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:23.856429 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e5bfb0-8486ccf4c4-kqwsv" Oct 30 17:08:23.980809 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:23.968208 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0562b1aa-d0da-4e5e-ba02-2042715e7b35-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0562b1aa-d0da-4e5e-ba02-2042715e7b35-maas-api-tls\") pod \"0562b1aa-d0da-4e5e-ba02-2042715e7b35\" (UID: \"0562b1aa-d0da-4e5e-ba02-2042715e7b35\") " Oct 30 17:08:23.980809 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:23.979889 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0562b1aa-d0da-4e5e-ba02-2042715e7b35-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0562b1aa-d0da-4e5e-ba02-2042715e7b35-openshift-service-ca\") pod \"0562b1aa-d0da-4e5e-ba02-2042715e7b35\" (UID: \"0562b1aa-d0da-4e5e-ba02-2042715e7b35\") " Oct 30 17:08:23.980809 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:23.979974 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0562b1aa-d0da-4e5e-ba02-2042715e7b35-kube-api-access-j7dtt\" (UniqueName: \"kubernetes.io/projected/0562b1aa-d0da-4e5e-ba02-2042715e7b35-kube-api-access-j7dtt\") pod \"0562b1aa-d0da-4e5e-ba02-2042715e7b35\" (UID: \"0562b1aa-d0da-4e5e-ba02-2042715e7b35\") " Oct 30 17:08:23.980809 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:23.980151 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0562b1aa-d0da-4e5e-ba02-2042715e7b35-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0562b1aa-d0da-4e5e-ba02-2042715e7b35-metrics-certs\") pod \"0562b1aa-d0da-4e5e-ba02-2042715e7b35\" (UID: \"0562b1aa-d0da-4e5e-ba02-2042715e7b35\") " Oct 30 17:08:23.981882 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:23.981834 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0562b1aa-d0da-4e5e-ba02-2042715e7b35-openshift-service-ca" pod "0562b1aa-d0da-4e5e-ba02-2042715e7b35" (UID: "0562b1aa-d0da-4e5e-ba02-2042715e7b35"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:08:23.984762 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:23.984736 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0562b1aa-d0da-4e5e-ba02-2042715e7b35-maas-api-tls" pod "0562b1aa-d0da-4e5e-ba02-2042715e7b35" (UID: "0562b1aa-d0da-4e5e-ba02-2042715e7b35"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:08:23.990440 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:23.990180 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0562b1aa-d0da-4e5e-ba02-2042715e7b35-metrics-certs" pod "0562b1aa-d0da-4e5e-ba02-2042715e7b35" (UID: "0562b1aa-d0da-4e5e-ba02-2042715e7b35"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:08:23.990440 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:23.990241 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0562b1aa-d0da-4e5e-ba02-2042715e7b35-kube-api-access-j7dtt" pod "0562b1aa-d0da-4e5e-ba02-2042715e7b35" (UID: "0562b1aa-d0da-4e5e-ba02-2042715e7b35"). InnerVolumeSpecName "kube-api-access-j7dtt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:08:24.082384 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:24.082351 5052 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0562b1aa-d0da-4e5e-ba02-2042715e7b35-openshift-service-ca\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:08:24.082384 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:24.082378 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7dtt\" (UniqueName: \"kubernetes.io/projected/0562b1aa-d0da-4e5e-ba02-2042715e7b35-kube-api-access-j7dtt\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:08:24.082384 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:24.082392 5052 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0562b1aa-d0da-4e5e-ba02-2042715e7b35-metrics-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:08:24.082682 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:24.082401 5052 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0562b1aa-d0da-4e5e-ba02-2042715e7b35-maas-api-tls\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:08:24.419518 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:24.419482 5052 generic.go:356] "Generic (PLEG): container finished" podID="0562b1aa-d0da-4e5e-ba02-2042715e7b35" containerID="35196dbf053d6615411723a65ad229bd92c2de53f9603b2988032fd58936133a" exitCode=0 Oct 30 17:08:24.420076 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:24.419530 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e5bfb0-8486ccf4c4-kqwsv" Oct 30 17:08:24.420076 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:24.419572 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e5bfb0-8486ccf4c4-kqwsv" event={"ID":"0562b1aa-d0da-4e5e-ba02-2042715e7b35","Type":"ContainerDied","Data":"35196dbf053d6615411723a65ad229bd92c2de53f9603b2988032fd58936133a"} Oct 30 17:08:24.420076 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:24.419656 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e5bfb0-8486ccf4c4-kqwsv" event={"ID":"0562b1aa-d0da-4e5e-ba02-2042715e7b35","Type":"ContainerDied","Data":"ee78c96aca15c2398ac3c007d73f8fdfb9fa319e10f0b82a1c214defd2d870bd"} Oct 30 17:08:24.420076 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:24.419685 5052 scope.go:122] "RemoveContainer" containerID="35196dbf053d6615411723a65ad229bd92c2de53f9603b2988032fd58936133a" Oct 30 17:08:24.431740 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:24.431718 5052 scope.go:122] "RemoveContainer" containerID="35196dbf053d6615411723a65ad229bd92c2de53f9603b2988032fd58936133a" Oct 30 17:08:24.432019 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:08:24.431995 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35196dbf053d6615411723a65ad229bd92c2de53f9603b2988032fd58936133a\": container with ID starting with 35196dbf053d6615411723a65ad229bd92c2de53f9603b2988032fd58936133a not found: ID does not exist" containerID="35196dbf053d6615411723a65ad229bd92c2de53f9603b2988032fd58936133a" Oct 30 17:08:24.432126 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:24.432023 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35196dbf053d6615411723a65ad229bd92c2de53f9603b2988032fd58936133a"} err="failed to get container status \"35196dbf053d6615411723a65ad229bd92c2de53f9603b2988032fd58936133a\": rpc error: code = NotFound desc = could not find container \"35196dbf053d6615411723a65ad229bd92c2de53f9603b2988032fd58936133a\": container with ID starting with 35196dbf053d6615411723a65ad229bd92c2de53f9603b2988032fd58936133a not found: ID does not exist" Oct 30 17:08:24.517228 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:24.517195 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e5bfb0-8486ccf4c4-kqwsv"] Oct 30 17:08:24.525490 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:24.525445 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e5bfb0-8486ccf4c4-kqwsv"] Oct 30 17:08:24.604334 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:24.604281 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0562b1aa-d0da-4e5e-ba02-2042715e7b35" path="/var/lib/kubelet/pods/0562b1aa-d0da-4e5e-ba02-2042715e7b35/volumes" Oct 30 17:08:25.175418 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:25.175382 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-97a47d-openshift-default-5dcd657cdf-8l2c4"] Oct 30 17:08:25.176170 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:25.176154 5052 cpu_manager.go:414] "containerMap: removing container" podUID="d5691fdc-cd71-42ca-9eb7-98d431c101cc" containerName="payload-pre-processing" Oct 30 17:08:25.176233 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:25.176171 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5691fdc-cd71-42ca-9eb7-98d431c101cc" containerName="payload-pre-processing" Oct 30 17:08:25.176275 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:25.176246 5052 cpu_manager.go:414] "containerMap: removing container" podUID="0562b1aa-d0da-4e5e-ba02-2042715e7b35" containerName="maas-api" Oct 30 17:08:25.176275 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:25.176253 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0562b1aa-d0da-4e5e-ba02-2042715e7b35" containerName="maas-api" Oct 30 17:08:25.176395 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:25.176384 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="0562b1aa-d0da-4e5e-ba02-2042715e7b35" containerName="maas-api" Oct 30 17:08:25.176434 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:25.176400 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5691fdc-cd71-42ca-9eb7-98d431c101cc" containerName="payload-pre-processing" Oct 30 17:08:25.181921 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:25.181894 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-97a47d-openshift-default-5dcd657cdf-8l2c4" Oct 30 17:08:25.208157 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:25.207856 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-97a47d-openshift-default-5dcd657cdf-8l2c4"] Oct 30 17:08:25.265828 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:25.265787 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-97a47d-openshift-default-dockercfg-7jj48\"" Oct 30 17:08:25.356600 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:25.343838 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/837038f3-5741-4b1f-8bb8-f8c8e234128f-istiod-ca-cert\") pod \"e2e-ipp-b-97a47d-openshift-default-5dcd657cdf-8l2c4\" (UID: \"837038f3-5741-4b1f-8bb8-f8c8e234128f\") " pod="openshift-ingress/e2e-ipp-b-97a47d-openshift-default-5dcd657cdf-8l2c4" Oct 30 17:08:25.356600 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:25.343973 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/837038f3-5741-4b1f-8bb8-f8c8e234128f-istio-envoy\") pod \"e2e-ipp-b-97a47d-openshift-default-5dcd657cdf-8l2c4\" (UID: \"837038f3-5741-4b1f-8bb8-f8c8e234128f\") " pod="openshift-ingress/e2e-ipp-b-97a47d-openshift-default-5dcd657cdf-8l2c4" Oct 30 17:08:25.356600 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:25.344047 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/837038f3-5741-4b1f-8bb8-f8c8e234128f-istio-podinfo\") pod \"e2e-ipp-b-97a47d-openshift-default-5dcd657cdf-8l2c4\" (UID: \"837038f3-5741-4b1f-8bb8-f8c8e234128f\") " pod="openshift-ingress/e2e-ipp-b-97a47d-openshift-default-5dcd657cdf-8l2c4" Oct 30 17:08:25.356600 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:25.344114 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/837038f3-5741-4b1f-8bb8-f8c8e234128f-credential-socket\") pod \"e2e-ipp-b-97a47d-openshift-default-5dcd657cdf-8l2c4\" (UID: \"837038f3-5741-4b1f-8bb8-f8c8e234128f\") " pod="openshift-ingress/e2e-ipp-b-97a47d-openshift-default-5dcd657cdf-8l2c4" Oct 30 17:08:25.356600 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:25.344144 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9dwr\" (UniqueName: \"kubernetes.io/projected/837038f3-5741-4b1f-8bb8-f8c8e234128f-kube-api-access-k9dwr\") pod \"e2e-ipp-b-97a47d-openshift-default-5dcd657cdf-8l2c4\" (UID: \"837038f3-5741-4b1f-8bb8-f8c8e234128f\") " pod="openshift-ingress/e2e-ipp-b-97a47d-openshift-default-5dcd657cdf-8l2c4" Oct 30 17:08:25.356600 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:25.344306 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/837038f3-5741-4b1f-8bb8-f8c8e234128f-istio-token\") pod \"e2e-ipp-b-97a47d-openshift-default-5dcd657cdf-8l2c4\" (UID: \"837038f3-5741-4b1f-8bb8-f8c8e234128f\") " pod="openshift-ingress/e2e-ipp-b-97a47d-openshift-default-5dcd657cdf-8l2c4" Oct 30 17:08:25.356600 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:25.344366 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/837038f3-5741-4b1f-8bb8-f8c8e234128f-workload-socket\") pod \"e2e-ipp-b-97a47d-openshift-default-5dcd657cdf-8l2c4\" (UID: \"837038f3-5741-4b1f-8bb8-f8c8e234128f\") " pod="openshift-ingress/e2e-ipp-b-97a47d-openshift-default-5dcd657cdf-8l2c4" Oct 30 17:08:25.356600 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:25.344438 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/837038f3-5741-4b1f-8bb8-f8c8e234128f-istio-data\") pod \"e2e-ipp-b-97a47d-openshift-default-5dcd657cdf-8l2c4\" (UID: \"837038f3-5741-4b1f-8bb8-f8c8e234128f\") " pod="openshift-ingress/e2e-ipp-b-97a47d-openshift-default-5dcd657cdf-8l2c4" Oct 30 17:08:25.356600 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:25.344629 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/837038f3-5741-4b1f-8bb8-f8c8e234128f-workload-certs\") pod \"e2e-ipp-b-97a47d-openshift-default-5dcd657cdf-8l2c4\" (UID: \"837038f3-5741-4b1f-8bb8-f8c8e234128f\") " pod="openshift-ingress/e2e-ipp-b-97a47d-openshift-default-5dcd657cdf-8l2c4" Oct 30 17:08:25.469757 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:25.465337 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/837038f3-5741-4b1f-8bb8-f8c8e234128f-workload-certs\") pod \"e2e-ipp-b-97a47d-openshift-default-5dcd657cdf-8l2c4\" (UID: \"837038f3-5741-4b1f-8bb8-f8c8e234128f\") " pod="openshift-ingress/e2e-ipp-b-97a47d-openshift-default-5dcd657cdf-8l2c4" Oct 30 17:08:25.469757 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:25.465476 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/837038f3-5741-4b1f-8bb8-f8c8e234128f-istiod-ca-cert\") pod \"e2e-ipp-b-97a47d-openshift-default-5dcd657cdf-8l2c4\" (UID: \"837038f3-5741-4b1f-8bb8-f8c8e234128f\") " pod="openshift-ingress/e2e-ipp-b-97a47d-openshift-default-5dcd657cdf-8l2c4" Oct 30 17:08:25.469757 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:25.465509 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/837038f3-5741-4b1f-8bb8-f8c8e234128f-istio-envoy\") pod \"e2e-ipp-b-97a47d-openshift-default-5dcd657cdf-8l2c4\" (UID: \"837038f3-5741-4b1f-8bb8-f8c8e234128f\") " pod="openshift-ingress/e2e-ipp-b-97a47d-openshift-default-5dcd657cdf-8l2c4" Oct 30 17:08:25.469757 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:25.465683 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/837038f3-5741-4b1f-8bb8-f8c8e234128f-istio-podinfo\") pod \"e2e-ipp-b-97a47d-openshift-default-5dcd657cdf-8l2c4\" (UID: \"837038f3-5741-4b1f-8bb8-f8c8e234128f\") " pod="openshift-ingress/e2e-ipp-b-97a47d-openshift-default-5dcd657cdf-8l2c4" Oct 30 17:08:25.469757 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:25.465750 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/837038f3-5741-4b1f-8bb8-f8c8e234128f-credential-socket\") pod \"e2e-ipp-b-97a47d-openshift-default-5dcd657cdf-8l2c4\" (UID: \"837038f3-5741-4b1f-8bb8-f8c8e234128f\") " pod="openshift-ingress/e2e-ipp-b-97a47d-openshift-default-5dcd657cdf-8l2c4" Oct 30 17:08:25.469757 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:25.465777 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9dwr\" (UniqueName: \"kubernetes.io/projected/837038f3-5741-4b1f-8bb8-f8c8e234128f-kube-api-access-k9dwr\") pod \"e2e-ipp-b-97a47d-openshift-default-5dcd657cdf-8l2c4\" (UID: \"837038f3-5741-4b1f-8bb8-f8c8e234128f\") " pod="openshift-ingress/e2e-ipp-b-97a47d-openshift-default-5dcd657cdf-8l2c4" Oct 30 17:08:25.469757 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:25.465904 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/837038f3-5741-4b1f-8bb8-f8c8e234128f-istio-token\") pod \"e2e-ipp-b-97a47d-openshift-default-5dcd657cdf-8l2c4\" (UID: \"837038f3-5741-4b1f-8bb8-f8c8e234128f\") " pod="openshift-ingress/e2e-ipp-b-97a47d-openshift-default-5dcd657cdf-8l2c4" Oct 30 17:08:25.469757 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:25.465954 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/837038f3-5741-4b1f-8bb8-f8c8e234128f-workload-socket\") pod \"e2e-ipp-b-97a47d-openshift-default-5dcd657cdf-8l2c4\" (UID: \"837038f3-5741-4b1f-8bb8-f8c8e234128f\") " pod="openshift-ingress/e2e-ipp-b-97a47d-openshift-default-5dcd657cdf-8l2c4" Oct 30 17:08:25.469757 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:25.466017 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/837038f3-5741-4b1f-8bb8-f8c8e234128f-istio-data\") pod \"e2e-ipp-b-97a47d-openshift-default-5dcd657cdf-8l2c4\" (UID: \"837038f3-5741-4b1f-8bb8-f8c8e234128f\") " pod="openshift-ingress/e2e-ipp-b-97a47d-openshift-default-5dcd657cdf-8l2c4" Oct 30 17:08:25.469757 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:25.466560 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/837038f3-5741-4b1f-8bb8-f8c8e234128f-istio-data\") pod \"e2e-ipp-b-97a47d-openshift-default-5dcd657cdf-8l2c4\" (UID: \"837038f3-5741-4b1f-8bb8-f8c8e234128f\") " pod="openshift-ingress/e2e-ipp-b-97a47d-openshift-default-5dcd657cdf-8l2c4" Oct 30 17:08:25.497612 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:25.476765 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/837038f3-5741-4b1f-8bb8-f8c8e234128f-credential-socket\") pod \"e2e-ipp-b-97a47d-openshift-default-5dcd657cdf-8l2c4\" (UID: \"837038f3-5741-4b1f-8bb8-f8c8e234128f\") " pod="openshift-ingress/e2e-ipp-b-97a47d-openshift-default-5dcd657cdf-8l2c4" Oct 30 17:08:25.497612 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:25.477278 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/837038f3-5741-4b1f-8bb8-f8c8e234128f-workload-socket\") pod \"e2e-ipp-b-97a47d-openshift-default-5dcd657cdf-8l2c4\" (UID: \"837038f3-5741-4b1f-8bb8-f8c8e234128f\") " pod="openshift-ingress/e2e-ipp-b-97a47d-openshift-default-5dcd657cdf-8l2c4" Oct 30 17:08:25.497612 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:25.477918 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/837038f3-5741-4b1f-8bb8-f8c8e234128f-istiod-ca-cert\") pod \"e2e-ipp-b-97a47d-openshift-default-5dcd657cdf-8l2c4\" (UID: \"837038f3-5741-4b1f-8bb8-f8c8e234128f\") " pod="openshift-ingress/e2e-ipp-b-97a47d-openshift-default-5dcd657cdf-8l2c4" Oct 30 17:08:25.497612 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:25.478280 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/837038f3-5741-4b1f-8bb8-f8c8e234128f-workload-certs\") pod \"e2e-ipp-b-97a47d-openshift-default-5dcd657cdf-8l2c4\" (UID: \"837038f3-5741-4b1f-8bb8-f8c8e234128f\") " pod="openshift-ingress/e2e-ipp-b-97a47d-openshift-default-5dcd657cdf-8l2c4" Oct 30 17:08:25.533613 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:25.519922 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/837038f3-5741-4b1f-8bb8-f8c8e234128f-istio-envoy\") pod \"e2e-ipp-b-97a47d-openshift-default-5dcd657cdf-8l2c4\" (UID: \"837038f3-5741-4b1f-8bb8-f8c8e234128f\") " pod="openshift-ingress/e2e-ipp-b-97a47d-openshift-default-5dcd657cdf-8l2c4" Oct 30 17:08:25.533613 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:25.520460 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/837038f3-5741-4b1f-8bb8-f8c8e234128f-istio-podinfo\") pod \"e2e-ipp-b-97a47d-openshift-default-5dcd657cdf-8l2c4\" (UID: \"837038f3-5741-4b1f-8bb8-f8c8e234128f\") " pod="openshift-ingress/e2e-ipp-b-97a47d-openshift-default-5dcd657cdf-8l2c4" Oct 30 17:08:25.534979 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:25.534911 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9dwr\" (UniqueName: \"kubernetes.io/projected/837038f3-5741-4b1f-8bb8-f8c8e234128f-kube-api-access-k9dwr\") pod \"e2e-ipp-b-97a47d-openshift-default-5dcd657cdf-8l2c4\" (UID: \"837038f3-5741-4b1f-8bb8-f8c8e234128f\") " pod="openshift-ingress/e2e-ipp-b-97a47d-openshift-default-5dcd657cdf-8l2c4" Oct 30 17:08:25.542219 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:25.535656 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/837038f3-5741-4b1f-8bb8-f8c8e234128f-istio-token\") pod \"e2e-ipp-b-97a47d-openshift-default-5dcd657cdf-8l2c4\" (UID: \"837038f3-5741-4b1f-8bb8-f8c8e234128f\") " pod="openshift-ingress/e2e-ipp-b-97a47d-openshift-default-5dcd657cdf-8l2c4" Oct 30 17:08:25.809821 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:25.809739 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-97a47d-openshift-default-5dcd657cdf-8l2c4" Oct 30 17:08:26.150799 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:26.150765 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-97a47d-openshift-default-5dcd657cdf-8l2c4"] Oct 30 17:08:26.152068 ip-10-0-131-136 kubenswrapper[5052]: W1030 17:08:26.152036 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod837038f3_5741_4b1f_8bb8_f8c8e234128f.slice/crio-00350f34d5be6acf192c00300818919279fa96c41ed425a8f91de8c4c45a4328 WatchSource:0}: Error finding container 00350f34d5be6acf192c00300818919279fa96c41ed425a8f91de8c4c45a4328: Status 404 returned error can't find the container with id 00350f34d5be6acf192c00300818919279fa96c41ed425a8f91de8c4c45a4328 Oct 30 17:08:26.155371 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:26.155338 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:08:26.155461 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:26.155409 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:08:26.155461 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:26.155442 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:08:26.522097 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:26.522007 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-97a47d-openshift-default-5dcd657cdf-8l2c4" event={"ID":"837038f3-5741-4b1f-8bb8-f8c8e234128f","Type":"ContainerStarted","Data":"8899d0b5a7283c14ed4e600ce5a5a154d462e7f984936dbc51debcd7da8241c2"} Oct 30 17:08:26.522097 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:26.522086 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-97a47d-openshift-default-5dcd657cdf-8l2c4" event={"ID":"837038f3-5741-4b1f-8bb8-f8c8e234128f","Type":"ContainerStarted","Data":"00350f34d5be6acf192c00300818919279fa96c41ed425a8f91de8c4c45a4328"} Oct 30 17:08:26.672824 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:26.672782 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e5bfb0-8486ccf4c4-kqwsv" podUID="0562b1aa-d0da-4e5e-ba02-2042715e7b35" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.119:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:08:26.809969 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:26.809884 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-97a47d-openshift-default-5dcd657cdf-8l2c4" Oct 30 17:08:26.857460 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:26.857419 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-e5bfb0-86457d6567-v7zt5" podUID="72b342d0-d3cc-43c3-8013-ef235a934f3b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:08:26.858291 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:26.858266 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-e5bfb0-6f84945dc8-s9w5w" podUID="731de2b2-ab6b-499b-a12e-21b7a550ea86" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:08:27.818276 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:27.818240 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-97a47d-openshift-default-5dcd657cdf-8l2c4" Oct 30 17:08:27.881963 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:27.881910 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-97a47d-openshift-default-5dcd657cdf-8l2c4" podStartSLOduration=2.8818976689999998 podStartE2EDuration="2.881897669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:08:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:08:26.584655245 +0000 UTC m=+2862.111990465" watchObservedRunningTime="2026-10-30 17:08:27.881897669 +0000 UTC m=+2863.409232888" Oct 30 17:08:28.537504 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:28.537468 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-97a47d-openshift-default-5dcd657cdf-8l2c4" Oct 30 17:08:28.538536 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:28.538515 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-97a47d-openshift-default-5dcd657cdf-8l2c4" Oct 30 17:08:29.566577 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:29.566538 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-cb57p"] Oct 30 17:08:29.571400 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:29.571378 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-cb57p" Oct 30 17:08:29.601438 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:29.601411 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-ae5afd-openshift-default-dockercfg-mqb49\"" Oct 30 17:08:29.606887 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:29.606861 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-cb57p"] Oct 30 17:08:29.677490 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:29.677460 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4036d90-5e91-477e-b81f-91c2dceee6c9-istiod-ca-cert\") pod \"e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-cb57p\" (UID: \"e4036d90-5e91-477e-b81f-91c2dceee6c9\") " pod="openshift-ingress/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-cb57p" Oct 30 17:08:29.677645 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:29.677503 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4036d90-5e91-477e-b81f-91c2dceee6c9-workload-certs\") pod \"e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-cb57p\" (UID: \"e4036d90-5e91-477e-b81f-91c2dceee6c9\") " pod="openshift-ingress/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-cb57p" Oct 30 17:08:29.677645 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:29.677525 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4036d90-5e91-477e-b81f-91c2dceee6c9-istio-podinfo\") pod \"e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-cb57p\" (UID: \"e4036d90-5e91-477e-b81f-91c2dceee6c9\") " pod="openshift-ingress/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-cb57p" Oct 30 17:08:29.677746 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:29.677682 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4036d90-5e91-477e-b81f-91c2dceee6c9-workload-socket\") pod \"e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-cb57p\" (UID: \"e4036d90-5e91-477e-b81f-91c2dceee6c9\") " pod="openshift-ingress/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-cb57p" Oct 30 17:08:29.677896 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:29.677879 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4036d90-5e91-477e-b81f-91c2dceee6c9-istio-token\") pod \"e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-cb57p\" (UID: \"e4036d90-5e91-477e-b81f-91c2dceee6c9\") " pod="openshift-ingress/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-cb57p" Oct 30 17:08:29.677966 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:29.677950 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4036d90-5e91-477e-b81f-91c2dceee6c9-credential-socket\") pod \"e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-cb57p\" (UID: \"e4036d90-5e91-477e-b81f-91c2dceee6c9\") " pod="openshift-ingress/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-cb57p" Oct 30 17:08:29.678108 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:29.678074 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4036d90-5e91-477e-b81f-91c2dceee6c9-istio-envoy\") pod \"e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-cb57p\" (UID: \"e4036d90-5e91-477e-b81f-91c2dceee6c9\") " pod="openshift-ingress/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-cb57p" Oct 30 17:08:29.678332 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:29.678306 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4036d90-5e91-477e-b81f-91c2dceee6c9-istio-data\") pod \"e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-cb57p\" (UID: \"e4036d90-5e91-477e-b81f-91c2dceee6c9\") " pod="openshift-ingress/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-cb57p" Oct 30 17:08:29.678427 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:29.678357 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkpv6\" (UniqueName: \"kubernetes.io/projected/e4036d90-5e91-477e-b81f-91c2dceee6c9-kube-api-access-mkpv6\") pod \"e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-cb57p\" (UID: \"e4036d90-5e91-477e-b81f-91c2dceee6c9\") " pod="openshift-ingress/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-cb57p" Oct 30 17:08:29.779928 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:29.779898 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4036d90-5e91-477e-b81f-91c2dceee6c9-istio-data\") pod \"e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-cb57p\" (UID: \"e4036d90-5e91-477e-b81f-91c2dceee6c9\") " pod="openshift-ingress/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-cb57p" Oct 30 17:08:29.780081 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:29.779937 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkpv6\" (UniqueName: \"kubernetes.io/projected/e4036d90-5e91-477e-b81f-91c2dceee6c9-kube-api-access-mkpv6\") pod \"e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-cb57p\" (UID: \"e4036d90-5e91-477e-b81f-91c2dceee6c9\") " pod="openshift-ingress/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-cb57p" Oct 30 17:08:29.780081 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:29.780024 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4036d90-5e91-477e-b81f-91c2dceee6c9-istiod-ca-cert\") pod \"e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-cb57p\" (UID: \"e4036d90-5e91-477e-b81f-91c2dceee6c9\") " pod="openshift-ingress/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-cb57p" Oct 30 17:08:29.780081 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:29.780059 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4036d90-5e91-477e-b81f-91c2dceee6c9-workload-certs\") pod \"e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-cb57p\" (UID: \"e4036d90-5e91-477e-b81f-91c2dceee6c9\") " pod="openshift-ingress/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-cb57p" Oct 30 17:08:29.780267 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:29.780084 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4036d90-5e91-477e-b81f-91c2dceee6c9-istio-podinfo\") pod \"e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-cb57p\" (UID: \"e4036d90-5e91-477e-b81f-91c2dceee6c9\") " pod="openshift-ingress/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-cb57p" Oct 30 17:08:29.780267 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:29.780112 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4036d90-5e91-477e-b81f-91c2dceee6c9-workload-socket\") pod \"e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-cb57p\" (UID: \"e4036d90-5e91-477e-b81f-91c2dceee6c9\") " pod="openshift-ingress/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-cb57p" Oct 30 17:08:29.780387 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:29.780284 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4036d90-5e91-477e-b81f-91c2dceee6c9-istio-token\") pod \"e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-cb57p\" (UID: \"e4036d90-5e91-477e-b81f-91c2dceee6c9\") " pod="openshift-ingress/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-cb57p" Oct 30 17:08:29.780387 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:29.780359 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4036d90-5e91-477e-b81f-91c2dceee6c9-credential-socket\") pod \"e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-cb57p\" (UID: \"e4036d90-5e91-477e-b81f-91c2dceee6c9\") " pod="openshift-ingress/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-cb57p" Oct 30 17:08:29.780387 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:29.780364 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4036d90-5e91-477e-b81f-91c2dceee6c9-istio-data\") pod \"e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-cb57p\" (UID: \"e4036d90-5e91-477e-b81f-91c2dceee6c9\") " pod="openshift-ingress/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-cb57p" Oct 30 17:08:29.780558 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:29.780422 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4036d90-5e91-477e-b81f-91c2dceee6c9-istio-envoy\") pod \"e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-cb57p\" (UID: \"e4036d90-5e91-477e-b81f-91c2dceee6c9\") " pod="openshift-ingress/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-cb57p" Oct 30 17:08:29.780558 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:29.780422 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4036d90-5e91-477e-b81f-91c2dceee6c9-workload-certs\") pod \"e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-cb57p\" (UID: \"e4036d90-5e91-477e-b81f-91c2dceee6c9\") " pod="openshift-ingress/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-cb57p" Oct 30 17:08:29.780845 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:29.780817 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4036d90-5e91-477e-b81f-91c2dceee6c9-istiod-ca-cert\") pod \"e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-cb57p\" (UID: \"e4036d90-5e91-477e-b81f-91c2dceee6c9\") " pod="openshift-ingress/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-cb57p" Oct 30 17:08:29.780978 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:29.780849 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4036d90-5e91-477e-b81f-91c2dceee6c9-credential-socket\") pod \"e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-cb57p\" (UID: \"e4036d90-5e91-477e-b81f-91c2dceee6c9\") " pod="openshift-ingress/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-cb57p" Oct 30 17:08:29.780978 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:29.780901 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4036d90-5e91-477e-b81f-91c2dceee6c9-workload-socket\") pod \"e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-cb57p\" (UID: \"e4036d90-5e91-477e-b81f-91c2dceee6c9\") " pod="openshift-ingress/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-cb57p" Oct 30 17:08:29.783223 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:29.783200 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4036d90-5e91-477e-b81f-91c2dceee6c9-istio-envoy\") pod \"e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-cb57p\" (UID: \"e4036d90-5e91-477e-b81f-91c2dceee6c9\") " pod="openshift-ingress/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-cb57p" Oct 30 17:08:29.783513 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:29.783494 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4036d90-5e91-477e-b81f-91c2dceee6c9-istio-podinfo\") pod \"e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-cb57p\" (UID: \"e4036d90-5e91-477e-b81f-91c2dceee6c9\") " pod="openshift-ingress/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-cb57p" Oct 30 17:08:29.805300 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:29.805275 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4036d90-5e91-477e-b81f-91c2dceee6c9-istio-token\") pod \"e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-cb57p\" (UID: \"e4036d90-5e91-477e-b81f-91c2dceee6c9\") " pod="openshift-ingress/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-cb57p" Oct 30 17:08:29.805425 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:29.805308 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkpv6\" (UniqueName: \"kubernetes.io/projected/e4036d90-5e91-477e-b81f-91c2dceee6c9-kube-api-access-mkpv6\") pod \"e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-cb57p\" (UID: \"e4036d90-5e91-477e-b81f-91c2dceee6c9\") " pod="openshift-ingress/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-cb57p" Oct 30 17:08:29.886979 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:29.886909 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-cb57p" Oct 30 17:08:30.221135 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:30.220965 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-cb57p"] Oct 30 17:08:30.224528 ip-10-0-131-136 kubenswrapper[5052]: W1030 17:08:30.224496 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4036d90_5e91_477e_b81f_91c2dceee6c9.slice/crio-521908381a2a94c97ccc673e0cbe34907905b8ae68879361f1c4dfc95cdbd8b7 WatchSource:0}: Error finding container 521908381a2a94c97ccc673e0cbe34907905b8ae68879361f1c4dfc95cdbd8b7: Status 404 returned error can't find the container with id 521908381a2a94c97ccc673e0cbe34907905b8ae68879361f1c4dfc95cdbd8b7 Oct 30 17:08:30.227499 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:30.227459 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:08:30.227634 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:30.227544 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:08:30.227634 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:30.227577 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:08:30.554521 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:30.554428 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-cb57p" event={"ID":"e4036d90-5e91-477e-b81f-91c2dceee6c9","Type":"ContainerStarted","Data":"670199c105c11a949f324aa937df5522c7ee9bff0b95204e0ca9c22d9ce3c414"} Oct 30 17:08:30.554521 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:30.554492 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-cb57p" event={"ID":"e4036d90-5e91-477e-b81f-91c2dceee6c9","Type":"ContainerStarted","Data":"521908381a2a94c97ccc673e0cbe34907905b8ae68879361f1c4dfc95cdbd8b7"} Oct 30 17:08:30.605624 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:30.605531 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-cb57p" podStartSLOduration=1.6055177 podStartE2EDuration="1.6055177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:08:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:08:30.604952869 +0000 UTC m=+2866.132288109" watchObservedRunningTime="2026-10-30 17:08:30.6055177 +0000 UTC m=+2866.132852919" Oct 30 17:08:30.887873 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:30.887780 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-cb57p" Oct 30 17:08:31.889631 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:31.889546 5052 patch_prober.go:29] interesting pod/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-cb57p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body= Oct 30 17:08:31.890136 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:31.889661 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-cb57p" podUID="e4036d90-5e91-477e-b81f-91c2dceee6c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" Oct 30 17:08:32.889151 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:32.889107 5052 patch_prober.go:29] interesting pod/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-cb57p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body= Oct 30 17:08:32.893073 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:32.893032 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-cb57p" podUID="e4036d90-5e91-477e-b81f-91c2dceee6c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" Oct 30 17:08:33.852614 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:33.847198 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cdwcj"] Oct 30 17:08:33.894610 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:33.894057 5052 patch_prober.go:29] interesting pod/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-cb57p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body= Oct 30 17:08:33.894610 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:33.894120 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-cb57p" podUID="e4036d90-5e91-477e-b81f-91c2dceee6c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" Oct 30 17:08:34.895611 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:34.893034 5052 patch_prober.go:29] interesting pod/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-cb57p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body= Oct 30 17:08:34.895611 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:34.893121 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-cb57p" podUID="e4036d90-5e91-477e-b81f-91c2dceee6c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" Oct 30 17:08:35.218701 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:35.218669 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-e5bfb0-6f84945dc8-s9w5w" Oct 30 17:08:35.381655 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:35.380797 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-e5bfb0-86457d6567-v7zt5" Oct 30 17:08:35.414427 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:35.414355 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/731de2b2-ab6b-499b-a12e-21b7a550ea86-kube-api-access-rr9sf\" (UniqueName: \"kubernetes.io/projected/731de2b2-ab6b-499b-a12e-21b7a550ea86-kube-api-access-rr9sf\") pod \"731de2b2-ab6b-499b-a12e-21b7a550ea86\" (UID: \"731de2b2-ab6b-499b-a12e-21b7a550ea86\") " Oct 30 17:08:35.414740 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:35.414716 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/731de2b2-ab6b-499b-a12e-21b7a550ea86-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/731de2b2-ab6b-499b-a12e-21b7a550ea86-plugins-config-volume\") pod \"731de2b2-ab6b-499b-a12e-21b7a550ea86\" (UID: \"731de2b2-ab6b-499b-a12e-21b7a550ea86\") " Oct 30 17:08:35.435611 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:35.432804 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/731de2b2-ab6b-499b-a12e-21b7a550ea86-plugins-config-volume" pod "731de2b2-ab6b-499b-a12e-21b7a550ea86" (UID: "731de2b2-ab6b-499b-a12e-21b7a550ea86"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:08:35.435611 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:35.434220 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/731de2b2-ab6b-499b-a12e-21b7a550ea86-kube-api-access-rr9sf" pod "731de2b2-ab6b-499b-a12e-21b7a550ea86" (UID: "731de2b2-ab6b-499b-a12e-21b7a550ea86"). InnerVolumeSpecName "kube-api-access-rr9sf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:08:35.755603 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:35.753570 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72b342d0-d3cc-43c3-8013-ef235a934f3b-kube-api-access-k4tnc\" (UniqueName: \"kubernetes.io/projected/72b342d0-d3cc-43c3-8013-ef235a934f3b-kube-api-access-k4tnc\") pod \"72b342d0-d3cc-43c3-8013-ef235a934f3b\" (UID: \"72b342d0-d3cc-43c3-8013-ef235a934f3b\") " Oct 30 17:08:35.755603 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:35.753650 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72b342d0-d3cc-43c3-8013-ef235a934f3b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72b342d0-d3cc-43c3-8013-ef235a934f3b-plugins-config-volume\") pod \"72b342d0-d3cc-43c3-8013-ef235a934f3b\" (UID: \"72b342d0-d3cc-43c3-8013-ef235a934f3b\") " Oct 30 17:08:35.789516 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:35.775511 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72b342d0-d3cc-43c3-8013-ef235a934f3b-plugins-config-volume" pod "72b342d0-d3cc-43c3-8013-ef235a934f3b" (UID: "72b342d0-d3cc-43c3-8013-ef235a934f3b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:08:35.789516 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:35.777333 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rr9sf\" (UniqueName: \"kubernetes.io/projected/731de2b2-ab6b-499b-a12e-21b7a550ea86-kube-api-access-rr9sf\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:08:35.789516 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:35.777359 5052 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72b342d0-d3cc-43c3-8013-ef235a934f3b-plugins-config-volume\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:08:35.789516 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:35.777373 5052 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/731de2b2-ab6b-499b-a12e-21b7a550ea86-plugins-config-volume\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:08:35.806620 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:35.803162 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72b342d0-d3cc-43c3-8013-ef235a934f3b-kube-api-access-k4tnc" pod "72b342d0-d3cc-43c3-8013-ef235a934f3b" (UID: "72b342d0-d3cc-43c3-8013-ef235a934f3b"). InnerVolumeSpecName "kube-api-access-k4tnc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:08:35.820185 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:35.819027 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-e5bfb0-86457d6567-v7zt5" Oct 30 17:08:35.827054 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:35.827022 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-e5bfb0-86457d6567-v7zt5" event={"ID":"72b342d0-d3cc-43c3-8013-ef235a934f3b","Type":"ContainerDied","Data":"6f8a74af7c300932373f5973ace89ca43d6806bc45da9f4a264beeffecd593be"} Oct 30 17:08:35.827138 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:35.827098 5052 scope.go:122] "RemoveContainer" containerID="6f8a74af7c300932373f5973ace89ca43d6806bc45da9f4a264beeffecd593be" Oct 30 17:08:35.888408 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:35.888373 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4tnc\" (UniqueName: \"kubernetes.io/projected/72b342d0-d3cc-43c3-8013-ef235a934f3b-kube-api-access-k4tnc\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:08:35.928832 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:35.911655 5052 patch_prober.go:29] interesting pod/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-cb57p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body= Oct 30 17:08:35.929439 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:35.929405 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-cb57p" podUID="e4036d90-5e91-477e-b81f-91c2dceee6c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" Oct 30 17:08:35.931600 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:35.815258 5052 generic.go:356] "Generic (PLEG): container finished" podID="72b342d0-d3cc-43c3-8013-ef235a934f3b" containerID="6f8a74af7c300932373f5973ace89ca43d6806bc45da9f4a264beeffecd593be" exitCode=0 Oct 30 17:08:35.966605 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:35.963286 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-e5bfb0-86457d6567-v7zt5" event={"ID":"72b342d0-d3cc-43c3-8013-ef235a934f3b","Type":"ContainerDied","Data":"4cc5fc714bab705bc0b60bd087b07c73c9903fdf94cab9bbe80f67bb40b15ba2"} Oct 30 17:08:36.013784 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:36.013679 5052 generic.go:356] "Generic (PLEG): container finished" podID="731de2b2-ab6b-499b-a12e-21b7a550ea86" containerID="9b128d66989128941673582c034464e89983bf1d1394b7b2a189515dc22f85b0" exitCode=0 Oct 30 17:08:36.014098 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:36.014069 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-e5bfb0-6f84945dc8-s9w5w" event={"ID":"731de2b2-ab6b-499b-a12e-21b7a550ea86","Type":"ContainerDied","Data":"9b128d66989128941673582c034464e89983bf1d1394b7b2a189515dc22f85b0"} Oct 30 17:08:36.017646 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:36.014302 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-e5bfb0-6f84945dc8-s9w5w" event={"ID":"731de2b2-ab6b-499b-a12e-21b7a550ea86","Type":"ContainerDied","Data":"18925aa9090b8a36a10cc74e4c8d647f5447b6ad9a2f30e9c35cd0bc9624f736"} Oct 30 17:08:36.022706 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:36.022678 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-e5bfb0-6f84945dc8-s9w5w" Oct 30 17:08:36.343100 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:36.343066 5052 scope.go:122] "RemoveContainer" containerID="6f8a74af7c300932373f5973ace89ca43d6806bc45da9f4a264beeffecd593be" Oct 30 17:08:36.344211 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:08:36.344172 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f8a74af7c300932373f5973ace89ca43d6806bc45da9f4a264beeffecd593be\": container with ID starting with 6f8a74af7c300932373f5973ace89ca43d6806bc45da9f4a264beeffecd593be not found: ID does not exist" containerID="6f8a74af7c300932373f5973ace89ca43d6806bc45da9f4a264beeffecd593be" Oct 30 17:08:36.344364 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:36.344215 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f8a74af7c300932373f5973ace89ca43d6806bc45da9f4a264beeffecd593be"} err="failed to get container status \"6f8a74af7c300932373f5973ace89ca43d6806bc45da9f4a264beeffecd593be\": rpc error: code = NotFound desc = could not find container \"6f8a74af7c300932373f5973ace89ca43d6806bc45da9f4a264beeffecd593be\": container with ID starting with 6f8a74af7c300932373f5973ace89ca43d6806bc45da9f4a264beeffecd593be not found: ID does not exist" Oct 30 17:08:36.344364 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:36.344238 5052 scope.go:122] "RemoveContainer" containerID="9b128d66989128941673582c034464e89983bf1d1394b7b2a189515dc22f85b0" Oct 30 17:08:36.389409 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:36.389329 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-e5bfb0-86457d6567-v7zt5"] Oct 30 17:08:36.456541 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:36.453699 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-e5bfb0-86457d6567-v7zt5"] Oct 30 17:08:36.584604 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:36.579036 5052 scope.go:122] "RemoveContainer" containerID="9b128d66989128941673582c034464e89983bf1d1394b7b2a189515dc22f85b0" Oct 30 17:08:36.584604 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:08:36.579877 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b128d66989128941673582c034464e89983bf1d1394b7b2a189515dc22f85b0\": container with ID starting with 9b128d66989128941673582c034464e89983bf1d1394b7b2a189515dc22f85b0 not found: ID does not exist" containerID="9b128d66989128941673582c034464e89983bf1d1394b7b2a189515dc22f85b0" Oct 30 17:08:36.584604 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:36.579928 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b128d66989128941673582c034464e89983bf1d1394b7b2a189515dc22f85b0"} err="failed to get container status \"9b128d66989128941673582c034464e89983bf1d1394b7b2a189515dc22f85b0\": rpc error: code = NotFound desc = could not find container \"9b128d66989128941673582c034464e89983bf1d1394b7b2a189515dc22f85b0\": container with ID starting with 9b128d66989128941673582c034464e89983bf1d1394b7b2a189515dc22f85b0 not found: ID does not exist" Oct 30 17:08:36.584604 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:36.580519 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-e5bfb0-6f84945dc8-s9w5w"] Oct 30 17:08:36.953632 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:36.948098 5052 patch_prober.go:29] interesting pod/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-cb57p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body= Oct 30 17:08:36.953632 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:36.948178 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-cb57p" podUID="e4036d90-5e91-477e-b81f-91c2dceee6c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" Oct 30 17:08:37.131610 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:37.124158 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72b342d0-d3cc-43c3-8013-ef235a934f3b" path="/var/lib/kubelet/pods/72b342d0-d3cc-43c3-8013-ef235a934f3b/volumes" Oct 30 17:08:37.131610 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:37.125656 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-e5bfb0-6f84945dc8-s9w5w"] Oct 30 17:08:37.498221 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:37.498188 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-d693d0/distinct-2-w6-d693d0-kserve-6498fdc474-dpbjj"] Oct 30 17:08:37.889568 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:37.889001 5052 patch_prober.go:29] interesting pod/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-cb57p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body= Oct 30 17:08:37.889568 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:37.889082 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-cb57p" podUID="e4036d90-5e91-477e-b81f-91c2dceee6c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" Oct 30 17:08:38.600707 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:38.600672 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="731de2b2-ab6b-499b-a12e-21b7a550ea86" path="/var/lib/kubelet/pods/731de2b2-ab6b-499b-a12e-21b7a550ea86/volumes" Oct 30 17:08:38.892557 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:38.892482 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-cb57p" Oct 30 17:08:38.892994 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:38.892977 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-cb57p" Oct 30 17:08:38.893818 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:38.893795 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-cb57p" Oct 30 17:08:44.313817 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:44.275532 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e1e830-5ff5d8674b-clctx"] Oct 30 17:08:44.337603 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:44.335545 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e1e830-6dff758855-k2x7x"] Oct 30 17:08:45.929753 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:45.929713 5052 patch_prober.go:29] interesting pod/e2e-ipp-b-e1e830-openshift-default-5949c998b5-8zw7j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:08:45.930160 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:45.929765 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-e1e830-openshift-default-5949c998b5-8zw7j" podUID="91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:08:48.841884 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:08:48.841840 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 17:08:49.276673 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:49.276636 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-e1e830-5ff5d8674b-clctx" podUID="ef9859ce-d7fd-47cb-b62a-a6c8d605c188" containerName="payload-processing" containerID="cri-o://d1d19569d03bdb504a7d3cf065ffc9bc53c1fdb3879883be093d74cb8f783b80" gracePeriod=25 Oct 30 17:08:49.347533 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:49.336546 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e1e830-6dff758855-k2x7x" podUID="3860a637-7f6c-4b17-ba89-a2f039d081df" containerName="payload-pre-processing" containerID="cri-o://c1d2ee99a7a77087cfb20b8bf09781e83ad0f81bcbc83f3e396df65328d96fe3" gracePeriod=25 Oct 30 17:08:49.646083 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:49.639623 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e1e830-5ff5d8674b-clctx" Oct 30 17:08:49.674571 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:49.674545 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e1e830-6dff758855-k2x7x" Oct 30 17:08:49.751883 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:49.751851 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef9859ce-d7fd-47cb-b62a-a6c8d605c188-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef9859ce-d7fd-47cb-b62a-a6c8d605c188-plugins-config-volume\") pod \"ef9859ce-d7fd-47cb-b62a-a6c8d605c188\" (UID: \"ef9859ce-d7fd-47cb-b62a-a6c8d605c188\") " Oct 30 17:08:49.752085 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:49.751932 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3860a637-7f6c-4b17-ba89-a2f039d081df-kube-api-access-k6v8s\" (UniqueName: \"kubernetes.io/projected/3860a637-7f6c-4b17-ba89-a2f039d081df-kube-api-access-k6v8s\") pod \"3860a637-7f6c-4b17-ba89-a2f039d081df\" (UID: \"3860a637-7f6c-4b17-ba89-a2f039d081df\") " Oct 30 17:08:49.752085 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:49.751995 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef9859ce-d7fd-47cb-b62a-a6c8d605c188-kube-api-access-d8vwb\" (UniqueName: \"kubernetes.io/projected/ef9859ce-d7fd-47cb-b62a-a6c8d605c188-kube-api-access-d8vwb\") pod \"ef9859ce-d7fd-47cb-b62a-a6c8d605c188\" (UID: \"ef9859ce-d7fd-47cb-b62a-a6c8d605c188\") " Oct 30 17:08:49.752221 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:49.752117 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3860a637-7f6c-4b17-ba89-a2f039d081df-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3860a637-7f6c-4b17-ba89-a2f039d081df-plugins-config-volume\") pod \"3860a637-7f6c-4b17-ba89-a2f039d081df\" (UID: \"3860a637-7f6c-4b17-ba89-a2f039d081df\") " Oct 30 17:08:49.752337 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:49.752307 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef9859ce-d7fd-47cb-b62a-a6c8d605c188-plugins-config-volume" pod "ef9859ce-d7fd-47cb-b62a-a6c8d605c188" (UID: "ef9859ce-d7fd-47cb-b62a-a6c8d605c188"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:08:49.752811 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:49.752787 5052 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef9859ce-d7fd-47cb-b62a-a6c8d605c188-plugins-config-volume\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:08:49.752945 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:49.752857 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3860a637-7f6c-4b17-ba89-a2f039d081df-plugins-config-volume" pod "3860a637-7f6c-4b17-ba89-a2f039d081df" (UID: "3860a637-7f6c-4b17-ba89-a2f039d081df"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:08:49.754891 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:49.754867 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3860a637-7f6c-4b17-ba89-a2f039d081df-kube-api-access-k6v8s" pod "3860a637-7f6c-4b17-ba89-a2f039d081df" (UID: "3860a637-7f6c-4b17-ba89-a2f039d081df"). InnerVolumeSpecName "kube-api-access-k6v8s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:08:49.755197 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:49.755184 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef9859ce-d7fd-47cb-b62a-a6c8d605c188-kube-api-access-d8vwb" pod "ef9859ce-d7fd-47cb-b62a-a6c8d605c188" (UID: "ef9859ce-d7fd-47cb-b62a-a6c8d605c188"). InnerVolumeSpecName "kube-api-access-d8vwb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:08:49.853884 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:49.853807 5052 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3860a637-7f6c-4b17-ba89-a2f039d081df-plugins-config-volume\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:08:49.853884 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:49.853836 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6v8s\" (UniqueName: \"kubernetes.io/projected/3860a637-7f6c-4b17-ba89-a2f039d081df-kube-api-access-k6v8s\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:08:49.853884 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:49.853845 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8vwb\" (UniqueName: \"kubernetes.io/projected/ef9859ce-d7fd-47cb-b62a-a6c8d605c188-kube-api-access-d8vwb\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:08:50.317719 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:50.317655 5052 generic.go:356] "Generic (PLEG): container finished" podID="3860a637-7f6c-4b17-ba89-a2f039d081df" containerID="c1d2ee99a7a77087cfb20b8bf09781e83ad0f81bcbc83f3e396df65328d96fe3" exitCode=0 Oct 30 17:08:50.317719 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:50.317700 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e1e830-6dff758855-k2x7x" Oct 30 17:08:50.317943 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:50.317738 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e1e830-6dff758855-k2x7x" event={"ID":"3860a637-7f6c-4b17-ba89-a2f039d081df","Type":"ContainerDied","Data":"c1d2ee99a7a77087cfb20b8bf09781e83ad0f81bcbc83f3e396df65328d96fe3"} Oct 30 17:08:50.317943 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:50.317790 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e1e830-6dff758855-k2x7x" event={"ID":"3860a637-7f6c-4b17-ba89-a2f039d081df","Type":"ContainerDied","Data":"39579a4e13f10cbff29d3954044b178cd139396c4b2a988e995770b6f35e7cbf"} Oct 30 17:08:50.317943 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:50.317818 5052 scope.go:122] "RemoveContainer" containerID="c1d2ee99a7a77087cfb20b8bf09781e83ad0f81bcbc83f3e396df65328d96fe3" Oct 30 17:08:50.319285 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:50.319259 5052 generic.go:356] "Generic (PLEG): container finished" podID="ef9859ce-d7fd-47cb-b62a-a6c8d605c188" containerID="d1d19569d03bdb504a7d3cf065ffc9bc53c1fdb3879883be093d74cb8f783b80" exitCode=0 Oct 30 17:08:50.319419 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:50.319296 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e1e830-5ff5d8674b-clctx" Oct 30 17:08:50.319419 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:50.319308 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e1e830-5ff5d8674b-clctx" event={"ID":"ef9859ce-d7fd-47cb-b62a-a6c8d605c188","Type":"ContainerDied","Data":"d1d19569d03bdb504a7d3cf065ffc9bc53c1fdb3879883be093d74cb8f783b80"} Oct 30 17:08:50.319419 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:50.319363 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e1e830-5ff5d8674b-clctx" event={"ID":"ef9859ce-d7fd-47cb-b62a-a6c8d605c188","Type":"ContainerDied","Data":"27ffc4a156b734480cfa326e04114c1427b656b6f1b827e366d237e03c1859c8"} Oct 30 17:08:50.321433 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:50.321411 5052 generic.go:356] "Generic (PLEG): container finished" podID="91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a" containerID="607b3811a04d0437b68e538055a6316e57a386c63ffe58f35f134c1cf8009053" exitCode=0 Oct 30 17:08:50.321551 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:50.321463 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e1e830-openshift-default-5949c998b5-8zw7j" event={"ID":"91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a","Type":"ContainerDied","Data":"607b3811a04d0437b68e538055a6316e57a386c63ffe58f35f134c1cf8009053"} Oct 30 17:08:50.321891 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:50.321874 5052 scope.go:122] "RemoveContainer" containerID="607b3811a04d0437b68e538055a6316e57a386c63ffe58f35f134c1cf8009053" Oct 30 17:08:50.324034 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:50.324005 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:08:50.324111 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:50.324096 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:08:50.324161 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:50.324146 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:08:50.331174 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:50.331160 5052 scope.go:122] "RemoveContainer" containerID="c1d2ee99a7a77087cfb20b8bf09781e83ad0f81bcbc83f3e396df65328d96fe3" Oct 30 17:08:50.331471 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:08:50.331447 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1d2ee99a7a77087cfb20b8bf09781e83ad0f81bcbc83f3e396df65328d96fe3\": container with ID starting with c1d2ee99a7a77087cfb20b8bf09781e83ad0f81bcbc83f3e396df65328d96fe3 not found: ID does not exist" containerID="c1d2ee99a7a77087cfb20b8bf09781e83ad0f81bcbc83f3e396df65328d96fe3" Oct 30 17:08:50.331530 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:50.331483 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1d2ee99a7a77087cfb20b8bf09781e83ad0f81bcbc83f3e396df65328d96fe3"} err="failed to get container status \"c1d2ee99a7a77087cfb20b8bf09781e83ad0f81bcbc83f3e396df65328d96fe3\": rpc error: code = NotFound desc = could not find container \"c1d2ee99a7a77087cfb20b8bf09781e83ad0f81bcbc83f3e396df65328d96fe3\": container with ID starting with c1d2ee99a7a77087cfb20b8bf09781e83ad0f81bcbc83f3e396df65328d96fe3 not found: ID does not exist" Oct 30 17:08:50.331530 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:50.331505 5052 scope.go:122] "RemoveContainer" containerID="d1d19569d03bdb504a7d3cf065ffc9bc53c1fdb3879883be093d74cb8f783b80" Oct 30 17:08:50.341644 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:50.341628 5052 scope.go:122] "RemoveContainer" containerID="d1d19569d03bdb504a7d3cf065ffc9bc53c1fdb3879883be093d74cb8f783b80" Oct 30 17:08:50.341921 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:08:50.341902 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1d19569d03bdb504a7d3cf065ffc9bc53c1fdb3879883be093d74cb8f783b80\": container with ID starting with d1d19569d03bdb504a7d3cf065ffc9bc53c1fdb3879883be093d74cb8f783b80 not found: ID does not exist" containerID="d1d19569d03bdb504a7d3cf065ffc9bc53c1fdb3879883be093d74cb8f783b80" Oct 30 17:08:50.341973 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:50.341929 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1d19569d03bdb504a7d3cf065ffc9bc53c1fdb3879883be093d74cb8f783b80"} err="failed to get container status \"d1d19569d03bdb504a7d3cf065ffc9bc53c1fdb3879883be093d74cb8f783b80\": rpc error: code = NotFound desc = could not find container \"d1d19569d03bdb504a7d3cf065ffc9bc53c1fdb3879883be093d74cb8f783b80\": container with ID starting with d1d19569d03bdb504a7d3cf065ffc9bc53c1fdb3879883be093d74cb8f783b80 not found: ID does not exist" Oct 30 17:08:50.383296 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:50.383274 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e1e830-5ff5d8674b-clctx"] Oct 30 17:08:50.387629 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:50.387609 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e1e830-5ff5d8674b-clctx"] Oct 30 17:08:50.475748 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:50.475710 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e1e830-6dff758855-k2x7x"] Oct 30 17:08:50.499773 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:50.496837 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e1e830-6dff758855-k2x7x"] Oct 30 17:08:50.600814 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:50.600720 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3860a637-7f6c-4b17-ba89-a2f039d081df" path="/var/lib/kubelet/pods/3860a637-7f6c-4b17-ba89-a2f039d081df/volumes" Oct 30 17:08:50.601230 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:50.601211 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef9859ce-d7fd-47cb-b62a-a6c8d605c188" path="/var/lib/kubelet/pods/ef9859ce-d7fd-47cb-b62a-a6c8d605c188/volumes" Oct 30 17:08:51.332211 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:51.332176 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e1e830-openshift-default-5949c998b5-8zw7j" event={"ID":"91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a","Type":"ContainerStarted","Data":"a3a1c5e9354a9f5efe4de2dd6e65a05a0413dbd42e6cc0bb5e13b7e7ff6f115d"} Oct 30 17:08:52.001050 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:52.001014 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-e1e830-openshift-default-5949c998b5-8zw7j" Oct 30 17:08:52.001251 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:52.001169 5052 patch_prober.go:29] interesting pod/e2e-ipp-b-e1e830-openshift-default-5949c998b5-8zw7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body= Oct 30 17:08:52.001251 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:52.001224 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e1e830-openshift-default-5949c998b5-8zw7j" podUID="91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" Oct 30 17:08:52.522468 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:52.522410 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-d693d0/distinct-2-w6-d693d0-kserve-6498fdc474-dpbjj" podUID="4b98b8ff-72cd-4ddb-a4b8-67078162e611" containerName="main" containerID="cri-o://b03a9b24cafd8486dcb27b366c6f9a29cd35a431dee7b63a90e5706373940915" gracePeriod=45 Oct 30 17:08:52.912918 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:52.912884 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-d693d0/distinct-2-w6-d693d0-kserve-6498fdc474-dpbjj" Oct 30 17:08:53.002285 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:53.002248 5052 patch_prober.go:29] interesting pod/e2e-ipp-b-e1e830-openshift-default-5949c998b5-8zw7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body= Oct 30 17:08:53.002455 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:53.002311 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e1e830-openshift-default-5949c998b5-8zw7j" podUID="91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" Oct 30 17:08:53.016022 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:53.015900 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b98b8ff-72cd-4ddb-a4b8-67078162e611-home\" (UniqueName: \"kubernetes.io/empty-dir/4b98b8ff-72cd-4ddb-a4b8-67078162e611-home\") pod \"4b98b8ff-72cd-4ddb-a4b8-67078162e611\" (UID: \"4b98b8ff-72cd-4ddb-a4b8-67078162e611\") " Oct 30 17:08:53.016914 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:53.016169 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b98b8ff-72cd-4ddb-a4b8-67078162e611-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b98b8ff-72cd-4ddb-a4b8-67078162e611-tmp-dir\") pod \"4b98b8ff-72cd-4ddb-a4b8-67078162e611\" (UID: \"4b98b8ff-72cd-4ddb-a4b8-67078162e611\") " Oct 30 17:08:53.016914 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:53.016228 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b98b8ff-72cd-4ddb-a4b8-67078162e611-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b98b8ff-72cd-4ddb-a4b8-67078162e611-model-cache\") pod \"4b98b8ff-72cd-4ddb-a4b8-67078162e611\" (UID: \"4b98b8ff-72cd-4ddb-a4b8-67078162e611\") " Oct 30 17:08:53.016914 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:53.016261 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b98b8ff-72cd-4ddb-a4b8-67078162e611-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b98b8ff-72cd-4ddb-a4b8-67078162e611-dshm\") pod \"4b98b8ff-72cd-4ddb-a4b8-67078162e611\" (UID: \"4b98b8ff-72cd-4ddb-a4b8-67078162e611\") " Oct 30 17:08:53.016914 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:53.016353 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b98b8ff-72cd-4ddb-a4b8-67078162e611-kube-api-access-xzqt2\" (UniqueName: \"kubernetes.io/projected/4b98b8ff-72cd-4ddb-a4b8-67078162e611-kube-api-access-xzqt2\") pod \"4b98b8ff-72cd-4ddb-a4b8-67078162e611\" (UID: \"4b98b8ff-72cd-4ddb-a4b8-67078162e611\") " Oct 30 17:08:53.016914 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:53.016442 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b98b8ff-72cd-4ddb-a4b8-67078162e611-tls-certs\" (UniqueName: \"kubernetes.io/secret/4b98b8ff-72cd-4ddb-a4b8-67078162e611-tls-certs\") pod \"4b98b8ff-72cd-4ddb-a4b8-67078162e611\" (UID: \"4b98b8ff-72cd-4ddb-a4b8-67078162e611\") " Oct 30 17:08:53.016914 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:53.016537 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b98b8ff-72cd-4ddb-a4b8-67078162e611-home" pod "4b98b8ff-72cd-4ddb-a4b8-67078162e611" (UID: "4b98b8ff-72cd-4ddb-a4b8-67078162e611"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:08:53.017323 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:53.017226 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b98b8ff-72cd-4ddb-a4b8-67078162e611-model-cache" pod "4b98b8ff-72cd-4ddb-a4b8-67078162e611" (UID: "4b98b8ff-72cd-4ddb-a4b8-67078162e611"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:08:53.017427 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:53.017402 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b98b8ff-72cd-4ddb-a4b8-67078162e611-tmp-dir" pod "4b98b8ff-72cd-4ddb-a4b8-67078162e611" (UID: "4b98b8ff-72cd-4ddb-a4b8-67078162e611"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:08:53.017497 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:53.017479 5052 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b98b8ff-72cd-4ddb-a4b8-67078162e611-home\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:08:53.017497 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:53.017492 5052 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b98b8ff-72cd-4ddb-a4b8-67078162e611-model-cache\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:08:53.020507 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:53.020212 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b98b8ff-72cd-4ddb-a4b8-67078162e611-tls-certs" pod "4b98b8ff-72cd-4ddb-a4b8-67078162e611" (UID: "4b98b8ff-72cd-4ddb-a4b8-67078162e611"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:08:53.020507 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:53.020426 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b98b8ff-72cd-4ddb-a4b8-67078162e611-dshm" pod "4b98b8ff-72cd-4ddb-a4b8-67078162e611" (UID: "4b98b8ff-72cd-4ddb-a4b8-67078162e611"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:08:53.060170 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:53.060138 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b98b8ff-72cd-4ddb-a4b8-67078162e611-kube-api-access-xzqt2" pod "4b98b8ff-72cd-4ddb-a4b8-67078162e611" (UID: "4b98b8ff-72cd-4ddb-a4b8-67078162e611"). InnerVolumeSpecName "kube-api-access-xzqt2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:08:53.168657 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:53.142905 5052 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b98b8ff-72cd-4ddb-a4b8-67078162e611-tls-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:08:53.168657 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:53.142941 5052 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b98b8ff-72cd-4ddb-a4b8-67078162e611-tmp-dir\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:08:53.168657 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:53.142954 5052 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b98b8ff-72cd-4ddb-a4b8-67078162e611-dshm\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:08:53.168657 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:53.143024 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzqt2\" (UniqueName: \"kubernetes.io/projected/4b98b8ff-72cd-4ddb-a4b8-67078162e611-kube-api-access-xzqt2\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:08:53.378517 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:53.378485 5052 generic.go:356] "Generic (PLEG): container finished" podID="4b98b8ff-72cd-4ddb-a4b8-67078162e611" containerID="b03a9b24cafd8486dcb27b366c6f9a29cd35a431dee7b63a90e5706373940915" exitCode=0 Oct 30 17:08:53.378745 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:53.378543 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-d693d0/distinct-2-w6-d693d0-kserve-6498fdc474-dpbjj" Oct 30 17:08:53.378745 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:53.378629 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-d693d0/distinct-2-w6-d693d0-kserve-6498fdc474-dpbjj" event={"ID":"4b98b8ff-72cd-4ddb-a4b8-67078162e611","Type":"ContainerDied","Data":"b03a9b24cafd8486dcb27b366c6f9a29cd35a431dee7b63a90e5706373940915"} Oct 30 17:08:53.378745 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:53.378657 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-d693d0/distinct-2-w6-d693d0-kserve-6498fdc474-dpbjj" event={"ID":"4b98b8ff-72cd-4ddb-a4b8-67078162e611","Type":"ContainerDied","Data":"2ac477a7bd28303720ff3fcdff4e5036f0a74c2c96f6362e51462efc5b7365bd"} Oct 30 17:08:53.378745 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:53.378674 5052 scope.go:122] "RemoveContainer" containerID="b03a9b24cafd8486dcb27b366c6f9a29cd35a431dee7b63a90e5706373940915" Oct 30 17:08:53.395421 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:53.395402 5052 scope.go:122] "RemoveContainer" containerID="b03a9b24cafd8486dcb27b366c6f9a29cd35a431dee7b63a90e5706373940915" Oct 30 17:08:53.395854 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:08:53.395801 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b03a9b24cafd8486dcb27b366c6f9a29cd35a431dee7b63a90e5706373940915\": container with ID starting with b03a9b24cafd8486dcb27b366c6f9a29cd35a431dee7b63a90e5706373940915 not found: ID does not exist" containerID="b03a9b24cafd8486dcb27b366c6f9a29cd35a431dee7b63a90e5706373940915" Oct 30 17:08:53.395946 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:53.395865 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b03a9b24cafd8486dcb27b366c6f9a29cd35a431dee7b63a90e5706373940915"} err="failed to get container status \"b03a9b24cafd8486dcb27b366c6f9a29cd35a431dee7b63a90e5706373940915\": rpc error: code = NotFound desc = could not find container \"b03a9b24cafd8486dcb27b366c6f9a29cd35a431dee7b63a90e5706373940915\": container with ID starting with b03a9b24cafd8486dcb27b366c6f9a29cd35a431dee7b63a90e5706373940915 not found: ID does not exist" Oct 30 17:08:53.457530 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:53.455508 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-d693d0/distinct-2-w6-d693d0-kserve-6498fdc474-dpbjj"] Oct 30 17:08:53.464639 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:53.464563 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-d693d0/distinct-2-w6-d693d0-kserve-6498fdc474-dpbjj"] Oct 30 17:08:53.760725 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:53.760635 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d693d0/distinct-2-w6-d693d0-kserve-6498fdc474-dpbjj" podUID="4b98b8ff-72cd-4ddb-a4b8-67078162e611" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Oct 30 17:08:54.001891 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:54.001852 5052 patch_prober.go:29] interesting pod/e2e-ipp-b-e1e830-openshift-default-5949c998b5-8zw7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body= Oct 30 17:08:54.002071 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:54.001912 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e1e830-openshift-default-5949c998b5-8zw7j" podUID="91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" Oct 30 17:08:54.600992 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:54.600955 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b98b8ff-72cd-4ddb-a4b8-67078162e611" path="/var/lib/kubelet/pods/4b98b8ff-72cd-4ddb-a4b8-67078162e611/volumes" Oct 30 17:08:55.001204 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:55.001168 5052 patch_prober.go:29] interesting pod/e2e-ipp-b-e1e830-openshift-default-5949c998b5-8zw7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body= Oct 30 17:08:55.001746 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:55.001233 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e1e830-openshift-default-5949c998b5-8zw7j" podUID="91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" Oct 30 17:08:56.002227 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:56.002196 5052 patch_prober.go:29] interesting pod/e2e-ipp-b-e1e830-openshift-default-5949c998b5-8zw7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body= Oct 30 17:08:56.002653 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:56.002257 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e1e830-openshift-default-5949c998b5-8zw7j" podUID="91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" Oct 30 17:08:57.010622 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:57.010568 5052 patch_prober.go:29] interesting pod/e2e-ipp-b-e1e830-openshift-default-5949c998b5-8zw7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body= Oct 30 17:08:57.011309 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:57.011275 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e1e830-openshift-default-5949c998b5-8zw7j" podUID="91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" Oct 30 17:08:57.472556 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:57.472514 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9j9rt"] Oct 30 17:08:57.473645 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:57.473622 5052 cpu_manager.go:414] "containerMap: removing container" podUID="4b98b8ff-72cd-4ddb-a4b8-67078162e611" containerName="main" Oct 30 17:08:57.473728 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:57.473646 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b98b8ff-72cd-4ddb-a4b8-67078162e611" containerName="main" Oct 30 17:08:57.473728 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:57.473679 5052 cpu_manager.go:414] "containerMap: removing container" podUID="3860a637-7f6c-4b17-ba89-a2f039d081df" containerName="payload-pre-processing" Oct 30 17:08:57.473728 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:57.473684 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3860a637-7f6c-4b17-ba89-a2f039d081df" containerName="payload-pre-processing" Oct 30 17:08:57.473728 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:57.473716 5052 cpu_manager.go:414] "containerMap: removing container" podUID="731de2b2-ab6b-499b-a12e-21b7a550ea86" containerName="payload-processing" Oct 30 17:08:57.473728 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:57.473722 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="731de2b2-ab6b-499b-a12e-21b7a550ea86" containerName="payload-processing" Oct 30 17:08:57.473896 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:57.473731 5052 cpu_manager.go:414] "containerMap: removing container" podUID="ef9859ce-d7fd-47cb-b62a-a6c8d605c188" containerName="payload-processing" Oct 30 17:08:57.473896 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:57.473736 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef9859ce-d7fd-47cb-b62a-a6c8d605c188" containerName="payload-processing" Oct 30 17:08:57.473896 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:57.473748 5052 cpu_manager.go:414] "containerMap: removing container" podUID="72b342d0-d3cc-43c3-8013-ef235a934f3b" containerName="payload-pre-processing" Oct 30 17:08:57.473896 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:57.473753 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72b342d0-d3cc-43c3-8013-ef235a934f3b" containerName="payload-pre-processing" Oct 30 17:08:57.473896 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:57.473874 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="3860a637-7f6c-4b17-ba89-a2f039d081df" containerName="payload-pre-processing" Oct 30 17:08:57.473896 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:57.473887 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="731de2b2-ab6b-499b-a12e-21b7a550ea86" containerName="payload-processing" Oct 30 17:08:57.473896 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:57.473894 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef9859ce-d7fd-47cb-b62a-a6c8d605c188" containerName="payload-processing" Oct 30 17:08:57.474120 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:57.473902 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="72b342d0-d3cc-43c3-8013-ef235a934f3b" containerName="payload-pre-processing" Oct 30 17:08:57.474120 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:57.473910 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b98b8ff-72cd-4ddb-a4b8-67078162e611" containerName="main" Oct 30 17:08:57.479158 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:57.479129 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9j9rt" Oct 30 17:08:57.502815 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:57.502786 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9j9rt"] Oct 30 17:08:57.515041 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:57.515001 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9da694ce-2f3b-465d-abe2-675a039d38c0-credential-socket\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9j9rt\" (UID: \"9da694ce-2f3b-465d-abe2-675a039d38c0\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9j9rt" Oct 30 17:08:57.515218 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:57.515131 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9da694ce-2f3b-465d-abe2-675a039d38c0-istio-data\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9j9rt\" (UID: \"9da694ce-2f3b-465d-abe2-675a039d38c0\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9j9rt" Oct 30 17:08:57.515289 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:57.515251 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9da694ce-2f3b-465d-abe2-675a039d38c0-workload-certs\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9j9rt\" (UID: \"9da694ce-2f3b-465d-abe2-675a039d38c0\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9j9rt" Oct 30 17:08:57.515348 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:57.515299 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9da694ce-2f3b-465d-abe2-675a039d38c0-workload-socket\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9j9rt\" (UID: \"9da694ce-2f3b-465d-abe2-675a039d38c0\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9j9rt" Oct 30 17:08:57.515456 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:57.515374 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9da694ce-2f3b-465d-abe2-675a039d38c0-istio-token\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9j9rt\" (UID: \"9da694ce-2f3b-465d-abe2-675a039d38c0\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9j9rt" Oct 30 17:08:57.515621 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:57.515594 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9da694ce-2f3b-465d-abe2-675a039d38c0-istiod-ca-cert\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9j9rt\" (UID: \"9da694ce-2f3b-465d-abe2-675a039d38c0\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9j9rt" Oct 30 17:08:57.515749 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:57.515728 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9da694ce-2f3b-465d-abe2-675a039d38c0-istio-envoy\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9j9rt\" (UID: \"9da694ce-2f3b-465d-abe2-675a039d38c0\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9j9rt" Oct 30 17:08:57.515822 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:57.515768 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9da694ce-2f3b-465d-abe2-675a039d38c0-istio-podinfo\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9j9rt\" (UID: \"9da694ce-2f3b-465d-abe2-675a039d38c0\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9j9rt" Oct 30 17:08:57.515822 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:57.515814 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nsbx\" (UniqueName: \"kubernetes.io/projected/9da694ce-2f3b-465d-abe2-675a039d38c0-kube-api-access-4nsbx\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9j9rt\" (UID: \"9da694ce-2f3b-465d-abe2-675a039d38c0\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9j9rt" Oct 30 17:08:57.617310 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:57.617270 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9da694ce-2f3b-465d-abe2-675a039d38c0-istio-envoy\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9j9rt\" (UID: \"9da694ce-2f3b-465d-abe2-675a039d38c0\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9j9rt" Oct 30 17:08:57.617310 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:57.617316 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9da694ce-2f3b-465d-abe2-675a039d38c0-istio-podinfo\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9j9rt\" (UID: \"9da694ce-2f3b-465d-abe2-675a039d38c0\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9j9rt" Oct 30 17:08:57.617535 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:57.617345 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nsbx\" (UniqueName: \"kubernetes.io/projected/9da694ce-2f3b-465d-abe2-675a039d38c0-kube-api-access-4nsbx\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9j9rt\" (UID: \"9da694ce-2f3b-465d-abe2-675a039d38c0\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9j9rt" Oct 30 17:08:57.617535 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:57.617425 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9da694ce-2f3b-465d-abe2-675a039d38c0-credential-socket\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9j9rt\" (UID: \"9da694ce-2f3b-465d-abe2-675a039d38c0\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9j9rt" Oct 30 17:08:57.617535 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:57.617487 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9da694ce-2f3b-465d-abe2-675a039d38c0-istio-data\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9j9rt\" (UID: \"9da694ce-2f3b-465d-abe2-675a039d38c0\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9j9rt" Oct 30 17:08:57.617752 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:57.617551 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9da694ce-2f3b-465d-abe2-675a039d38c0-workload-certs\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9j9rt\" (UID: \"9da694ce-2f3b-465d-abe2-675a039d38c0\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9j9rt" Oct 30 17:08:57.617752 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:57.617603 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9da694ce-2f3b-465d-abe2-675a039d38c0-workload-socket\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9j9rt\" (UID: \"9da694ce-2f3b-465d-abe2-675a039d38c0\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9j9rt" Oct 30 17:08:57.617752 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:57.617635 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9da694ce-2f3b-465d-abe2-675a039d38c0-istio-token\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9j9rt\" (UID: \"9da694ce-2f3b-465d-abe2-675a039d38c0\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9j9rt" Oct 30 17:08:57.617928 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:57.617814 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9da694ce-2f3b-465d-abe2-675a039d38c0-istiod-ca-cert\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9j9rt\" (UID: \"9da694ce-2f3b-465d-abe2-675a039d38c0\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9j9rt" Oct 30 17:08:57.617928 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:57.617851 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9da694ce-2f3b-465d-abe2-675a039d38c0-credential-socket\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9j9rt\" (UID: \"9da694ce-2f3b-465d-abe2-675a039d38c0\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9j9rt" Oct 30 17:08:57.618038 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:57.617951 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9da694ce-2f3b-465d-abe2-675a039d38c0-istio-data\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9j9rt\" (UID: \"9da694ce-2f3b-465d-abe2-675a039d38c0\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9j9rt" Oct 30 17:08:57.618038 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:57.618017 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9da694ce-2f3b-465d-abe2-675a039d38c0-workload-certs\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9j9rt\" (UID: \"9da694ce-2f3b-465d-abe2-675a039d38c0\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9j9rt" Oct 30 17:08:57.618139 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:57.618098 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9da694ce-2f3b-465d-abe2-675a039d38c0-workload-socket\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9j9rt\" (UID: \"9da694ce-2f3b-465d-abe2-675a039d38c0\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9j9rt" Oct 30 17:08:57.618968 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:57.618942 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9da694ce-2f3b-465d-abe2-675a039d38c0-istiod-ca-cert\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9j9rt\" (UID: \"9da694ce-2f3b-465d-abe2-675a039d38c0\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9j9rt" Oct 30 17:08:57.619899 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:57.619879 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9da694ce-2f3b-465d-abe2-675a039d38c0-istio-envoy\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9j9rt\" (UID: \"9da694ce-2f3b-465d-abe2-675a039d38c0\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9j9rt" Oct 30 17:08:57.620449 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:57.620426 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9da694ce-2f3b-465d-abe2-675a039d38c0-istio-podinfo\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9j9rt\" (UID: \"9da694ce-2f3b-465d-abe2-675a039d38c0\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9j9rt" Oct 30 17:08:57.645066 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:57.645031 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9da694ce-2f3b-465d-abe2-675a039d38c0-istio-token\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9j9rt\" (UID: \"9da694ce-2f3b-465d-abe2-675a039d38c0\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9j9rt" Oct 30 17:08:57.645279 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:57.645254 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nsbx\" (UniqueName: \"kubernetes.io/projected/9da694ce-2f3b-465d-abe2-675a039d38c0-kube-api-access-4nsbx\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9j9rt\" (UID: \"9da694ce-2f3b-465d-abe2-675a039d38c0\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9j9rt" Oct 30 17:08:57.793978 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:57.793881 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9j9rt" Oct 30 17:08:58.006257 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:58.006223 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-e1e830-openshift-default-5949c998b5-8zw7j" Oct 30 17:08:58.006723 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:58.006697 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-e1e830-openshift-default-5949c998b5-8zw7j" Oct 30 17:08:58.007336 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:58.007310 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-e1e830-openshift-default-5949c998b5-8zw7j" Oct 30 17:08:58.141378 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:58.141347 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9j9rt"] Oct 30 17:08:58.144306 ip-10-0-131-136 kubenswrapper[5052]: W1030 17:08:58.144266 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9da694ce_2f3b_465d_abe2_675a039d38c0.slice/crio-02f89ac4ab1b757f4c36cd9ea3d44cc02e3e6f12d742b11cef98349979901ed2 WatchSource:0}: Error finding container 02f89ac4ab1b757f4c36cd9ea3d44cc02e3e6f12d742b11cef98349979901ed2: Status 404 returned error can't find the container with id 02f89ac4ab1b757f4c36cd9ea3d44cc02e3e6f12d742b11cef98349979901ed2 Oct 30 17:08:58.148224 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:58.148170 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:08:58.148493 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:58.148464 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:08:58.148727 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:58.148701 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:08:58.533329 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:58.533297 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9j9rt" event={"ID":"9da694ce-2f3b-465d-abe2-675a039d38c0","Type":"ContainerStarted","Data":"1f57818b297e08c26696f799e8d76487f06c074e4e83db6584c2d4206632b1b0"} Oct 30 17:08:58.533526 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:58.533346 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9j9rt" event={"ID":"9da694ce-2f3b-465d-abe2-675a039d38c0","Type":"ContainerStarted","Data":"02f89ac4ab1b757f4c36cd9ea3d44cc02e3e6f12d742b11cef98349979901ed2"} Oct 30 17:08:58.583772 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:58.583721 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9j9rt" podStartSLOduration=1.583707441 podStartE2EDuration="1.583707441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:08:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:08:58.580334926 +0000 UTC m=+2894.107670159" watchObservedRunningTime="2026-10-30 17:08:58.583707441 +0000 UTC m=+2894.111042659" Oct 30 17:08:58.794381 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:58.794305 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9j9rt" Oct 30 17:08:59.795833 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:59.795803 5052 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9j9rt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Oct 30 17:08:59.796253 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:08:59.795853 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9j9rt" podUID="9da694ce-2f3b-465d-abe2-675a039d38c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Oct 30 17:09:00.794375 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:00.794338 5052 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9j9rt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Oct 30 17:09:00.794565 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:00.794393 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9j9rt" podUID="9da694ce-2f3b-465d-abe2-675a039d38c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Oct 30 17:09:01.809602 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:01.795935 5052 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9j9rt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Oct 30 17:09:01.809602 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:01.796002 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9j9rt" podUID="9da694ce-2f3b-465d-abe2-675a039d38c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Oct 30 17:09:02.883024 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:02.881228 5052 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9j9rt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Oct 30 17:09:02.883024 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:02.881291 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9j9rt" podUID="9da694ce-2f3b-465d-abe2-675a039d38c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Oct 30 17:09:03.794757 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:03.794716 5052 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9j9rt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Oct 30 17:09:03.794942 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:03.794772 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9j9rt" podUID="9da694ce-2f3b-465d-abe2-675a039d38c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Oct 30 17:09:04.799874 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:04.799834 5052 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9j9rt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Oct 30 17:09:04.800466 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:04.800433 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9j9rt" podUID="9da694ce-2f3b-465d-abe2-675a039d38c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Oct 30 17:09:05.799236 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:05.799203 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9j9rt" Oct 30 17:09:05.799687 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:05.799673 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9j9rt" Oct 30 17:09:05.800279 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:05.800257 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9j9rt" Oct 30 17:09:18.842317 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:09:18.842275 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 17:09:19.234309 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:19.234271 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-97a47d-84c9dc8988-jss9k"] Oct 30 17:09:19.239888 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:19.239865 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-97a47d-84c9dc8988-jss9k" Oct 30 17:09:19.280865 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:19.280835 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-97a47d-serving-cert\"" Oct 30 17:09:19.284259 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:19.284230 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-97a47d-84c9dc8988-jss9k"] Oct 30 17:09:19.325792 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:19.325758 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlw5v\" (UniqueName: \"kubernetes.io/projected/3b174c28-3805-4b14-9547-cd003db3d27e-kube-api-access-jlw5v\") pod \"maas-api-e2e-ipp-b-97a47d-84c9dc8988-jss9k\" (UID: \"3b174c28-3805-4b14-9547-cd003db3d27e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-97a47d-84c9dc8988-jss9k" Oct 30 17:09:19.326016 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:19.325805 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b174c28-3805-4b14-9547-cd003db3d27e-maas-api-tls\") pod \"maas-api-e2e-ipp-b-97a47d-84c9dc8988-jss9k\" (UID: \"3b174c28-3805-4b14-9547-cd003db3d27e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-97a47d-84c9dc8988-jss9k" Oct 30 17:09:19.327344 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:19.327315 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b174c28-3805-4b14-9547-cd003db3d27e-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-97a47d-84c9dc8988-jss9k\" (UID: \"3b174c28-3805-4b14-9547-cd003db3d27e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-97a47d-84c9dc8988-jss9k" Oct 30 17:09:19.327648 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:19.327628 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b174c28-3805-4b14-9547-cd003db3d27e-metrics-certs\") pod \"maas-api-e2e-ipp-b-97a47d-84c9dc8988-jss9k\" (UID: \"3b174c28-3805-4b14-9547-cd003db3d27e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-97a47d-84c9dc8988-jss9k" Oct 30 17:09:19.429318 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:19.429285 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b174c28-3805-4b14-9547-cd003db3d27e-metrics-certs\") pod \"maas-api-e2e-ipp-b-97a47d-84c9dc8988-jss9k\" (UID: \"3b174c28-3805-4b14-9547-cd003db3d27e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-97a47d-84c9dc8988-jss9k" Oct 30 17:09:19.429513 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:19.429396 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlw5v\" (UniqueName: \"kubernetes.io/projected/3b174c28-3805-4b14-9547-cd003db3d27e-kube-api-access-jlw5v\") pod \"maas-api-e2e-ipp-b-97a47d-84c9dc8988-jss9k\" (UID: \"3b174c28-3805-4b14-9547-cd003db3d27e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-97a47d-84c9dc8988-jss9k" Oct 30 17:09:19.429513 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:19.429414 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b174c28-3805-4b14-9547-cd003db3d27e-maas-api-tls\") pod \"maas-api-e2e-ipp-b-97a47d-84c9dc8988-jss9k\" (UID: \"3b174c28-3805-4b14-9547-cd003db3d27e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-97a47d-84c9dc8988-jss9k" Oct 30 17:09:19.429513 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:19.429443 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b174c28-3805-4b14-9547-cd003db3d27e-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-97a47d-84c9dc8988-jss9k\" (UID: \"3b174c28-3805-4b14-9547-cd003db3d27e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-97a47d-84c9dc8988-jss9k" Oct 30 17:09:19.430119 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:19.430096 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b174c28-3805-4b14-9547-cd003db3d27e-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-97a47d-84c9dc8988-jss9k\" (UID: \"3b174c28-3805-4b14-9547-cd003db3d27e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-97a47d-84c9dc8988-jss9k" Oct 30 17:09:19.431975 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:19.431950 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b174c28-3805-4b14-9547-cd003db3d27e-maas-api-tls\") pod \"maas-api-e2e-ipp-b-97a47d-84c9dc8988-jss9k\" (UID: \"3b174c28-3805-4b14-9547-cd003db3d27e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-97a47d-84c9dc8988-jss9k" Oct 30 17:09:19.432074 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:19.431953 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b174c28-3805-4b14-9547-cd003db3d27e-metrics-certs\") pod \"maas-api-e2e-ipp-b-97a47d-84c9dc8988-jss9k\" (UID: \"3b174c28-3805-4b14-9547-cd003db3d27e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-97a47d-84c9dc8988-jss9k" Oct 30 17:09:19.454172 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:19.454135 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlw5v\" (UniqueName: \"kubernetes.io/projected/3b174c28-3805-4b14-9547-cd003db3d27e-kube-api-access-jlw5v\") pod \"maas-api-e2e-ipp-b-97a47d-84c9dc8988-jss9k\" (UID: \"3b174c28-3805-4b14-9547-cd003db3d27e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-97a47d-84c9dc8988-jss9k" Oct 30 17:09:19.557611 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:19.556734 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-97a47d-84c9dc8988-jss9k" Oct 30 17:09:19.886427 ip-10-0-131-136 kubenswrapper[5052]: W1030 17:09:19.886396 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b174c28_3805_4b14_9547_cd003db3d27e.slice/crio-921ac23ddbef8c711078fdd4ba723c8b9f21f75677a3162f54b9a13fcb93497b WatchSource:0}: Error finding container 921ac23ddbef8c711078fdd4ba723c8b9f21f75677a3162f54b9a13fcb93497b: Status 404 returned error can't find the container with id 921ac23ddbef8c711078fdd4ba723c8b9f21f75677a3162f54b9a13fcb93497b Oct 30 17:09:19.886853 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:19.886700 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-97a47d-84c9dc8988-jss9k"] Oct 30 17:09:19.887842 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:19.887821 5052 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 17:09:19.958497 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:19.958471 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-97a47d-84c9dc8988-jss9k" event={"ID":"3b174c28-3805-4b14-9547-cd003db3d27e","Type":"ContainerStarted","Data":"921ac23ddbef8c711078fdd4ba723c8b9f21f75677a3162f54b9a13fcb93497b"} Oct 30 17:09:20.969843 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:20.969801 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-97a47d-84c9dc8988-jss9k" event={"ID":"3b174c28-3805-4b14-9547-cd003db3d27e","Type":"ContainerStarted","Data":"7dc247faa20fc01daf58d491375d944528dce23733f12c04efc9701136592eaa"} Oct 30 17:09:20.970242 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:20.969932 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-97a47d-84c9dc8988-jss9k" Oct 30 17:09:21.014542 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:21.014163 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-97a47d-84c9dc8988-jss9k" podStartSLOduration=1.6061946809999998 podStartE2EDuration="2.014145462s" totalImagesPullingTime="407.950781ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:09:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:09:21.013784025 +0000 UTC m=+2916.541119272" watchObservedRunningTime="2026-10-30 17:09:21.014145462 +0000 UTC m=+2916.541480659" Oct 30 17:09:24.863736 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:24.863703 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-bfecdd/sub-test-model-bfecdd-kserve-d8464b565-47v4j"] Oct 30 17:09:26.984169 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:26.984133 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-97a47d-84c9dc8988-jss9k" Oct 30 17:09:30.953782 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:30.953731 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e4048c-d969759b4-c9knx"] Oct 30 17:09:30.973891 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:30.973859 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e4048c-5765848847-mg8pn"] Oct 30 17:09:35.954751 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:35.954705 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-e4048c-d969759b4-c9knx" podUID="95b7602a-63c6-4dd8-9f9f-198a1866ce43" containerName="payload-processing" containerID="cri-o://28d5dccc444218f7a3652ee04d2dd340dcfac5e68069543fefc5ced6a4799d35" gracePeriod=25 Oct 30 17:09:35.974846 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:35.974798 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e4048c-5765848847-mg8pn" podUID="8decf675-2bb4-431f-9ab6-da639afd4f55" containerName="payload-pre-processing" containerID="cri-o://4913bc65d1350df1ba0477c59d333376921e4ac9be591f82a22b50fec687a004" gracePeriod=25 Oct 30 17:09:36.344959 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:36.344912 5052 generic.go:356] "Generic (PLEG): container finished" podID="8decf675-2bb4-431f-9ab6-da639afd4f55" containerID="4913bc65d1350df1ba0477c59d333376921e4ac9be591f82a22b50fec687a004" exitCode=0 Oct 30 17:09:36.345267 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:36.345233 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e4048c-5765848847-mg8pn" event={"ID":"8decf675-2bb4-431f-9ab6-da639afd4f55","Type":"ContainerDied","Data":"4913bc65d1350df1ba0477c59d333376921e4ac9be591f82a22b50fec687a004"} Oct 30 17:09:36.347812 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:36.347732 5052 generic.go:356] "Generic (PLEG): container finished" podID="95b7602a-63c6-4dd8-9f9f-198a1866ce43" containerID="28d5dccc444218f7a3652ee04d2dd340dcfac5e68069543fefc5ced6a4799d35" exitCode=0 Oct 30 17:09:36.347812 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:36.347769 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e4048c-d969759b4-c9knx" event={"ID":"95b7602a-63c6-4dd8-9f9f-198a1866ce43","Type":"ContainerDied","Data":"28d5dccc444218f7a3652ee04d2dd340dcfac5e68069543fefc5ced6a4799d35"} Oct 30 17:09:36.609644 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:36.609608 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e4048c-d969759b4-c9knx" Oct 30 17:09:36.682522 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:36.682489 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95b7602a-63c6-4dd8-9f9f-198a1866ce43-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95b7602a-63c6-4dd8-9f9f-198a1866ce43-plugins-config-volume\") pod \"95b7602a-63c6-4dd8-9f9f-198a1866ce43\" (UID: \"95b7602a-63c6-4dd8-9f9f-198a1866ce43\") " Oct 30 17:09:36.682928 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:36.682903 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95b7602a-63c6-4dd8-9f9f-198a1866ce43-kube-api-access-vn4gt\" (UniqueName: \"kubernetes.io/projected/95b7602a-63c6-4dd8-9f9f-198a1866ce43-kube-api-access-vn4gt\") pod \"95b7602a-63c6-4dd8-9f9f-198a1866ce43\" (UID: \"95b7602a-63c6-4dd8-9f9f-198a1866ce43\") " Oct 30 17:09:36.683042 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:36.682935 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95b7602a-63c6-4dd8-9f9f-198a1866ce43-plugins-config-volume" pod "95b7602a-63c6-4dd8-9f9f-198a1866ce43" (UID: "95b7602a-63c6-4dd8-9f9f-198a1866ce43"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:09:36.683672 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:36.683651 5052 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95b7602a-63c6-4dd8-9f9f-198a1866ce43-plugins-config-volume\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:09:36.748072 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:36.748023 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95b7602a-63c6-4dd8-9f9f-198a1866ce43-kube-api-access-vn4gt" pod "95b7602a-63c6-4dd8-9f9f-198a1866ce43" (UID: "95b7602a-63c6-4dd8-9f9f-198a1866ce43"). InnerVolumeSpecName "kube-api-access-vn4gt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:09:36.750072 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:36.750045 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e4048c-5765848847-mg8pn" Oct 30 17:09:36.785869 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:36.785804 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vn4gt\" (UniqueName: \"kubernetes.io/projected/95b7602a-63c6-4dd8-9f9f-198a1866ce43-kube-api-access-vn4gt\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:09:36.887248 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:36.887210 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8decf675-2bb4-431f-9ab6-da639afd4f55-kube-api-access-js6vs\" (UniqueName: \"kubernetes.io/projected/8decf675-2bb4-431f-9ab6-da639afd4f55-kube-api-access-js6vs\") pod \"8decf675-2bb4-431f-9ab6-da639afd4f55\" (UID: \"8decf675-2bb4-431f-9ab6-da639afd4f55\") " Oct 30 17:09:36.887466 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:36.887448 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8decf675-2bb4-431f-9ab6-da639afd4f55-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8decf675-2bb4-431f-9ab6-da639afd4f55-plugins-config-volume\") pod \"8decf675-2bb4-431f-9ab6-da639afd4f55\" (UID: \"8decf675-2bb4-431f-9ab6-da639afd4f55\") " Oct 30 17:09:36.887841 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:36.887815 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8decf675-2bb4-431f-9ab6-da639afd4f55-plugins-config-volume" pod "8decf675-2bb4-431f-9ab6-da639afd4f55" (UID: "8decf675-2bb4-431f-9ab6-da639afd4f55"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:09:36.888148 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:36.888127 5052 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8decf675-2bb4-431f-9ab6-da639afd4f55-plugins-config-volume\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:09:36.889494 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:36.889467 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8decf675-2bb4-431f-9ab6-da639afd4f55-kube-api-access-js6vs" pod "8decf675-2bb4-431f-9ab6-da639afd4f55" (UID: "8decf675-2bb4-431f-9ab6-da639afd4f55"). InnerVolumeSpecName "kube-api-access-js6vs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:09:36.990090 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:36.990053 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-js6vs\" (UniqueName: \"kubernetes.io/projected/8decf675-2bb4-431f-9ab6-da639afd4f55-kube-api-access-js6vs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:09:37.357689 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:37.357610 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e4048c-d969759b4-c9knx" event={"ID":"95b7602a-63c6-4dd8-9f9f-198a1866ce43","Type":"ContainerDied","Data":"e794a5c6350ca56177871cfa4712848ff4282359d2e598b38215cd4ac889a22e"} Oct 30 17:09:37.357872 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:37.357697 5052 scope.go:122] "RemoveContainer" containerID="28d5dccc444218f7a3652ee04d2dd340dcfac5e68069543fefc5ced6a4799d35" Oct 30 17:09:37.357872 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:37.357625 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e4048c-d969759b4-c9knx" Oct 30 17:09:37.359281 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:37.359263 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e4048c-5765848847-mg8pn" event={"ID":"8decf675-2bb4-431f-9ab6-da639afd4f55","Type":"ContainerDied","Data":"7280e380a795d765f2d349f87ce3b4621f9a1349fef47869b11c0dfeac9b5668"} Oct 30 17:09:37.359383 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:37.359277 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e4048c-5765848847-mg8pn" Oct 30 17:09:37.369417 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:37.369397 5052 scope.go:122] "RemoveContainer" containerID="4913bc65d1350df1ba0477c59d333376921e4ac9be591f82a22b50fec687a004" Oct 30 17:09:37.415771 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:37.415748 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e4048c-5765848847-mg8pn"] Oct 30 17:09:37.428367 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:37.428344 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e4048c-5765848847-mg8pn"] Oct 30 17:09:37.476426 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:37.476397 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e4048c-d969759b4-c9knx"] Oct 30 17:09:37.484567 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:37.484539 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e4048c-d969759b4-c9knx"] Oct 30 17:09:38.602520 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:38.602480 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8decf675-2bb4-431f-9ab6-da639afd4f55" path="/var/lib/kubelet/pods/8decf675-2bb4-431f-9ab6-da639afd4f55/volumes" Oct 30 17:09:38.603172 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:38.603155 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95b7602a-63c6-4dd8-9f9f-198a1866ce43" path="/var/lib/kubelet/pods/95b7602a-63c6-4dd8-9f9f-198a1866ce43/volumes" Oct 30 17:09:39.886086 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:39.885987 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-bfecdd/sub-test-model-bfecdd-kserve-d8464b565-47v4j" podUID="55216e71-08aa-4b36-912c-696f933beb4b" containerName="main" containerID="cri-o://f109efa855489d46514a93cee7023f8cdd868fd25a3363f72635fea8b20de7ec" gracePeriod=45 Oct 30 17:09:39.911520 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:39.911493 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-kgzxp"] Oct 30 17:09:39.912254 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:39.912239 5052 cpu_manager.go:414] "containerMap: removing container" podUID="95b7602a-63c6-4dd8-9f9f-198a1866ce43" containerName="payload-processing" Oct 30 17:09:39.912309 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:39.912256 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95b7602a-63c6-4dd8-9f9f-198a1866ce43" containerName="payload-processing" Oct 30 17:09:39.912309 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:39.912287 5052 cpu_manager.go:414] "containerMap: removing container" podUID="8decf675-2bb4-431f-9ab6-da639afd4f55" containerName="payload-pre-processing" Oct 30 17:09:39.912309 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:39.912292 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8decf675-2bb4-431f-9ab6-da639afd4f55" containerName="payload-pre-processing" Oct 30 17:09:39.912446 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:39.912436 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="95b7602a-63c6-4dd8-9f9f-198a1866ce43" containerName="payload-processing" Oct 30 17:09:39.912485 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:39.912450 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="8decf675-2bb4-431f-9ab6-da639afd4f55" containerName="payload-pre-processing" Oct 30 17:09:39.915988 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:39.915969 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-kgzxp" Oct 30 17:09:39.940343 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:39.940315 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-kgzxp"] Oct 30 17:09:39.947456 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:39.947419 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w0-bfecdd/sub-test-model-bfecdd-kserve-d8464b565-47v4j" podUID="55216e71-08aa-4b36-912c-696f933beb4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.193:8000/ready\": dial tcp 10.132.0.193:8000: connect: connection refused" Oct 30 17:09:40.053606 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:40.052707 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/025e9d83-1372-496c-b75e-4f5205d48dcd-credential-socket\") pod \"e2e-ipp-a-e4048c-openshift-default-5c764747c9-kgzxp\" (UID: \"025e9d83-1372-496c-b75e-4f5205d48dcd\") " pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-kgzxp" Oct 30 17:09:40.053606 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:40.052798 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/025e9d83-1372-496c-b75e-4f5205d48dcd-istio-data\") pod \"e2e-ipp-a-e4048c-openshift-default-5c764747c9-kgzxp\" (UID: \"025e9d83-1372-496c-b75e-4f5205d48dcd\") " pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-kgzxp" Oct 30 17:09:40.053606 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:40.052838 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/025e9d83-1372-496c-b75e-4f5205d48dcd-istio-podinfo\") pod \"e2e-ipp-a-e4048c-openshift-default-5c764747c9-kgzxp\" (UID: \"025e9d83-1372-496c-b75e-4f5205d48dcd\") " pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-kgzxp" Oct 30 17:09:40.053606 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:40.052866 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/025e9d83-1372-496c-b75e-4f5205d48dcd-istiod-ca-cert\") pod \"e2e-ipp-a-e4048c-openshift-default-5c764747c9-kgzxp\" (UID: \"025e9d83-1372-496c-b75e-4f5205d48dcd\") " pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-kgzxp" Oct 30 17:09:40.053606 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:40.052988 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72vht\" (UniqueName: \"kubernetes.io/projected/025e9d83-1372-496c-b75e-4f5205d48dcd-kube-api-access-72vht\") pod \"e2e-ipp-a-e4048c-openshift-default-5c764747c9-kgzxp\" (UID: \"025e9d83-1372-496c-b75e-4f5205d48dcd\") " pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-kgzxp" Oct 30 17:09:40.065518 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:40.065455 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/025e9d83-1372-496c-b75e-4f5205d48dcd-workload-socket\") pod \"e2e-ipp-a-e4048c-openshift-default-5c764747c9-kgzxp\" (UID: \"025e9d83-1372-496c-b75e-4f5205d48dcd\") " pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-kgzxp" Oct 30 17:09:40.065814 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:40.065780 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/025e9d83-1372-496c-b75e-4f5205d48dcd-workload-certs\") pod \"e2e-ipp-a-e4048c-openshift-default-5c764747c9-kgzxp\" (UID: \"025e9d83-1372-496c-b75e-4f5205d48dcd\") " pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-kgzxp" Oct 30 17:09:40.065908 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:40.065821 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/025e9d83-1372-496c-b75e-4f5205d48dcd-istio-envoy\") pod \"e2e-ipp-a-e4048c-openshift-default-5c764747c9-kgzxp\" (UID: \"025e9d83-1372-496c-b75e-4f5205d48dcd\") " pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-kgzxp" Oct 30 17:09:40.065908 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:40.065901 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/025e9d83-1372-496c-b75e-4f5205d48dcd-istio-token\") pod \"e2e-ipp-a-e4048c-openshift-default-5c764747c9-kgzxp\" (UID: \"025e9d83-1372-496c-b75e-4f5205d48dcd\") " pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-kgzxp" Oct 30 17:09:40.168175 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:40.168133 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/025e9d83-1372-496c-b75e-4f5205d48dcd-istio-envoy\") pod \"e2e-ipp-a-e4048c-openshift-default-5c764747c9-kgzxp\" (UID: \"025e9d83-1372-496c-b75e-4f5205d48dcd\") " pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-kgzxp" Oct 30 17:09:40.168367 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:40.168231 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/025e9d83-1372-496c-b75e-4f5205d48dcd-istio-token\") pod \"e2e-ipp-a-e4048c-openshift-default-5c764747c9-kgzxp\" (UID: \"025e9d83-1372-496c-b75e-4f5205d48dcd\") " pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-kgzxp" Oct 30 17:09:40.168367 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:40.168319 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/025e9d83-1372-496c-b75e-4f5205d48dcd-credential-socket\") pod \"e2e-ipp-a-e4048c-openshift-default-5c764747c9-kgzxp\" (UID: \"025e9d83-1372-496c-b75e-4f5205d48dcd\") " pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-kgzxp" Oct 30 17:09:40.168492 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:40.168385 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/025e9d83-1372-496c-b75e-4f5205d48dcd-istio-data\") pod \"e2e-ipp-a-e4048c-openshift-default-5c764747c9-kgzxp\" (UID: \"025e9d83-1372-496c-b75e-4f5205d48dcd\") " pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-kgzxp" Oct 30 17:09:40.168492 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:40.168424 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/025e9d83-1372-496c-b75e-4f5205d48dcd-istio-podinfo\") pod \"e2e-ipp-a-e4048c-openshift-default-5c764747c9-kgzxp\" (UID: \"025e9d83-1372-496c-b75e-4f5205d48dcd\") " pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-kgzxp" Oct 30 17:09:40.168647 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:40.168490 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/025e9d83-1372-496c-b75e-4f5205d48dcd-istiod-ca-cert\") pod \"e2e-ipp-a-e4048c-openshift-default-5c764747c9-kgzxp\" (UID: \"025e9d83-1372-496c-b75e-4f5205d48dcd\") " pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-kgzxp" Oct 30 17:09:40.168647 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:40.168626 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72vht\" (UniqueName: \"kubernetes.io/projected/025e9d83-1372-496c-b75e-4f5205d48dcd-kube-api-access-72vht\") pod \"e2e-ipp-a-e4048c-openshift-default-5c764747c9-kgzxp\" (UID: \"025e9d83-1372-496c-b75e-4f5205d48dcd\") " pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-kgzxp" Oct 30 17:09:40.168756 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:40.168674 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/025e9d83-1372-496c-b75e-4f5205d48dcd-workload-socket\") pod \"e2e-ipp-a-e4048c-openshift-default-5c764747c9-kgzxp\" (UID: \"025e9d83-1372-496c-b75e-4f5205d48dcd\") " pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-kgzxp" Oct 30 17:09:40.168894 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:40.168869 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/025e9d83-1372-496c-b75e-4f5205d48dcd-workload-certs\") pod \"e2e-ipp-a-e4048c-openshift-default-5c764747c9-kgzxp\" (UID: \"025e9d83-1372-496c-b75e-4f5205d48dcd\") " pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-kgzxp" Oct 30 17:09:40.169516 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:40.169287 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/025e9d83-1372-496c-b75e-4f5205d48dcd-istio-data\") pod \"e2e-ipp-a-e4048c-openshift-default-5c764747c9-kgzxp\" (UID: \"025e9d83-1372-496c-b75e-4f5205d48dcd\") " pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-kgzxp" Oct 30 17:09:40.169516 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:40.169329 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/025e9d83-1372-496c-b75e-4f5205d48dcd-workload-certs\") pod \"e2e-ipp-a-e4048c-openshift-default-5c764747c9-kgzxp\" (UID: \"025e9d83-1372-496c-b75e-4f5205d48dcd\") " pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-kgzxp" Oct 30 17:09:40.169516 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:40.169409 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/025e9d83-1372-496c-b75e-4f5205d48dcd-credential-socket\") pod \"e2e-ipp-a-e4048c-openshift-default-5c764747c9-kgzxp\" (UID: \"025e9d83-1372-496c-b75e-4f5205d48dcd\") " pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-kgzxp" Oct 30 17:09:40.169847 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:40.169689 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/025e9d83-1372-496c-b75e-4f5205d48dcd-workload-socket\") pod \"e2e-ipp-a-e4048c-openshift-default-5c764747c9-kgzxp\" (UID: \"025e9d83-1372-496c-b75e-4f5205d48dcd\") " pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-kgzxp" Oct 30 17:09:40.170124 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:40.170103 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/025e9d83-1372-496c-b75e-4f5205d48dcd-istiod-ca-cert\") pod \"e2e-ipp-a-e4048c-openshift-default-5c764747c9-kgzxp\" (UID: \"025e9d83-1372-496c-b75e-4f5205d48dcd\") " pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-kgzxp" Oct 30 17:09:40.172239 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:40.172216 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/025e9d83-1372-496c-b75e-4f5205d48dcd-istio-podinfo\") pod \"e2e-ipp-a-e4048c-openshift-default-5c764747c9-kgzxp\" (UID: \"025e9d83-1372-496c-b75e-4f5205d48dcd\") " pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-kgzxp" Oct 30 17:09:40.176215 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:40.176189 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/025e9d83-1372-496c-b75e-4f5205d48dcd-istio-envoy\") pod \"e2e-ipp-a-e4048c-openshift-default-5c764747c9-kgzxp\" (UID: \"025e9d83-1372-496c-b75e-4f5205d48dcd\") " pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-kgzxp" Oct 30 17:09:40.193953 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:40.193932 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/025e9d83-1372-496c-b75e-4f5205d48dcd-istio-token\") pod \"e2e-ipp-a-e4048c-openshift-default-5c764747c9-kgzxp\" (UID: \"025e9d83-1372-496c-b75e-4f5205d48dcd\") " pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-kgzxp" Oct 30 17:09:40.194478 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:40.194451 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72vht\" (UniqueName: \"kubernetes.io/projected/025e9d83-1372-496c-b75e-4f5205d48dcd-kube-api-access-72vht\") pod \"e2e-ipp-a-e4048c-openshift-default-5c764747c9-kgzxp\" (UID: \"025e9d83-1372-496c-b75e-4f5205d48dcd\") " pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-kgzxp" Oct 30 17:09:40.224953 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:40.224932 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-bfecdd/sub-test-model-bfecdd-kserve-d8464b565-47v4j" Oct 30 17:09:40.270894 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:40.270758 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-kgzxp" Oct 30 17:09:40.372871 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:40.371369 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55216e71-08aa-4b36-912c-696f933beb4b-home\" (UniqueName: \"kubernetes.io/empty-dir/55216e71-08aa-4b36-912c-696f933beb4b-home\") pod \"55216e71-08aa-4b36-912c-696f933beb4b\" (UID: \"55216e71-08aa-4b36-912c-696f933beb4b\") " Oct 30 17:09:40.372871 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:40.371522 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55216e71-08aa-4b36-912c-696f933beb4b-tls-certs\" (UniqueName: \"kubernetes.io/secret/55216e71-08aa-4b36-912c-696f933beb4b-tls-certs\") pod \"55216e71-08aa-4b36-912c-696f933beb4b\" (UID: \"55216e71-08aa-4b36-912c-696f933beb4b\") " Oct 30 17:09:40.372871 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:40.371555 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55216e71-08aa-4b36-912c-696f933beb4b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/55216e71-08aa-4b36-912c-696f933beb4b-dshm\") pod \"55216e71-08aa-4b36-912c-696f933beb4b\" (UID: \"55216e71-08aa-4b36-912c-696f933beb4b\") " Oct 30 17:09:40.372871 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:40.371596 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55216e71-08aa-4b36-912c-696f933beb4b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55216e71-08aa-4b36-912c-696f933beb4b-tmp-dir\") pod \"55216e71-08aa-4b36-912c-696f933beb4b\" (UID: \"55216e71-08aa-4b36-912c-696f933beb4b\") " Oct 30 17:09:40.372871 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:40.371643 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55216e71-08aa-4b36-912c-696f933beb4b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55216e71-08aa-4b36-912c-696f933beb4b-model-cache\") pod \"55216e71-08aa-4b36-912c-696f933beb4b\" (UID: \"55216e71-08aa-4b36-912c-696f933beb4b\") " Oct 30 17:09:40.372871 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:40.371665 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55216e71-08aa-4b36-912c-696f933beb4b-kube-api-access-l9b6w\" (UniqueName: \"kubernetes.io/projected/55216e71-08aa-4b36-912c-696f933beb4b-kube-api-access-l9b6w\") pod \"55216e71-08aa-4b36-912c-696f933beb4b\" (UID: \"55216e71-08aa-4b36-912c-696f933beb4b\") " Oct 30 17:09:40.373433 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:40.372966 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55216e71-08aa-4b36-912c-696f933beb4b-home" pod "55216e71-08aa-4b36-912c-696f933beb4b" (UID: "55216e71-08aa-4b36-912c-696f933beb4b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:09:40.373433 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:40.373109 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55216e71-08aa-4b36-912c-696f933beb4b-model-cache" pod "55216e71-08aa-4b36-912c-696f933beb4b" (UID: "55216e71-08aa-4b36-912c-696f933beb4b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:09:40.373433 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:40.373195 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55216e71-08aa-4b36-912c-696f933beb4b-tmp-dir" pod "55216e71-08aa-4b36-912c-696f933beb4b" (UID: "55216e71-08aa-4b36-912c-696f933beb4b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:09:40.375986 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:40.375957 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55216e71-08aa-4b36-912c-696f933beb4b-kube-api-access-l9b6w" pod "55216e71-08aa-4b36-912c-696f933beb4b" (UID: "55216e71-08aa-4b36-912c-696f933beb4b"). InnerVolumeSpecName "kube-api-access-l9b6w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:09:40.376762 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:40.376740 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55216e71-08aa-4b36-912c-696f933beb4b-dshm" pod "55216e71-08aa-4b36-912c-696f933beb4b" (UID: "55216e71-08aa-4b36-912c-696f933beb4b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:09:40.377366 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:40.377341 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55216e71-08aa-4b36-912c-696f933beb4b-tls-certs" pod "55216e71-08aa-4b36-912c-696f933beb4b" (UID: "55216e71-08aa-4b36-912c-696f933beb4b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:09:40.397127 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:40.397089 5052 generic.go:356] "Generic (PLEG): container finished" podID="55216e71-08aa-4b36-912c-696f933beb4b" containerID="f109efa855489d46514a93cee7023f8cdd868fd25a3363f72635fea8b20de7ec" exitCode=0 Oct 30 17:09:40.397319 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:40.397154 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-bfecdd/sub-test-model-bfecdd-kserve-d8464b565-47v4j" Oct 30 17:09:40.397319 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:40.397179 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-bfecdd/sub-test-model-bfecdd-kserve-d8464b565-47v4j" event={"ID":"55216e71-08aa-4b36-912c-696f933beb4b","Type":"ContainerDied","Data":"f109efa855489d46514a93cee7023f8cdd868fd25a3363f72635fea8b20de7ec"} Oct 30 17:09:40.397319 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:40.397233 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-bfecdd/sub-test-model-bfecdd-kserve-d8464b565-47v4j" event={"ID":"55216e71-08aa-4b36-912c-696f933beb4b","Type":"ContainerDied","Data":"da441f9cade97642529002db07de5b1e7e616e7265695b992da2725eda0e2d83"} Oct 30 17:09:40.397319 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:40.397257 5052 scope.go:122] "RemoveContainer" containerID="f109efa855489d46514a93cee7023f8cdd868fd25a3363f72635fea8b20de7ec" Oct 30 17:09:40.401213 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:40.401190 5052 generic.go:356] "Generic (PLEG): container finished" podID="1fb4a801-fd82-4036-ba89-79fdf32d3be3" containerID="8ecd743ed8132bc3c3146a7ea6e84cb5048848ec8e3982af6e37ab90b57a97a0" exitCode=0 Oct 30 17:09:40.401381 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:40.401361 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-6jzqj" event={"ID":"1fb4a801-fd82-4036-ba89-79fdf32d3be3","Type":"ContainerDied","Data":"8ecd743ed8132bc3c3146a7ea6e84cb5048848ec8e3982af6e37ab90b57a97a0"} Oct 30 17:09:40.401864 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:40.401845 5052 scope.go:122] "RemoveContainer" containerID="8ecd743ed8132bc3c3146a7ea6e84cb5048848ec8e3982af6e37ab90b57a97a0" Oct 30 17:09:40.403895 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:40.403866 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:09:40.403988 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:40.403959 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:09:40.404050 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:40.404010 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:09:40.413770 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:40.413744 5052 scope.go:122] "RemoveContainer" containerID="f109efa855489d46514a93cee7023f8cdd868fd25a3363f72635fea8b20de7ec" Oct 30 17:09:40.414074 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:09:40.414045 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f109efa855489d46514a93cee7023f8cdd868fd25a3363f72635fea8b20de7ec\": container with ID starting with f109efa855489d46514a93cee7023f8cdd868fd25a3363f72635fea8b20de7ec not found: ID does not exist" containerID="f109efa855489d46514a93cee7023f8cdd868fd25a3363f72635fea8b20de7ec" Oct 30 17:09:40.414181 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:40.414100 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f109efa855489d46514a93cee7023f8cdd868fd25a3363f72635fea8b20de7ec"} err="failed to get container status \"f109efa855489d46514a93cee7023f8cdd868fd25a3363f72635fea8b20de7ec\": rpc error: code = NotFound desc = could not find container \"f109efa855489d46514a93cee7023f8cdd868fd25a3363f72635fea8b20de7ec\": container with ID starting with f109efa855489d46514a93cee7023f8cdd868fd25a3363f72635fea8b20de7ec not found: ID does not exist" Oct 30 17:09:40.454644 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:40.454615 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-bfecdd/sub-test-model-bfecdd-kserve-d8464b565-47v4j"] Oct 30 17:09:40.459037 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:40.459011 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-bfecdd/sub-test-model-bfecdd-kserve-d8464b565-47v4j"] Oct 30 17:09:40.474537 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:40.474509 5052 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55216e71-08aa-4b36-912c-696f933beb4b-home\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:09:40.474537 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:40.474536 5052 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55216e71-08aa-4b36-912c-696f933beb4b-tls-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:09:40.474739 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:40.474545 5052 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55216e71-08aa-4b36-912c-696f933beb4b-dshm\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:09:40.474739 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:40.474554 5052 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55216e71-08aa-4b36-912c-696f933beb4b-tmp-dir\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:09:40.474739 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:40.474561 5052 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55216e71-08aa-4b36-912c-696f933beb4b-model-cache\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:09:40.474739 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:40.474571 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9b6w\" (UniqueName: \"kubernetes.io/projected/55216e71-08aa-4b36-912c-696f933beb4b-kube-api-access-l9b6w\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:09:40.632467 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:40.630519 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55216e71-08aa-4b36-912c-696f933beb4b" path="/var/lib/kubelet/pods/55216e71-08aa-4b36-912c-696f933beb4b/volumes" Oct 30 17:09:40.636378 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:40.635754 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-kgzxp"] Oct 30 17:09:40.636904 ip-10-0-131-136 kubenswrapper[5052]: W1030 17:09:40.636869 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod025e9d83_1372_496c_b75e_4f5205d48dcd.slice/crio-b3ba54dc2b2ab591ad8d57526daac4c0496b1174fafcd1dc10fe6f3699d01c22 WatchSource:0}: Error finding container b3ba54dc2b2ab591ad8d57526daac4c0496b1174fafcd1dc10fe6f3699d01c22: Status 404 returned error can't find the container with id b3ba54dc2b2ab591ad8d57526daac4c0496b1174fafcd1dc10fe6f3699d01c22 Oct 30 17:09:40.640446 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:40.640407 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:09:40.640507 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:40.640485 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:09:40.640547 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:40.640516 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:09:41.410599 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:41.410541 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-kgzxp" event={"ID":"025e9d83-1372-496c-b75e-4f5205d48dcd","Type":"ContainerStarted","Data":"538de255129a6d08fdad51382b6432e2f8d70e4102e68fef4706550d0780ae26"} Oct 30 17:09:41.411079 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:41.410617 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-kgzxp" event={"ID":"025e9d83-1372-496c-b75e-4f5205d48dcd","Type":"ContainerStarted","Data":"b3ba54dc2b2ab591ad8d57526daac4c0496b1174fafcd1dc10fe6f3699d01c22"} Oct 30 17:09:41.413620 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:41.413573 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-6jzqj" event={"ID":"1fb4a801-fd82-4036-ba89-79fdf32d3be3","Type":"ContainerStarted","Data":"b9e3fbca33f82af327ba9d2612b8bfb6b0763849081e591fa80f05e22433665a"} Oct 30 17:09:41.462670 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:41.462604 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-kgzxp" podStartSLOduration=2.462571414 podStartE2EDuration="2.462571414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:09:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:09:41.460026249 +0000 UTC m=+2936.987361495" watchObservedRunningTime="2026-10-30 17:09:41.462571414 +0000 UTC m=+2936.989906636" Oct 30 17:09:41.927298 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:41.927263 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-6jzqj" Oct 30 17:09:41.927748 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:41.927719 5052 patch_prober.go:29] interesting pod/e2e-ipp-a-e4048c-openshift-default-5c764747c9-6jzqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Oct 30 17:09:41.927978 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:41.927945 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-6jzqj" podUID="1fb4a801-fd82-4036-ba89-79fdf32d3be3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Oct 30 17:09:42.257778 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:42.257691 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-6jzqj" Oct 30 17:09:42.274549 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:42.274510 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-kgzxp" Oct 30 17:09:42.274924 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:42.274900 5052 patch_prober.go:29] interesting pod/e2e-ipp-a-e4048c-openshift-default-5c764747c9-kgzxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body= Oct 30 17:09:42.275078 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:42.275055 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-kgzxp" podUID="025e9d83-1372-496c-b75e-4f5205d48dcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" Oct 30 17:09:42.790608 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:42.781967 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cdwcj"] Oct 30 17:09:42.921603 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:42.921452 5052 patch_prober.go:29] interesting pod/e2e-ipp-a-e4048c-openshift-default-5c764747c9-6jzqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Oct 30 17:09:42.921603 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:42.921511 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-6jzqj" podUID="1fb4a801-fd82-4036-ba89-79fdf32d3be3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Oct 30 17:09:43.286607 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:43.284951 5052 patch_prober.go:29] interesting pod/e2e-ipp-a-e4048c-openshift-default-5c764747c9-kgzxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body= Oct 30 17:09:43.286607 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:43.285014 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-kgzxp" podUID="025e9d83-1372-496c-b75e-4f5205d48dcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" Oct 30 17:09:43.915992 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:43.915956 5052 patch_prober.go:29] interesting pod/e2e-ipp-a-e4048c-openshift-default-5c764747c9-6jzqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Oct 30 17:09:43.916815 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:43.916780 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-6jzqj" podUID="1fb4a801-fd82-4036-ba89-79fdf32d3be3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Oct 30 17:09:44.279668 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:44.279179 5052 patch_prober.go:29] interesting pod/e2e-ipp-a-e4048c-openshift-default-5c764747c9-kgzxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body= Oct 30 17:09:44.279668 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:44.279240 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-kgzxp" podUID="025e9d83-1372-496c-b75e-4f5205d48dcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" Oct 30 17:09:44.916605 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:44.915806 5052 patch_prober.go:29] interesting pod/e2e-ipp-a-e4048c-openshift-default-5c764747c9-6jzqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Oct 30 17:09:44.916605 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:44.915922 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-6jzqj" podUID="1fb4a801-fd82-4036-ba89-79fdf32d3be3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Oct 30 17:09:45.239795 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:45.239677 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/distinct-2-w4-263f00-kserve-8464897cf5-q8ck8" podUID="15ed43ef-5cdd-489a-a70e-063239822ff3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:09:45.282671 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:45.275073 5052 patch_prober.go:29] interesting pod/e2e-ipp-a-e4048c-openshift-default-5c764747c9-kgzxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body= Oct 30 17:09:45.282671 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:45.275133 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-kgzxp" podUID="025e9d83-1372-496c-b75e-4f5205d48dcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" Oct 30 17:09:45.726626 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:45.724663 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-de18d5-openshift-default-748c669-n82k6"] Oct 30 17:09:45.787621 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:45.785084 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/distinct-w5-967fac-kserve-69cbcccd8b-b85bn" podUID="8f6b9354-378f-4451-9ff7-be7f7fec12f2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:09:45.787621 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:45.785525 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-750dcb/distinct-w2-750dcb-kserve-86bbdfcbb-bpn77" podUID="c45190af-d795-4e13-8f2f-670b2c597658" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded" Oct 30 17:09:45.984708 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:45.960445 5052 patch_prober.go:29] interesting pod/e2e-ipp-a-e4048c-openshift-default-5c764747c9-6jzqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Oct 30 17:09:45.984708 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:45.960508 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-6jzqj" podUID="1fb4a801-fd82-4036-ba89-79fdf32d3be3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Oct 30 17:09:46.003611 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:45.991291 5052 cpu_manager.go:414] "containerMap: removing container" podUID="55216e71-08aa-4b36-912c-696f933beb4b" containerName="main" Oct 30 17:09:46.024996 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:46.024958 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55216e71-08aa-4b36-912c-696f933beb4b" containerName="main" Oct 30 17:09:46.033058 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:46.033024 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="55216e71-08aa-4b36-912c-696f933beb4b" containerName="main" Oct 30 17:09:46.218903 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:46.218870 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-de18d5-openshift-default-748c669-n82k6"] Oct 30 17:09:46.245665 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:46.220189 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-de18d5-openshift-default-748c669-n82k6" Oct 30 17:09:46.271605 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:46.251631 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/distinct-2-w4-263f00-kserve-8464897cf5-q8ck8" podUID="15ed43ef-5cdd-489a-a70e-063239822ff3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:09:46.271605 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:46.270412 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-263f00/distinct-2-w4-263f00-kserve-8464897cf5-q8ck8" Oct 30 17:09:46.313606 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:46.283399 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-de18d5-openshift-default-dockercfg-b6dkw\"" Oct 30 17:09:46.313896 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:46.290785 5052 patch_prober.go:29] interesting pod/e2e-ipp-a-e4048c-openshift-default-5c764747c9-kgzxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body= Oct 30 17:09:46.314123 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:46.314078 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-kgzxp" podUID="025e9d83-1372-496c-b75e-4f5205d48dcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" Oct 30 17:09:46.316764 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:46.316544 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/505e6f2b-634b-40ae-b5db-6ced8e1bf322-istio-data\") pod \"e2e-ipp-a-de18d5-openshift-default-748c669-n82k6\" (UID: \"505e6f2b-634b-40ae-b5db-6ced8e1bf322\") " pod="openshift-ingress/e2e-ipp-a-de18d5-openshift-default-748c669-n82k6" Oct 30 17:09:46.334513 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:46.334478 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nvgd\" (UniqueName: \"kubernetes.io/projected/505e6f2b-634b-40ae-b5db-6ced8e1bf322-kube-api-access-4nvgd\") pod \"e2e-ipp-a-de18d5-openshift-default-748c669-n82k6\" (UID: \"505e6f2b-634b-40ae-b5db-6ced8e1bf322\") " pod="openshift-ingress/e2e-ipp-a-de18d5-openshift-default-748c669-n82k6" Oct 30 17:09:46.335334 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:46.334913 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/505e6f2b-634b-40ae-b5db-6ced8e1bf322-workload-socket\") pod \"e2e-ipp-a-de18d5-openshift-default-748c669-n82k6\" (UID: \"505e6f2b-634b-40ae-b5db-6ced8e1bf322\") " pod="openshift-ingress/e2e-ipp-a-de18d5-openshift-default-748c669-n82k6" Oct 30 17:09:46.335534 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:46.335514 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/505e6f2b-634b-40ae-b5db-6ced8e1bf322-istio-envoy\") pod \"e2e-ipp-a-de18d5-openshift-default-748c669-n82k6\" (UID: \"505e6f2b-634b-40ae-b5db-6ced8e1bf322\") " pod="openshift-ingress/e2e-ipp-a-de18d5-openshift-default-748c669-n82k6" Oct 30 17:09:46.335696 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:46.335678 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/505e6f2b-634b-40ae-b5db-6ced8e1bf322-credential-socket\") pod \"e2e-ipp-a-de18d5-openshift-default-748c669-n82k6\" (UID: \"505e6f2b-634b-40ae-b5db-6ced8e1bf322\") " pod="openshift-ingress/e2e-ipp-a-de18d5-openshift-default-748c669-n82k6" Oct 30 17:09:46.335879 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:46.335862 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/505e6f2b-634b-40ae-b5db-6ced8e1bf322-workload-certs\") pod \"e2e-ipp-a-de18d5-openshift-default-748c669-n82k6\" (UID: \"505e6f2b-634b-40ae-b5db-6ced8e1bf322\") " pod="openshift-ingress/e2e-ipp-a-de18d5-openshift-default-748c669-n82k6" Oct 30 17:09:46.341189 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:46.335994 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/505e6f2b-634b-40ae-b5db-6ced8e1bf322-istio-token\") pod \"e2e-ipp-a-de18d5-openshift-default-748c669-n82k6\" (UID: \"505e6f2b-634b-40ae-b5db-6ced8e1bf322\") " pod="openshift-ingress/e2e-ipp-a-de18d5-openshift-default-748c669-n82k6" Oct 30 17:09:46.368754 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:46.363827 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/505e6f2b-634b-40ae-b5db-6ced8e1bf322-istio-podinfo\") pod \"e2e-ipp-a-de18d5-openshift-default-748c669-n82k6\" (UID: \"505e6f2b-634b-40ae-b5db-6ced8e1bf322\") " pod="openshift-ingress/e2e-ipp-a-de18d5-openshift-default-748c669-n82k6" Oct 30 17:09:46.368754 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:46.363896 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/505e6f2b-634b-40ae-b5db-6ced8e1bf322-istiod-ca-cert\") pod \"e2e-ipp-a-de18d5-openshift-default-748c669-n82k6\" (UID: \"505e6f2b-634b-40ae-b5db-6ced8e1bf322\") " pod="openshift-ingress/e2e-ipp-a-de18d5-openshift-default-748c669-n82k6" Oct 30 17:09:46.513658 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:46.495409 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/505e6f2b-634b-40ae-b5db-6ced8e1bf322-workload-certs\") pod \"e2e-ipp-a-de18d5-openshift-default-748c669-n82k6\" (UID: \"505e6f2b-634b-40ae-b5db-6ced8e1bf322\") " pod="openshift-ingress/e2e-ipp-a-de18d5-openshift-default-748c669-n82k6" Oct 30 17:09:46.513658 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:46.495472 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/505e6f2b-634b-40ae-b5db-6ced8e1bf322-istio-token\") pod \"e2e-ipp-a-de18d5-openshift-default-748c669-n82k6\" (UID: \"505e6f2b-634b-40ae-b5db-6ced8e1bf322\") " pod="openshift-ingress/e2e-ipp-a-de18d5-openshift-default-748c669-n82k6" Oct 30 17:09:46.513658 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:46.495570 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/505e6f2b-634b-40ae-b5db-6ced8e1bf322-istio-podinfo\") pod \"e2e-ipp-a-de18d5-openshift-default-748c669-n82k6\" (UID: \"505e6f2b-634b-40ae-b5db-6ced8e1bf322\") " pod="openshift-ingress/e2e-ipp-a-de18d5-openshift-default-748c669-n82k6" Oct 30 17:09:46.513658 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:46.495619 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/505e6f2b-634b-40ae-b5db-6ced8e1bf322-istiod-ca-cert\") pod \"e2e-ipp-a-de18d5-openshift-default-748c669-n82k6\" (UID: \"505e6f2b-634b-40ae-b5db-6ced8e1bf322\") " pod="openshift-ingress/e2e-ipp-a-de18d5-openshift-default-748c669-n82k6" Oct 30 17:09:46.513658 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:46.495684 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/505e6f2b-634b-40ae-b5db-6ced8e1bf322-istio-data\") pod \"e2e-ipp-a-de18d5-openshift-default-748c669-n82k6\" (UID: \"505e6f2b-634b-40ae-b5db-6ced8e1bf322\") " pod="openshift-ingress/e2e-ipp-a-de18d5-openshift-default-748c669-n82k6" Oct 30 17:09:46.513658 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:46.495836 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nvgd\" (UniqueName: \"kubernetes.io/projected/505e6f2b-634b-40ae-b5db-6ced8e1bf322-kube-api-access-4nvgd\") pod \"e2e-ipp-a-de18d5-openshift-default-748c669-n82k6\" (UID: \"505e6f2b-634b-40ae-b5db-6ced8e1bf322\") " pod="openshift-ingress/e2e-ipp-a-de18d5-openshift-default-748c669-n82k6" Oct 30 17:09:46.513658 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:46.496006 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/505e6f2b-634b-40ae-b5db-6ced8e1bf322-workload-socket\") pod \"e2e-ipp-a-de18d5-openshift-default-748c669-n82k6\" (UID: \"505e6f2b-634b-40ae-b5db-6ced8e1bf322\") " pod="openshift-ingress/e2e-ipp-a-de18d5-openshift-default-748c669-n82k6" Oct 30 17:09:46.513658 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:46.496058 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/505e6f2b-634b-40ae-b5db-6ced8e1bf322-istio-envoy\") pod \"e2e-ipp-a-de18d5-openshift-default-748c669-n82k6\" (UID: \"505e6f2b-634b-40ae-b5db-6ced8e1bf322\") " pod="openshift-ingress/e2e-ipp-a-de18d5-openshift-default-748c669-n82k6" Oct 30 17:09:46.513658 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:46.496096 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/505e6f2b-634b-40ae-b5db-6ced8e1bf322-credential-socket\") pod \"e2e-ipp-a-de18d5-openshift-default-748c669-n82k6\" (UID: \"505e6f2b-634b-40ae-b5db-6ced8e1bf322\") " pod="openshift-ingress/e2e-ipp-a-de18d5-openshift-default-748c669-n82k6" Oct 30 17:09:46.513658 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:46.496573 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/505e6f2b-634b-40ae-b5db-6ced8e1bf322-credential-socket\") pod \"e2e-ipp-a-de18d5-openshift-default-748c669-n82k6\" (UID: \"505e6f2b-634b-40ae-b5db-6ced8e1bf322\") " pod="openshift-ingress/e2e-ipp-a-de18d5-openshift-default-748c669-n82k6" Oct 30 17:09:46.513658 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:46.496913 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/505e6f2b-634b-40ae-b5db-6ced8e1bf322-workload-certs\") pod \"e2e-ipp-a-de18d5-openshift-default-748c669-n82k6\" (UID: \"505e6f2b-634b-40ae-b5db-6ced8e1bf322\") " pod="openshift-ingress/e2e-ipp-a-de18d5-openshift-default-748c669-n82k6" Oct 30 17:09:46.513658 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:46.501382 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/505e6f2b-634b-40ae-b5db-6ced8e1bf322-istiod-ca-cert\") pod \"e2e-ipp-a-de18d5-openshift-default-748c669-n82k6\" (UID: \"505e6f2b-634b-40ae-b5db-6ced8e1bf322\") " pod="openshift-ingress/e2e-ipp-a-de18d5-openshift-default-748c669-n82k6" Oct 30 17:09:46.513658 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:46.501948 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/505e6f2b-634b-40ae-b5db-6ced8e1bf322-istio-data\") pod \"e2e-ipp-a-de18d5-openshift-default-748c669-n82k6\" (UID: \"505e6f2b-634b-40ae-b5db-6ced8e1bf322\") " pod="openshift-ingress/e2e-ipp-a-de18d5-openshift-default-748c669-n82k6" Oct 30 17:09:46.513658 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:46.502191 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/505e6f2b-634b-40ae-b5db-6ced8e1bf322-workload-socket\") pod \"e2e-ipp-a-de18d5-openshift-default-748c669-n82k6\" (UID: \"505e6f2b-634b-40ae-b5db-6ced8e1bf322\") " pod="openshift-ingress/e2e-ipp-a-de18d5-openshift-default-748c669-n82k6" Oct 30 17:09:46.864657 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:46.838919 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx" podUID="bd662f3a-659f-477c-b8d5-94c64288c0c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 17:09:46.892604 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:46.866052 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/distinct-w5-967fac-kserve-69cbcccd8b-b85bn" podUID="8f6b9354-378f-4451-9ff7-be7f7fec12f2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:09:46.892604 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:46.867697 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-967fac/distinct-w5-967fac-kserve-69cbcccd8b-b85bn" Oct 30 17:09:46.892604 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:46.878515 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-750dcb/distinct-w2-750dcb-kserve-86bbdfcbb-bpn77" podUID="c45190af-d795-4e13-8f2f-670b2c597658" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 17:09:46.892604 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:46.889727 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-750dcb/distinct-w2-750dcb-kserve-86bbdfcbb-bpn77" Oct 30 17:09:46.918608 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:46.917347 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/505e6f2b-634b-40ae-b5db-6ced8e1bf322-istio-token\") pod \"e2e-ipp-a-de18d5-openshift-default-748c669-n82k6\" (UID: \"505e6f2b-634b-40ae-b5db-6ced8e1bf322\") " pod="openshift-ingress/e2e-ipp-a-de18d5-openshift-default-748c669-n82k6" Oct 30 17:09:46.934613 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:46.931096 5052 patch_prober.go:29] interesting pod/e2e-ipp-a-e4048c-openshift-default-5c764747c9-6jzqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Oct 30 17:09:46.934613 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:46.931157 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-6jzqj" podUID="1fb4a801-fd82-4036-ba89-79fdf32d3be3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Oct 30 17:09:46.978332 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:46.978287 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/505e6f2b-634b-40ae-b5db-6ced8e1bf322-istio-envoy\") pod \"e2e-ipp-a-de18d5-openshift-default-748c669-n82k6\" (UID: \"505e6f2b-634b-40ae-b5db-6ced8e1bf322\") " pod="openshift-ingress/e2e-ipp-a-de18d5-openshift-default-748c669-n82k6" Oct 30 17:09:46.993320 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:46.993275 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/505e6f2b-634b-40ae-b5db-6ced8e1bf322-istio-podinfo\") pod \"e2e-ipp-a-de18d5-openshift-default-748c669-n82k6\" (UID: \"505e6f2b-634b-40ae-b5db-6ced8e1bf322\") " pod="openshift-ingress/e2e-ipp-a-de18d5-openshift-default-748c669-n82k6" Oct 30 17:09:47.039221 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:47.035302 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nvgd\" (UniqueName: \"kubernetes.io/projected/505e6f2b-634b-40ae-b5db-6ced8e1bf322-kube-api-access-4nvgd\") pod \"e2e-ipp-a-de18d5-openshift-default-748c669-n82k6\" (UID: \"505e6f2b-634b-40ae-b5db-6ced8e1bf322\") " pod="openshift-ingress/e2e-ipp-a-de18d5-openshift-default-748c669-n82k6" Oct 30 17:09:47.051824 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:47.051510 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-de18d5-openshift-default-748c669-n82k6" Oct 30 17:09:47.292606 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:47.254535 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/distinct-2-w4-263f00-kserve-8464897cf5-q8ck8" podUID="15ed43ef-5cdd-489a-a70e-063239822ff3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:09:48.031610 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:47.465689 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-750dcb/distinct-w2-750dcb-kserve-86bbdfcbb-bpn77" Oct 30 17:09:48.046755 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:47.655189 5052 patch_prober.go:29] interesting pod/e2e-ipp-a-e4048c-openshift-default-5c764747c9-kgzxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body= Oct 30 17:09:48.047042 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:48.047012 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-kgzxp" podUID="025e9d83-1372-496c-b75e-4f5205d48dcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" Oct 30 17:09:48.047173 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:47.877810 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/distinct-w5-967fac-kserve-69cbcccd8b-b85bn" podUID="8f6b9354-378f-4451-9ff7-be7f7fec12f2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Oct 30 17:09:48.047602 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:47.856738 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx" podUID="bd662f3a-659f-477c-b8d5-94c64288c0c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:09:48.048825 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:48.028860 5052 patch_prober.go:29] interesting pod/e2e-ipp-a-e4048c-openshift-default-5c764747c9-6jzqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Oct 30 17:09:48.048990 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:48.048966 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-6jzqj" podUID="1fb4a801-fd82-4036-ba89-79fdf32d3be3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Oct 30 17:09:48.049197 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:48.049137 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-967fac/distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx" Oct 30 17:09:48.390974 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:48.390883 5052 patch_prober.go:29] interesting pod/e2e-ipp-a-e4048c-openshift-default-5c764747c9-kgzxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body= Oct 30 17:09:48.391472 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:48.391437 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-kgzxp" podUID="025e9d83-1372-496c-b75e-4f5205d48dcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" Oct 30 17:09:49.036608 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:49.033486 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/distinct-2-w4-263f00-kserve-8464897cf5-q8ck8" podUID="15ed43ef-5cdd-489a-a70e-063239822ff3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 30 17:09:49.094610 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:49.051604 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx" podUID="bd662f3a-659f-477c-b8d5-94c64288c0c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 17:09:49.094610 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:49.055640 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/distinct-w5-967fac-kserve-69cbcccd8b-b85bn" podUID="8f6b9354-378f-4451-9ff7-be7f7fec12f2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Oct 30 17:09:49.094610 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:49.066609 5052 patch_prober.go:29] interesting pod/e2e-ipp-a-e4048c-openshift-default-5c764747c9-6jzqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Oct 30 17:09:49.094610 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:49.066657 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-6jzqj" podUID="1fb4a801-fd82-4036-ba89-79fdf32d3be3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Oct 30 17:09:49.421605 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:49.417963 5052 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-263f00/distinct-2-w4-263f00-kserve-8464897cf5-q8ck8" podUID="15ed43ef-5cdd-489a-a70e-063239822ff3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:09:49.559760 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:49.507192 5052 patch_prober.go:29] interesting pod/e2e-ipp-a-e4048c-openshift-default-5c764747c9-kgzxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body= Oct 30 17:09:49.559760 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:49.507261 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-kgzxp" podUID="025e9d83-1372-496c-b75e-4f5205d48dcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" Oct 30 17:09:49.559760 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:09:49.520418 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 17:09:49.775754 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:49.775693 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-750dcb/distinct-w2-750dcb-kserve-86bbdfcbb-bpn77" podUID="c45190af-d795-4e13-8f2f-670b2c597658" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:09:50.110681 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:50.039727 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/distinct-2-w4-263f00-kserve-8464897cf5-q8ck8" podUID="15ed43ef-5cdd-489a-a70e-063239822ff3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:09:50.165190 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:09:50.165132 5052 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.568s" Oct 30 17:09:50.170778 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:50.170127 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-6jzqj" Oct 30 17:09:50.171034 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:50.171015 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4ac125-9dcf56996-v4vjr"] Oct 30 17:09:50.171187 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:50.171175 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4ac125-84b8c9c875-7gtbm"] Oct 30 17:09:50.171332 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:50.171319 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-de18d5-openshift-default-748c669-n82k6"] Oct 30 17:09:50.172908 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:50.172890 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-967fac/distinct-w5-967fac-kserve-69cbcccd8b-b85bn" Oct 30 17:09:50.173055 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:50.173043 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-967fac/distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx" Oct 30 17:09:50.173195 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:50.173183 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-263f00/distinct-2-w4-263f00-kserve-8464897cf5-q8ck8" Oct 30 17:09:50.177807 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:50.177774 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:09:50.178502 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:50.178470 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:09:50.178742 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:50.178716 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:09:50.181809 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:50.181767 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-6jzqj" Oct 30 17:09:50.506257 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:50.505916 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-kgzxp" Oct 30 17:09:50.506907 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:50.506697 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-kgzxp" Oct 30 17:09:50.507939 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:50.507917 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-kgzxp" Oct 30 17:09:51.001069 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:51.001034 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-de18d5-openshift-default-748c669-n82k6" event={"ID":"505e6f2b-634b-40ae-b5db-6ced8e1bf322","Type":"ContainerStarted","Data":"0e8e12f2d38c3d2d9f6d112c53f873cd3f077945188e70180365ce297c14af94"} Oct 30 17:09:51.001313 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:51.001293 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-de18d5-openshift-default-748c669-n82k6" event={"ID":"505e6f2b-634b-40ae-b5db-6ced8e1bf322","Type":"ContainerStarted","Data":"b47d31d9166a135eb965f2a1ad98cac4101fac5729f9d138b3c6ed27079f5d57"} Oct 30 17:09:51.094531 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:51.094465 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-de18d5-openshift-default-748c669-n82k6" podStartSLOduration=6.094446545 podStartE2EDuration="6.094446545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:09:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:09:51.0939627 +0000 UTC m=+2946.621297923" watchObservedRunningTime="2026-10-30 17:09:51.094446545 +0000 UTC m=+2946.621781765" Oct 30 17:09:51.095370 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:51.095345 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-de18d5-openshift-default-748c669-n82k6" Oct 30 17:09:51.095637 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:51.095606 5052 patch_prober.go:29] interesting pod/e2e-ipp-a-de18d5-openshift-default-748c669-n82k6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body= Oct 30 17:09:51.095733 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:51.095661 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-de18d5-openshift-default-748c669-n82k6" podUID="505e6f2b-634b-40ae-b5db-6ced8e1bf322" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" Oct 30 17:09:52.097989 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:52.097957 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-de18d5-openshift-default-748c669-n82k6" Oct 30 17:09:53.019209 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:53.019179 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-de18d5-openshift-default-748c669-n82k6" Oct 30 17:09:53.020077 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:53.020056 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-de18d5-openshift-default-748c669-n82k6" Oct 30 17:09:55.175719 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:55.175666 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ac125-84b8c9c875-7gtbm" podUID="565bcd81-6055-4578-ae1a-628829ed7163" containerName="payload-pre-processing" containerID="cri-o://24edfa1ba57b5fa857c286285a96152769e94bba56816ccd75f7974cecf01420" gracePeriod=25 Oct 30 17:09:55.177692 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:55.177627 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-4ac125-9dcf56996-v4vjr" podUID="d0efd4aa-b83a-48d8-868e-be7d351899e2" containerName="payload-processing" containerID="cri-o://1ebf48a099ae97701dcbd8b9380c62a74822c1d0bfd47804d55764f4ebfd9abd" gracePeriod=25 Oct 30 17:09:55.234022 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:09:55.233980 5052 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0efd4aa_b83a_48d8_868e_be7d351899e2.slice/crio-1ebf48a099ae97701dcbd8b9380c62a74822c1d0bfd47804d55764f4ebfd9abd.scope\": RecentStats: unable to find data in memory cache]" Oct 30 17:09:55.343692 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:55.343657 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-de18d5-7cb866fd58-n9vgg"] Oct 30 17:09:55.421801 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:55.420533 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-de18d5-7cb866fd58-n9vgg"] Oct 30 17:09:55.421801 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:55.420608 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-de18d5-5fcc8b7f9f-sl9fh"] Oct 30 17:09:55.424064 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:55.423847 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-de18d5-7cb866fd58-n9vgg" Oct 30 17:09:55.480364 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:55.477344 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-de18d5-5fcc8b7f9f-sl9fh"] Oct 30 17:09:55.480364 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:55.477481 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-de18d5-5fcc8b7f9f-sl9fh" Oct 30 17:09:55.599091 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:55.587307 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-de18d5-dockercfg-bjws9\"" Oct 30 17:09:55.599091 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:55.587406 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-de18d5\"" Oct 30 17:09:55.599091 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:55.590337 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1da97b6c-928b-4f33-950e-065cf95c366e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-de18d5-5fcc8b7f9f-sl9fh\" (UID: \"1da97b6c-928b-4f33-950e-065cf95c366e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-de18d5-5fcc8b7f9f-sl9fh" Oct 30 17:09:55.599091 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:55.590452 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0c628c1-f21f-4a83-a007-98dc19cbf65b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-de18d5-7cb866fd58-n9vgg\" (UID: \"c0c628c1-f21f-4a83-a007-98dc19cbf65b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-de18d5-7cb866fd58-n9vgg" Oct 30 17:09:55.599091 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:55.590482 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4wx7\" (UniqueName: \"kubernetes.io/projected/c0c628c1-f21f-4a83-a007-98dc19cbf65b-kube-api-access-r4wx7\") pod \"payload-processing-e2e-ipp-a-de18d5-7cb866fd58-n9vgg\" (UID: \"c0c628c1-f21f-4a83-a007-98dc19cbf65b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-de18d5-7cb866fd58-n9vgg" Oct 30 17:09:55.599950 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:55.599537 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zzm8\" (UniqueName: \"kubernetes.io/projected/1da97b6c-928b-4f33-950e-065cf95c366e-kube-api-access-5zzm8\") pod \"payload-pre-processing-e2e-ipp-a-de18d5-5fcc8b7f9f-sl9fh\" (UID: \"1da97b6c-928b-4f33-950e-065cf95c366e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-de18d5-5fcc8b7f9f-sl9fh" Oct 30 17:09:55.725552 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:55.706552 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1da97b6c-928b-4f33-950e-065cf95c366e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-de18d5-5fcc8b7f9f-sl9fh\" (UID: \"1da97b6c-928b-4f33-950e-065cf95c366e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-de18d5-5fcc8b7f9f-sl9fh" Oct 30 17:09:55.732496 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:55.732430 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0c628c1-f21f-4a83-a007-98dc19cbf65b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-de18d5-7cb866fd58-n9vgg\" (UID: \"c0c628c1-f21f-4a83-a007-98dc19cbf65b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-de18d5-7cb866fd58-n9vgg" Oct 30 17:09:55.734185 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:55.734158 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4wx7\" (UniqueName: \"kubernetes.io/projected/c0c628c1-f21f-4a83-a007-98dc19cbf65b-kube-api-access-r4wx7\") pod \"payload-processing-e2e-ipp-a-de18d5-7cb866fd58-n9vgg\" (UID: \"c0c628c1-f21f-4a83-a007-98dc19cbf65b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-de18d5-7cb866fd58-n9vgg" Oct 30 17:09:55.734634 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:55.734569 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zzm8\" (UniqueName: \"kubernetes.io/projected/1da97b6c-928b-4f33-950e-065cf95c366e-kube-api-access-5zzm8\") pod \"payload-pre-processing-e2e-ipp-a-de18d5-5fcc8b7f9f-sl9fh\" (UID: \"1da97b6c-928b-4f33-950e-065cf95c366e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-de18d5-5fcc8b7f9f-sl9fh" Oct 30 17:09:55.735709 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:55.729301 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1da97b6c-928b-4f33-950e-065cf95c366e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-de18d5-5fcc8b7f9f-sl9fh\" (UID: \"1da97b6c-928b-4f33-950e-065cf95c366e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-de18d5-5fcc8b7f9f-sl9fh" Oct 30 17:09:55.759123 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:55.733969 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0c628c1-f21f-4a83-a007-98dc19cbf65b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-de18d5-7cb866fd58-n9vgg\" (UID: \"c0c628c1-f21f-4a83-a007-98dc19cbf65b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-de18d5-7cb866fd58-n9vgg" Oct 30 17:09:55.782504 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:55.782471 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zzm8\" (UniqueName: \"kubernetes.io/projected/1da97b6c-928b-4f33-950e-065cf95c366e-kube-api-access-5zzm8\") pod \"payload-pre-processing-e2e-ipp-a-de18d5-5fcc8b7f9f-sl9fh\" (UID: \"1da97b6c-928b-4f33-950e-065cf95c366e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-de18d5-5fcc8b7f9f-sl9fh" Oct 30 17:09:55.798745 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:55.798678 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4wx7\" (UniqueName: \"kubernetes.io/projected/c0c628c1-f21f-4a83-a007-98dc19cbf65b-kube-api-access-r4wx7\") pod \"payload-processing-e2e-ipp-a-de18d5-7cb866fd58-n9vgg\" (UID: \"c0c628c1-f21f-4a83-a007-98dc19cbf65b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-de18d5-7cb866fd58-n9vgg" Oct 30 17:09:55.954135 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:55.954083 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-de18d5-7cb866fd58-n9vgg" Oct 30 17:09:55.980370 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:55.980285 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ac125-84b8c9c875-7gtbm" Oct 30 17:09:56.019749 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:56.019719 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4ac125-9dcf56996-v4vjr" Oct 30 17:09:56.049869 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:56.049832 5052 generic.go:356] "Generic (PLEG): container finished" podID="565bcd81-6055-4578-ae1a-628829ed7163" containerID="24edfa1ba57b5fa857c286285a96152769e94bba56816ccd75f7974cecf01420" exitCode=0 Oct 30 17:09:56.050075 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:56.049884 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ac125-84b8c9c875-7gtbm" Oct 30 17:09:56.050075 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:56.049933 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ac125-84b8c9c875-7gtbm" event={"ID":"565bcd81-6055-4578-ae1a-628829ed7163","Type":"ContainerDied","Data":"24edfa1ba57b5fa857c286285a96152769e94bba56816ccd75f7974cecf01420"} Oct 30 17:09:56.050075 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:56.050047 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ac125-84b8c9c875-7gtbm" event={"ID":"565bcd81-6055-4578-ae1a-628829ed7163","Type":"ContainerDied","Data":"07bc9cfabc08e97a7655bd241a7b7b8b786efb42fc6c20d6e6d1fdb4c9bf06a4"} Oct 30 17:09:56.050221 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:56.050087 5052 scope.go:122] "RemoveContainer" containerID="24edfa1ba57b5fa857c286285a96152769e94bba56816ccd75f7974cecf01420" Oct 30 17:09:56.051687 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:56.051664 5052 generic.go:356] "Generic (PLEG): container finished" podID="d0efd4aa-b83a-48d8-868e-be7d351899e2" containerID="1ebf48a099ae97701dcbd8b9380c62a74822c1d0bfd47804d55764f4ebfd9abd" exitCode=0 Oct 30 17:09:56.051825 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:56.051719 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4ac125-9dcf56996-v4vjr" Oct 30 17:09:56.051825 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:56.051750 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4ac125-9dcf56996-v4vjr" event={"ID":"d0efd4aa-b83a-48d8-868e-be7d351899e2","Type":"ContainerDied","Data":"1ebf48a099ae97701dcbd8b9380c62a74822c1d0bfd47804d55764f4ebfd9abd"} Oct 30 17:09:56.051825 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:56.051810 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4ac125-9dcf56996-v4vjr" event={"ID":"d0efd4aa-b83a-48d8-868e-be7d351899e2","Type":"ContainerDied","Data":"6aa69ba44ab2a49249ce945980f95ea261c2b41241ea8a1d5b48d45b77179a95"} Oct 30 17:09:56.052365 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:56.052343 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/565bcd81-6055-4578-ae1a-628829ed7163-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/565bcd81-6055-4578-ae1a-628829ed7163-plugins-config-volume\") pod \"565bcd81-6055-4578-ae1a-628829ed7163\" (UID: \"565bcd81-6055-4578-ae1a-628829ed7163\") " Oct 30 17:09:56.052485 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:56.052432 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/565bcd81-6055-4578-ae1a-628829ed7163-kube-api-access-pqd8x\" (UniqueName: \"kubernetes.io/projected/565bcd81-6055-4578-ae1a-628829ed7163-kube-api-access-pqd8x\") pod \"565bcd81-6055-4578-ae1a-628829ed7163\" (UID: \"565bcd81-6055-4578-ae1a-628829ed7163\") " Oct 30 17:09:56.052647 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:56.052626 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0efd4aa-b83a-48d8-868e-be7d351899e2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0efd4aa-b83a-48d8-868e-be7d351899e2-plugins-config-volume\") pod \"d0efd4aa-b83a-48d8-868e-be7d351899e2\" (UID: \"d0efd4aa-b83a-48d8-868e-be7d351899e2\") " Oct 30 17:09:56.053021 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:56.052995 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0efd4aa-b83a-48d8-868e-be7d351899e2-kube-api-access-cxbdd\" (UniqueName: \"kubernetes.io/projected/d0efd4aa-b83a-48d8-868e-be7d351899e2-kube-api-access-cxbdd\") pod \"d0efd4aa-b83a-48d8-868e-be7d351899e2\" (UID: \"d0efd4aa-b83a-48d8-868e-be7d351899e2\") " Oct 30 17:09:56.072952 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:56.072918 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0efd4aa-b83a-48d8-868e-be7d351899e2-plugins-config-volume" pod "d0efd4aa-b83a-48d8-868e-be7d351899e2" (UID: "d0efd4aa-b83a-48d8-868e-be7d351899e2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:09:56.074897 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:56.074868 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/565bcd81-6055-4578-ae1a-628829ed7163-kube-api-access-pqd8x" pod "565bcd81-6055-4578-ae1a-628829ed7163" (UID: "565bcd81-6055-4578-ae1a-628829ed7163"). InnerVolumeSpecName "kube-api-access-pqd8x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:09:56.075020 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:56.074994 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-de18d5-5fcc8b7f9f-sl9fh" Oct 30 17:09:56.075081 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:56.075044 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0efd4aa-b83a-48d8-868e-be7d351899e2-kube-api-access-cxbdd" pod "d0efd4aa-b83a-48d8-868e-be7d351899e2" (UID: "d0efd4aa-b83a-48d8-868e-be7d351899e2"). InnerVolumeSpecName "kube-api-access-cxbdd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:09:56.078031 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:56.078008 5052 scope.go:122] "RemoveContainer" containerID="24edfa1ba57b5fa857c286285a96152769e94bba56816ccd75f7974cecf01420" Oct 30 17:09:56.078355 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:09:56.078335 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24edfa1ba57b5fa857c286285a96152769e94bba56816ccd75f7974cecf01420\": container with ID starting with 24edfa1ba57b5fa857c286285a96152769e94bba56816ccd75f7974cecf01420 not found: ID does not exist" containerID="24edfa1ba57b5fa857c286285a96152769e94bba56816ccd75f7974cecf01420" Oct 30 17:09:56.078450 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:56.078363 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24edfa1ba57b5fa857c286285a96152769e94bba56816ccd75f7974cecf01420"} err="failed to get container status \"24edfa1ba57b5fa857c286285a96152769e94bba56816ccd75f7974cecf01420\": rpc error: code = NotFound desc = could not find container \"24edfa1ba57b5fa857c286285a96152769e94bba56816ccd75f7974cecf01420\": container with ID starting with 24edfa1ba57b5fa857c286285a96152769e94bba56816ccd75f7974cecf01420 not found: ID does not exist" Oct 30 17:09:56.078450 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:56.078380 5052 scope.go:122] "RemoveContainer" containerID="1ebf48a099ae97701dcbd8b9380c62a74822c1d0bfd47804d55764f4ebfd9abd" Oct 30 17:09:56.092749 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:56.090970 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/565bcd81-6055-4578-ae1a-628829ed7163-plugins-config-volume" pod "565bcd81-6055-4578-ae1a-628829ed7163" (UID: "565bcd81-6055-4578-ae1a-628829ed7163"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:09:56.098525 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:56.098112 5052 scope.go:122] "RemoveContainer" containerID="1ebf48a099ae97701dcbd8b9380c62a74822c1d0bfd47804d55764f4ebfd9abd" Oct 30 17:09:56.098619 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:09:56.098525 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ebf48a099ae97701dcbd8b9380c62a74822c1d0bfd47804d55764f4ebfd9abd\": container with ID starting with 1ebf48a099ae97701dcbd8b9380c62a74822c1d0bfd47804d55764f4ebfd9abd not found: ID does not exist" containerID="1ebf48a099ae97701dcbd8b9380c62a74822c1d0bfd47804d55764f4ebfd9abd" Oct 30 17:09:56.098619 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:56.098559 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ebf48a099ae97701dcbd8b9380c62a74822c1d0bfd47804d55764f4ebfd9abd"} err="failed to get container status \"1ebf48a099ae97701dcbd8b9380c62a74822c1d0bfd47804d55764f4ebfd9abd\": rpc error: code = NotFound desc = could not find container \"1ebf48a099ae97701dcbd8b9380c62a74822c1d0bfd47804d55764f4ebfd9abd\": container with ID starting with 1ebf48a099ae97701dcbd8b9380c62a74822c1d0bfd47804d55764f4ebfd9abd not found: ID does not exist" Oct 30 17:09:56.157633 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:56.156608 5052 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0efd4aa-b83a-48d8-868e-be7d351899e2-plugins-config-volume\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:09:56.157633 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:56.156642 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxbdd\" (UniqueName: \"kubernetes.io/projected/d0efd4aa-b83a-48d8-868e-be7d351899e2-kube-api-access-cxbdd\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:09:56.157633 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:56.156683 5052 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/565bcd81-6055-4578-ae1a-628829ed7163-plugins-config-volume\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:09:56.157633 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:56.156698 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqd8x\" (UniqueName: \"kubernetes.io/projected/565bcd81-6055-4578-ae1a-628829ed7163-kube-api-access-pqd8x\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:09:56.407312 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:56.407274 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4ac125-84b8c9c875-7gtbm"] Oct 30 17:09:56.413852 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:56.413821 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4ac125-84b8c9c875-7gtbm"] Oct 30 17:09:56.431837 ip-10-0-131-136 kubenswrapper[5052]: W1030 17:09:56.431809 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0c628c1_f21f_4a83_a007_98dc19cbf65b.slice/crio-e21404f86757e8227b80a49db1af523a9045a2d60abf8883402cd6293ec6d5ad WatchSource:0}: Error finding container e21404f86757e8227b80a49db1af523a9045a2d60abf8883402cd6293ec6d5ad: Status 404 returned error can't find the container with id e21404f86757e8227b80a49db1af523a9045a2d60abf8883402cd6293ec6d5ad Oct 30 17:09:56.433078 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:56.433054 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-de18d5-7cb866fd58-n9vgg"] Oct 30 17:09:56.445677 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:56.445571 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4ac125-9dcf56996-v4vjr"] Oct 30 17:09:56.450065 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:56.450037 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4ac125-9dcf56996-v4vjr"] Oct 30 17:09:56.557865 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:56.557832 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-de18d5-5fcc8b7f9f-sl9fh"] Oct 30 17:09:56.558455 ip-10-0-131-136 kubenswrapper[5052]: W1030 17:09:56.558430 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1da97b6c_928b_4f33_950e_065cf95c366e.slice/crio-7b9d02fb3fbe7e6e7942e052d4f5b6bab75a6095c0239e3518276c73069b1c4e WatchSource:0}: Error finding container 7b9d02fb3fbe7e6e7942e052d4f5b6bab75a6095c0239e3518276c73069b1c4e: Status 404 returned error can't find the container with id 7b9d02fb3fbe7e6e7942e052d4f5b6bab75a6095c0239e3518276c73069b1c4e Oct 30 17:09:56.602248 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:56.602222 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="565bcd81-6055-4578-ae1a-628829ed7163" path="/var/lib/kubelet/pods/565bcd81-6055-4578-ae1a-628829ed7163/volumes" Oct 30 17:09:56.602760 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:56.602742 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0efd4aa-b83a-48d8-868e-be7d351899e2" path="/var/lib/kubelet/pods/d0efd4aa-b83a-48d8-868e-be7d351899e2/volumes" Oct 30 17:09:57.067878 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:57.067847 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-de18d5-7cb866fd58-n9vgg" event={"ID":"c0c628c1-f21f-4a83-a007-98dc19cbf65b","Type":"ContainerStarted","Data":"4e961c468e339a53b0fe38b272f744427df37bd95f412660636378dc4b7bdd61"} Oct 30 17:09:57.068138 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:57.067896 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-de18d5-7cb866fd58-n9vgg" event={"ID":"c0c628c1-f21f-4a83-a007-98dc19cbf65b","Type":"ContainerStarted","Data":"e21404f86757e8227b80a49db1af523a9045a2d60abf8883402cd6293ec6d5ad"} Oct 30 17:09:57.068138 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:57.067950 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-de18d5-7cb866fd58-n9vgg" Oct 30 17:09:57.069672 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:57.069562 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-de18d5-5fcc8b7f9f-sl9fh" event={"ID":"1da97b6c-928b-4f33-950e-065cf95c366e","Type":"ContainerStarted","Data":"3084d47fe4817d3526059122282e8f9583b115590c999ec375bb96764240acf2"} Oct 30 17:09:57.069672 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:57.069658 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-de18d5-5fcc8b7f9f-sl9fh" Oct 30 17:09:57.069850 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:57.069683 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-de18d5-5fcc8b7f9f-sl9fh" event={"ID":"1da97b6c-928b-4f33-950e-065cf95c366e","Type":"ContainerStarted","Data":"7b9d02fb3fbe7e6e7942e052d4f5b6bab75a6095c0239e3518276c73069b1c4e"} Oct 30 17:09:57.113677 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:57.113624 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-de18d5-7cb866fd58-n9vgg" podStartSLOduration=2.113607145 podStartE2EDuration="2.113607145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:09:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:09:57.110989328 +0000 UTC m=+2952.638324565" watchObservedRunningTime="2026-10-30 17:09:57.113607145 +0000 UTC m=+2952.640942363" Oct 30 17:09:57.157541 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:57.157486 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-de18d5-5fcc8b7f9f-sl9fh" podStartSLOduration=2.15747197 podStartE2EDuration="2.15747197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:09:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:09:57.152970133 +0000 UTC m=+2952.680305373" watchObservedRunningTime="2026-10-30 17:09:57.15747197 +0000 UTC m=+2952.684807189" Oct 30 17:09:59.579711 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:59.579677 5052 patch_prober.go:29] interesting pod/e2e-ipp-b-97a47d-openshift-default-5dcd657cdf-8l2c4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:09:59.580176 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:09:59.579737 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-97a47d-openshift-default-5dcd657cdf-8l2c4" podUID="837038f3-5741-4b1f-8bb8-f8c8e234128f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:10:00.061761 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:10:00.061716 5052 patch_prober.go:29] interesting pod/e2e-ipp-a-4ac125-openshift-default-6569d9988c-4zscj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:10:00.062003 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:10:00.061779 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-4ac125-openshift-default-6569d9988c-4zscj" podUID="4be3cd3f-d7d6-40ec-ad72-8012cfbdb946" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:10:01.988380 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:10:01.988334 5052 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod837038f3_5741_4b1f_8bb8_f8c8e234128f.slice/crio-conmon-8899d0b5a7283c14ed4e600ce5a5a154d462e7f984936dbc51debcd7da8241c2.scope\": RecentStats: unable to find data in memory cache]" Oct 30 17:10:02.113128 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:10:02.113090 5052 generic.go:356] "Generic (PLEG): container finished" podID="837038f3-5741-4b1f-8bb8-f8c8e234128f" containerID="8899d0b5a7283c14ed4e600ce5a5a154d462e7f984936dbc51debcd7da8241c2" exitCode=0 Oct 30 17:10:02.113353 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:10:02.113174 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-97a47d-openshift-default-5dcd657cdf-8l2c4" event={"ID":"837038f3-5741-4b1f-8bb8-f8c8e234128f","Type":"ContainerDied","Data":"8899d0b5a7283c14ed4e600ce5a5a154d462e7f984936dbc51debcd7da8241c2"} Oct 30 17:10:02.113890 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:10:02.113785 5052 scope.go:122] "RemoveContainer" containerID="8899d0b5a7283c14ed4e600ce5a5a154d462e7f984936dbc51debcd7da8241c2" Oct 30 17:10:02.116142 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:10:02.116107 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:10:02.116277 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:10:02.116204 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:10:02.116277 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:10:02.116256 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:10:03.122503 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:10:03.122466 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-97a47d-openshift-default-5dcd657cdf-8l2c4" event={"ID":"837038f3-5741-4b1f-8bb8-f8c8e234128f","Type":"ContainerStarted","Data":"232ede6426254611cacbac9aabff3fbe6a196749b4b91b7efacd57f75a59dc04"} Oct 30 17:10:03.815602 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:10:03.815192 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-97a47d-openshift-default-5dcd657cdf-8l2c4" Oct 30 17:10:03.821600 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:10:03.816031 5052 patch_prober.go:29] interesting pod/e2e-ipp-b-97a47d-openshift-default-5dcd657cdf-8l2c4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body= Oct 30 17:10:03.821600 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:10:03.816084 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-97a47d-openshift-default-5dcd657cdf-8l2c4" podUID="837038f3-5741-4b1f-8bb8-f8c8e234128f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" Oct 30 17:10:04.810426 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:10:04.810299 5052 patch_prober.go:29] interesting pod/e2e-ipp-b-97a47d-openshift-default-5dcd657cdf-8l2c4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body= Oct 30 17:10:04.810426 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:10:04.810359 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-97a47d-openshift-default-5dcd657cdf-8l2c4" podUID="837038f3-5741-4b1f-8bb8-f8c8e234128f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" Oct 30 17:10:05.826824 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:10:05.812712 5052 patch_prober.go:29] interesting pod/e2e-ipp-b-97a47d-openshift-default-5dcd657cdf-8l2c4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body= Oct 30 17:10:05.826824 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:10:05.812794 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-97a47d-openshift-default-5dcd657cdf-8l2c4" podUID="837038f3-5741-4b1f-8bb8-f8c8e234128f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" Oct 30 17:10:06.181391 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:10:06.181354 5052 generic.go:356] "Generic (PLEG): container finished" podID="4be3cd3f-d7d6-40ec-ad72-8012cfbdb946" containerID="81768e865fda8052bd1932d84fde94c0d357a5da7e2b3420fecccefb935956fc" exitCode=0 Oct 30 17:10:06.181566 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:10:06.181399 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4ac125-openshift-default-6569d9988c-4zscj" event={"ID":"4be3cd3f-d7d6-40ec-ad72-8012cfbdb946","Type":"ContainerDied","Data":"81768e865fda8052bd1932d84fde94c0d357a5da7e2b3420fecccefb935956fc"} Oct 30 17:10:06.181958 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:10:06.181938 5052 scope.go:122] "RemoveContainer" containerID="81768e865fda8052bd1932d84fde94c0d357a5da7e2b3420fecccefb935956fc" Oct 30 17:10:06.183933 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:10:06.183899 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:10:06.184066 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:10:06.183990 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:10:06.184066 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:10:06.184060 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:10:06.810078 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:10:06.810046 5052 patch_prober.go:29] interesting pod/e2e-ipp-b-97a47d-openshift-default-5dcd657cdf-8l2c4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body= Oct 30 17:10:06.810281 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:10:06.810115 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-97a47d-openshift-default-5dcd657cdf-8l2c4" podUID="837038f3-5741-4b1f-8bb8-f8c8e234128f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" Oct 30 17:10:07.191190 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:10:07.191148 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4ac125-openshift-default-6569d9988c-4zscj" event={"ID":"4be3cd3f-d7d6-40ec-ad72-8012cfbdb946","Type":"ContainerStarted","Data":"f1c6f98407745add1f293feea3bee17bc861c368fe6984268cec2cd76aac0adf"} Oct 30 17:10:07.559034 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:10:07.558942 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-4ac125-openshift-default-6569d9988c-4zscj" Oct 30 17:10:07.559656 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:10:07.559630 5052 patch_prober.go:29] interesting pod/e2e-ipp-a-4ac125-openshift-default-6569d9988c-4zscj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body= Oct 30 17:10:07.559870 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:10:07.559837 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4ac125-openshift-default-6569d9988c-4zscj" podUID="4be3cd3f-d7d6-40ec-ad72-8012cfbdb946" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" Oct 30 17:10:07.810153 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:10:07.810061 5052 patch_prober.go:29] interesting pod/e2e-ipp-b-97a47d-openshift-default-5dcd657cdf-8l2c4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body= Oct 30 17:10:07.810324 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:10:07.810154 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-97a47d-openshift-default-5dcd657cdf-8l2c4" podUID="837038f3-5741-4b1f-8bb8-f8c8e234128f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" Oct 30 17:10:08.086448 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:10:08.086355 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-de18d5-5fcc8b7f9f-sl9fh" Oct 30 17:10:08.087327 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:10:08.086998 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-de18d5-7cb866fd58-n9vgg" Oct 30 17:10:08.557841 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:10:08.557807 5052 patch_prober.go:29] interesting pod/e2e-ipp-a-4ac125-openshift-default-6569d9988c-4zscj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body= Oct 30 17:10:08.558328 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:10:08.557866 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4ac125-openshift-default-6569d9988c-4zscj" podUID="4be3cd3f-d7d6-40ec-ad72-8012cfbdb946" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" Oct 30 17:10:08.814556 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:10:08.814473 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-97a47d-openshift-default-5dcd657cdf-8l2c4" Oct 30 17:10:08.815010 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:10:08.814984 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-97a47d-openshift-default-5dcd657cdf-8l2c4" Oct 30 17:10:08.815694 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:10:08.815676 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-97a47d-openshift-default-5dcd657cdf-8l2c4" Oct 30 17:10:09.557353 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:10:09.557317 5052 patch_prober.go:29] interesting pod/e2e-ipp-a-4ac125-openshift-default-6569d9988c-4zscj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body= Oct 30 17:10:09.557562 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:10:09.557370 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4ac125-openshift-default-6569d9988c-4zscj" podUID="4be3cd3f-d7d6-40ec-ad72-8012cfbdb946" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" Oct 30 17:10:10.558137 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:10:10.558092 5052 patch_prober.go:29] interesting pod/e2e-ipp-a-4ac125-openshift-default-6569d9988c-4zscj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body= Oct 30 17:10:10.558724 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:10:10.558145 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4ac125-openshift-default-6569d9988c-4zscj" podUID="4be3cd3f-d7d6-40ec-ad72-8012cfbdb946" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" Oct 30 17:10:11.558311 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:10:11.558274 5052 patch_prober.go:29] interesting pod/e2e-ipp-a-4ac125-openshift-default-6569d9988c-4zscj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body= Oct 30 17:10:11.558983 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:10:11.558950 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4ac125-openshift-default-6569d9988c-4zscj" podUID="4be3cd3f-d7d6-40ec-ad72-8012cfbdb946" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" Oct 30 17:10:12.557966 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:10:12.557927 5052 patch_prober.go:29] interesting pod/e2e-ipp-a-4ac125-openshift-default-6569d9988c-4zscj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body= Oct 30 17:10:12.558177 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:10:12.558008 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4ac125-openshift-default-6569d9988c-4zscj" podUID="4be3cd3f-d7d6-40ec-ad72-8012cfbdb946" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" Oct 30 17:10:13.562205 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:10:13.562175 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-4ac125-openshift-default-6569d9988c-4zscj" Oct 30 17:10:13.562714 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:10:13.562621 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-4ac125-openshift-default-6569d9988c-4zscj" Oct 30 17:10:13.563407 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:10:13.563386 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-4ac125-openshift-default-6569d9988c-4zscj" Oct 30 17:10:18.952400 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:10:18.952322 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 17:10:40.669865 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:10:40.669806 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cdwcj"] Oct 30 17:10:45.422320 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:10:45.422252 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4def7e-6cd68b6f59-nkp59"] Oct 30 17:10:45.422987 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:10:45.422971 5052 cpu_manager.go:414] "containerMap: removing container" podUID="565bcd81-6055-4578-ae1a-628829ed7163" containerName="payload-pre-processing" Oct 30 17:10:45.423037 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:10:45.422989 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="565bcd81-6055-4578-ae1a-628829ed7163" containerName="payload-pre-processing" Oct 30 17:10:45.423075 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:10:45.423042 5052 cpu_manager.go:414] "containerMap: removing container" podUID="d0efd4aa-b83a-48d8-868e-be7d351899e2" containerName="payload-processing" Oct 30 17:10:45.423075 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:10:45.423048 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0efd4aa-b83a-48d8-868e-be7d351899e2" containerName="payload-processing" Oct 30 17:10:45.423170 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:10:45.423160 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="565bcd81-6055-4578-ae1a-628829ed7163" containerName="payload-pre-processing" Oct 30 17:10:45.423212 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:10:45.423183 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0efd4aa-b83a-48d8-868e-be7d351899e2" containerName="payload-processing" Oct 30 17:10:45.428279 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:10:45.428258 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4def7e-6cd68b6f59-nkp59" Oct 30 17:10:45.445818 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:10:45.445790 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-4def7e-dockercfg-twmzr\"" Oct 30 17:10:45.451518 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:10:45.451487 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4def7e-6cd68b6f59-nkp59"] Oct 30 17:10:45.459244 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:10:45.459219 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-4def7e\"" Oct 30 17:10:45.717694 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:10:45.613180 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ad13120-2103-4526-a234-93b2e4e852ea-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-4def7e-6cd68b6f59-nkp59\" (UID: \"6ad13120-2103-4526-a234-93b2e4e852ea\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4def7e-6cd68b6f59-nkp59" Oct 30 17:10:45.773697 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:10:45.773649 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsszn\" (UniqueName: \"kubernetes.io/projected/6ad13120-2103-4526-a234-93b2e4e852ea-kube-api-access-qsszn\") pod \"payload-pre-processing-e2e-ipp-b-4def7e-6cd68b6f59-nkp59\" (UID: \"6ad13120-2103-4526-a234-93b2e4e852ea\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4def7e-6cd68b6f59-nkp59" Oct 30 17:10:45.792693 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:10:45.792657 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4def7e-7d4b86c85b-46559"] Oct 30 17:10:45.807636 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:10:45.807606 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4def7e-7d4b86c85b-46559"] Oct 30 17:10:45.807947 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:10:45.807931 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-4def7e-7d4b86c85b-46559" Oct 30 17:10:45.847929 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:10:45.847905 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4wrf8_4f0f46f4-5cda-4d67-b6f3-47df06eb844c/console-operator/2.log" Oct 30 17:10:45.851975 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:10:45.851955 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4wrf8_4f0f46f4-5cda-4d67-b6f3-47df06eb844c/console-operator/2.log" Oct 30 17:10:45.875536 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:10:45.875508 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsszn\" (UniqueName: \"kubernetes.io/projected/6ad13120-2103-4526-a234-93b2e4e852ea-kube-api-access-qsszn\") pod \"payload-pre-processing-e2e-ipp-b-4def7e-6cd68b6f59-nkp59\" (UID: \"6ad13120-2103-4526-a234-93b2e4e852ea\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4def7e-6cd68b6f59-nkp59" Oct 30 17:10:45.875684 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:10:45.875641 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ad13120-2103-4526-a234-93b2e4e852ea-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-4def7e-6cd68b6f59-nkp59\" (UID: \"6ad13120-2103-4526-a234-93b2e4e852ea\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4def7e-6cd68b6f59-nkp59" Oct 30 17:10:45.875758 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:10:45.875737 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2400f0d-42a6-453a-8122-3ddd83396635-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-4def7e-7d4b86c85b-46559\" (UID: \"f2400f0d-42a6-453a-8122-3ddd83396635\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4def7e-7d4b86c85b-46559" Oct 30 17:10:45.875854 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:10:45.875834 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vkx9\" (UniqueName: \"kubernetes.io/projected/f2400f0d-42a6-453a-8122-3ddd83396635-kube-api-access-6vkx9\") pod \"payload-processing-e2e-ipp-b-4def7e-7d4b86c85b-46559\" (UID: \"f2400f0d-42a6-453a-8122-3ddd83396635\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4def7e-7d4b86c85b-46559" Oct 30 17:10:45.876349 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:10:45.876325 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ad13120-2103-4526-a234-93b2e4e852ea-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-4def7e-6cd68b6f59-nkp59\" (UID: \"6ad13120-2103-4526-a234-93b2e4e852ea\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4def7e-6cd68b6f59-nkp59" Oct 30 17:10:45.900121 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:10:45.900088 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsszn\" (UniqueName: \"kubernetes.io/projected/6ad13120-2103-4526-a234-93b2e4e852ea-kube-api-access-qsszn\") pod \"payload-pre-processing-e2e-ipp-b-4def7e-6cd68b6f59-nkp59\" (UID: \"6ad13120-2103-4526-a234-93b2e4e852ea\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4def7e-6cd68b6f59-nkp59" Oct 30 17:10:45.976896 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:10:45.976819 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2400f0d-42a6-453a-8122-3ddd83396635-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-4def7e-7d4b86c85b-46559\" (UID: \"f2400f0d-42a6-453a-8122-3ddd83396635\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4def7e-7d4b86c85b-46559" Oct 30 17:10:45.977014 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:10:45.976920 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vkx9\" (UniqueName: \"kubernetes.io/projected/f2400f0d-42a6-453a-8122-3ddd83396635-kube-api-access-6vkx9\") pod \"payload-processing-e2e-ipp-b-4def7e-7d4b86c85b-46559\" (UID: \"f2400f0d-42a6-453a-8122-3ddd83396635\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4def7e-7d4b86c85b-46559" Oct 30 17:10:45.977566 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:10:45.977519 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2400f0d-42a6-453a-8122-3ddd83396635-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-4def7e-7d4b86c85b-46559\" (UID: \"f2400f0d-42a6-453a-8122-3ddd83396635\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4def7e-7d4b86c85b-46559" Oct 30 17:10:46.001181 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:10:46.001160 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vkx9\" (UniqueName: \"kubernetes.io/projected/f2400f0d-42a6-453a-8122-3ddd83396635-kube-api-access-6vkx9\") pod \"payload-processing-e2e-ipp-b-4def7e-7d4b86c85b-46559\" (UID: \"f2400f0d-42a6-453a-8122-3ddd83396635\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4def7e-7d4b86c85b-46559" Oct 30 17:10:46.043189 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:10:46.043164 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4def7e-6cd68b6f59-nkp59" Oct 30 17:10:46.139589 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:10:46.133079 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-4def7e-7d4b86c85b-46559" Oct 30 17:10:46.361966 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:10:46.361941 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4def7e-6cd68b6f59-nkp59"] Oct 30 17:10:46.362491 ip-10-0-131-136 kubenswrapper[5052]: W1030 17:10:46.362465 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ad13120_2103_4526_a234_93b2e4e852ea.slice/crio-a8b64cd2952ac478a35e54ab2255104ec3128e7fded06997cc0dbb1ded613e1f WatchSource:0}: Error finding container a8b64cd2952ac478a35e54ab2255104ec3128e7fded06997cc0dbb1ded613e1f: Status 404 returned error can't find the container with id a8b64cd2952ac478a35e54ab2255104ec3128e7fded06997cc0dbb1ded613e1f Oct 30 17:10:46.447974 ip-10-0-131-136 kubenswrapper[5052]: W1030 17:10:46.447950 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2400f0d_42a6_453a_8122_3ddd83396635.slice/crio-8bb685ebf72a07970488dab4fc0ff377917a728451e70351a05fdd51a1298904 WatchSource:0}: Error finding container 8bb685ebf72a07970488dab4fc0ff377917a728451e70351a05fdd51a1298904: Status 404 returned error can't find the container with id 8bb685ebf72a07970488dab4fc0ff377917a728451e70351a05fdd51a1298904 Oct 30 17:10:46.451349 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:10:46.451316 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4def7e-7d4b86c85b-46559"] Oct 30 17:10:47.121793 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:10:47.121758 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-4def7e-7d4b86c85b-46559" event={"ID":"f2400f0d-42a6-453a-8122-3ddd83396635","Type":"ContainerStarted","Data":"2f89eae6b5d253a65dc97691897ef5e88f51d958f8ffd152ff34427129ef7641"} Oct 30 17:10:47.122010 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:10:47.121820 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-4def7e-7d4b86c85b-46559" event={"ID":"f2400f0d-42a6-453a-8122-3ddd83396635","Type":"ContainerStarted","Data":"8bb685ebf72a07970488dab4fc0ff377917a728451e70351a05fdd51a1298904"} Oct 30 17:10:47.122010 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:10:47.121850 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-4def7e-7d4b86c85b-46559" Oct 30 17:10:47.123566 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:10:47.123535 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4def7e-6cd68b6f59-nkp59" event={"ID":"6ad13120-2103-4526-a234-93b2e4e852ea","Type":"ContainerStarted","Data":"de71e4eca834b8ba05f1943dbb5f7a3187be64111ba5d64eb8448ce5b2e75275"} Oct 30 17:10:47.123699 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:10:47.123601 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4def7e-6cd68b6f59-nkp59" event={"ID":"6ad13120-2103-4526-a234-93b2e4e852ea","Type":"ContainerStarted","Data":"a8b64cd2952ac478a35e54ab2255104ec3128e7fded06997cc0dbb1ded613e1f"} Oct 30 17:10:47.123699 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:10:47.123628 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4def7e-6cd68b6f59-nkp59" Oct 30 17:10:47.167665 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:10:47.167612 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-4def7e-7d4b86c85b-46559" podStartSLOduration=2.167593915 podStartE2EDuration="2.167593915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:10:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:10:47.163645582 +0000 UTC m=+3002.690980826" watchObservedRunningTime="2026-10-30 17:10:47.167593915 +0000 UTC m=+3002.694929149" Oct 30 17:10:47.212211 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:10:47.212152 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4def7e-6cd68b6f59-nkp59" podStartSLOduration=2.212131903 podStartE2EDuration="2.212131903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:10:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:10:47.208243014 +0000 UTC m=+3002.735578232" watchObservedRunningTime="2026-10-30 17:10:47.212131903 +0000 UTC m=+3002.739467123" Oct 30 17:10:48.842235 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:10:48.842197 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 17:10:58.199226 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:10:58.198952 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-4def7e-7d4b86c85b-46559" Oct 30 17:10:58.226640 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:10:58.226599 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4def7e-6cd68b6f59-nkp59" Oct 30 17:11:01.909166 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:01.909131 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-0ae3fe-openshift-default-69fdb67df-sbgql"] Oct 30 17:11:01.916492 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:01.916464 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-0ae3fe-openshift-default-69fdb67df-sbgql" Oct 30 17:11:01.932684 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:01.932630 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-0ae3fe-openshift-default-69fdb67df-sbgql"] Oct 30 17:11:01.945621 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:01.943854 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-0ae3fe-openshift-default-dockercfg-x842m\"" Oct 30 17:11:01.978200 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:01.978148 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/666e4b84-2d87-4557-a8b5-10f890a1740b-istio-podinfo\") pod \"e2e-ipp-b-0ae3fe-openshift-default-69fdb67df-sbgql\" (UID: \"666e4b84-2d87-4557-a8b5-10f890a1740b\") " pod="openshift-ingress/e2e-ipp-b-0ae3fe-openshift-default-69fdb67df-sbgql" Oct 30 17:11:01.978422 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:01.978280 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/666e4b84-2d87-4557-a8b5-10f890a1740b-credential-socket\") pod \"e2e-ipp-b-0ae3fe-openshift-default-69fdb67df-sbgql\" (UID: \"666e4b84-2d87-4557-a8b5-10f890a1740b\") " pod="openshift-ingress/e2e-ipp-b-0ae3fe-openshift-default-69fdb67df-sbgql" Oct 30 17:11:01.978422 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:01.978350 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/666e4b84-2d87-4557-a8b5-10f890a1740b-workload-certs\") pod \"e2e-ipp-b-0ae3fe-openshift-default-69fdb67df-sbgql\" (UID: \"666e4b84-2d87-4557-a8b5-10f890a1740b\") " pod="openshift-ingress/e2e-ipp-b-0ae3fe-openshift-default-69fdb67df-sbgql" Oct 30 17:11:01.978422 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:01.978396 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbpnc\" (UniqueName: \"kubernetes.io/projected/666e4b84-2d87-4557-a8b5-10f890a1740b-kube-api-access-lbpnc\") pod \"e2e-ipp-b-0ae3fe-openshift-default-69fdb67df-sbgql\" (UID: \"666e4b84-2d87-4557-a8b5-10f890a1740b\") " pod="openshift-ingress/e2e-ipp-b-0ae3fe-openshift-default-69fdb67df-sbgql" Oct 30 17:11:01.978630 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:01.978468 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/666e4b84-2d87-4557-a8b5-10f890a1740b-istio-data\") pod \"e2e-ipp-b-0ae3fe-openshift-default-69fdb67df-sbgql\" (UID: \"666e4b84-2d87-4557-a8b5-10f890a1740b\") " pod="openshift-ingress/e2e-ipp-b-0ae3fe-openshift-default-69fdb67df-sbgql" Oct 30 17:11:01.978630 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:01.978492 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/666e4b84-2d87-4557-a8b5-10f890a1740b-istio-token\") pod \"e2e-ipp-b-0ae3fe-openshift-default-69fdb67df-sbgql\" (UID: \"666e4b84-2d87-4557-a8b5-10f890a1740b\") " pod="openshift-ingress/e2e-ipp-b-0ae3fe-openshift-default-69fdb67df-sbgql" Oct 30 17:11:01.978630 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:01.978610 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/666e4b84-2d87-4557-a8b5-10f890a1740b-istio-envoy\") pod \"e2e-ipp-b-0ae3fe-openshift-default-69fdb67df-sbgql\" (UID: \"666e4b84-2d87-4557-a8b5-10f890a1740b\") " pod="openshift-ingress/e2e-ipp-b-0ae3fe-openshift-default-69fdb67df-sbgql" Oct 30 17:11:01.978809 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:01.978653 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/666e4b84-2d87-4557-a8b5-10f890a1740b-istiod-ca-cert\") pod \"e2e-ipp-b-0ae3fe-openshift-default-69fdb67df-sbgql\" (UID: \"666e4b84-2d87-4557-a8b5-10f890a1740b\") " pod="openshift-ingress/e2e-ipp-b-0ae3fe-openshift-default-69fdb67df-sbgql" Oct 30 17:11:01.978809 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:01.978772 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/666e4b84-2d87-4557-a8b5-10f890a1740b-workload-socket\") pod \"e2e-ipp-b-0ae3fe-openshift-default-69fdb67df-sbgql\" (UID: \"666e4b84-2d87-4557-a8b5-10f890a1740b\") " pod="openshift-ingress/e2e-ipp-b-0ae3fe-openshift-default-69fdb67df-sbgql" Oct 30 17:11:02.081023 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:02.080981 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/666e4b84-2d87-4557-a8b5-10f890a1740b-credential-socket\") pod \"e2e-ipp-b-0ae3fe-openshift-default-69fdb67df-sbgql\" (UID: \"666e4b84-2d87-4557-a8b5-10f890a1740b\") " pod="openshift-ingress/e2e-ipp-b-0ae3fe-openshift-default-69fdb67df-sbgql" Oct 30 17:11:02.082115 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:02.082079 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/666e4b84-2d87-4557-a8b5-10f890a1740b-workload-certs\") pod \"e2e-ipp-b-0ae3fe-openshift-default-69fdb67df-sbgql\" (UID: \"666e4b84-2d87-4557-a8b5-10f890a1740b\") " pod="openshift-ingress/e2e-ipp-b-0ae3fe-openshift-default-69fdb67df-sbgql" Oct 30 17:11:02.097774 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:02.083340 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbpnc\" (UniqueName: \"kubernetes.io/projected/666e4b84-2d87-4557-a8b5-10f890a1740b-kube-api-access-lbpnc\") pod \"e2e-ipp-b-0ae3fe-openshift-default-69fdb67df-sbgql\" (UID: \"666e4b84-2d87-4557-a8b5-10f890a1740b\") " pod="openshift-ingress/e2e-ipp-b-0ae3fe-openshift-default-69fdb67df-sbgql" Oct 30 17:11:02.098167 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:02.098129 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/666e4b84-2d87-4557-a8b5-10f890a1740b-istio-data\") pod \"e2e-ipp-b-0ae3fe-openshift-default-69fdb67df-sbgql\" (UID: \"666e4b84-2d87-4557-a8b5-10f890a1740b\") " pod="openshift-ingress/e2e-ipp-b-0ae3fe-openshift-default-69fdb67df-sbgql" Oct 30 17:11:02.098356 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:02.098337 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/666e4b84-2d87-4557-a8b5-10f890a1740b-istio-token\") pod \"e2e-ipp-b-0ae3fe-openshift-default-69fdb67df-sbgql\" (UID: \"666e4b84-2d87-4557-a8b5-10f890a1740b\") " pod="openshift-ingress/e2e-ipp-b-0ae3fe-openshift-default-69fdb67df-sbgql" Oct 30 17:11:02.098595 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:02.098562 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/666e4b84-2d87-4557-a8b5-10f890a1740b-istio-envoy\") pod \"e2e-ipp-b-0ae3fe-openshift-default-69fdb67df-sbgql\" (UID: \"666e4b84-2d87-4557-a8b5-10f890a1740b\") " pod="openshift-ingress/e2e-ipp-b-0ae3fe-openshift-default-69fdb67df-sbgql" Oct 30 17:11:02.098801 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:02.098783 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/666e4b84-2d87-4557-a8b5-10f890a1740b-istiod-ca-cert\") pod \"e2e-ipp-b-0ae3fe-openshift-default-69fdb67df-sbgql\" (UID: \"666e4b84-2d87-4557-a8b5-10f890a1740b\") " pod="openshift-ingress/e2e-ipp-b-0ae3fe-openshift-default-69fdb67df-sbgql" Oct 30 17:11:02.098909 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:02.091282 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/666e4b84-2d87-4557-a8b5-10f890a1740b-workload-certs\") pod \"e2e-ipp-b-0ae3fe-openshift-default-69fdb67df-sbgql\" (UID: \"666e4b84-2d87-4557-a8b5-10f890a1740b\") " pod="openshift-ingress/e2e-ipp-b-0ae3fe-openshift-default-69fdb67df-sbgql" Oct 30 17:11:02.098976 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:02.081958 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/666e4b84-2d87-4557-a8b5-10f890a1740b-credential-socket\") pod \"e2e-ipp-b-0ae3fe-openshift-default-69fdb67df-sbgql\" (UID: \"666e4b84-2d87-4557-a8b5-10f890a1740b\") " pod="openshift-ingress/e2e-ipp-b-0ae3fe-openshift-default-69fdb67df-sbgql" Oct 30 17:11:02.098976 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:02.098611 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/666e4b84-2d87-4557-a8b5-10f890a1740b-istio-data\") pod \"e2e-ipp-b-0ae3fe-openshift-default-69fdb67df-sbgql\" (UID: \"666e4b84-2d87-4557-a8b5-10f890a1740b\") " pod="openshift-ingress/e2e-ipp-b-0ae3fe-openshift-default-69fdb67df-sbgql" Oct 30 17:11:02.099854 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:02.099831 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/666e4b84-2d87-4557-a8b5-10f890a1740b-istiod-ca-cert\") pod \"e2e-ipp-b-0ae3fe-openshift-default-69fdb67df-sbgql\" (UID: \"666e4b84-2d87-4557-a8b5-10f890a1740b\") " pod="openshift-ingress/e2e-ipp-b-0ae3fe-openshift-default-69fdb67df-sbgql" Oct 30 17:11:02.100481 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:02.100457 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/666e4b84-2d87-4557-a8b5-10f890a1740b-workload-socket\") pod \"e2e-ipp-b-0ae3fe-openshift-default-69fdb67df-sbgql\" (UID: \"666e4b84-2d87-4557-a8b5-10f890a1740b\") " pod="openshift-ingress/e2e-ipp-b-0ae3fe-openshift-default-69fdb67df-sbgql" Oct 30 17:11:02.100746 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:02.100725 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/666e4b84-2d87-4557-a8b5-10f890a1740b-istio-podinfo\") pod \"e2e-ipp-b-0ae3fe-openshift-default-69fdb67df-sbgql\" (UID: \"666e4b84-2d87-4557-a8b5-10f890a1740b\") " pod="openshift-ingress/e2e-ipp-b-0ae3fe-openshift-default-69fdb67df-sbgql" Oct 30 17:11:02.125013 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:02.124980 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/666e4b84-2d87-4557-a8b5-10f890a1740b-workload-socket\") pod \"e2e-ipp-b-0ae3fe-openshift-default-69fdb67df-sbgql\" (UID: \"666e4b84-2d87-4557-a8b5-10f890a1740b\") " pod="openshift-ingress/e2e-ipp-b-0ae3fe-openshift-default-69fdb67df-sbgql" Oct 30 17:11:02.161657 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:02.161475 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/666e4b84-2d87-4557-a8b5-10f890a1740b-istio-podinfo\") pod \"e2e-ipp-b-0ae3fe-openshift-default-69fdb67df-sbgql\" (UID: \"666e4b84-2d87-4557-a8b5-10f890a1740b\") " pod="openshift-ingress/e2e-ipp-b-0ae3fe-openshift-default-69fdb67df-sbgql" Oct 30 17:11:02.174997 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:02.162447 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbpnc\" (UniqueName: \"kubernetes.io/projected/666e4b84-2d87-4557-a8b5-10f890a1740b-kube-api-access-lbpnc\") pod \"e2e-ipp-b-0ae3fe-openshift-default-69fdb67df-sbgql\" (UID: \"666e4b84-2d87-4557-a8b5-10f890a1740b\") " pod="openshift-ingress/e2e-ipp-b-0ae3fe-openshift-default-69fdb67df-sbgql" Oct 30 17:11:02.174997 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:02.162791 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/666e4b84-2d87-4557-a8b5-10f890a1740b-istio-envoy\") pod \"e2e-ipp-b-0ae3fe-openshift-default-69fdb67df-sbgql\" (UID: \"666e4b84-2d87-4557-a8b5-10f890a1740b\") " pod="openshift-ingress/e2e-ipp-b-0ae3fe-openshift-default-69fdb67df-sbgql" Oct 30 17:11:02.174997 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:02.174084 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/666e4b84-2d87-4557-a8b5-10f890a1740b-istio-token\") pod \"e2e-ipp-b-0ae3fe-openshift-default-69fdb67df-sbgql\" (UID: \"666e4b84-2d87-4557-a8b5-10f890a1740b\") " pod="openshift-ingress/e2e-ipp-b-0ae3fe-openshift-default-69fdb67df-sbgql" Oct 30 17:11:02.242169 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:02.242127 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-0ae3fe-openshift-default-69fdb67df-sbgql" Oct 30 17:11:02.658218 ip-10-0-131-136 kubenswrapper[5052]: W1030 17:11:02.658189 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod666e4b84_2d87_4557_a8b5_10f890a1740b.slice/crio-d97758c7a1bed262c7b4b3707d838af14ceaa224a0f7b30f16228cdf98e6ec14 WatchSource:0}: Error finding container d97758c7a1bed262c7b4b3707d838af14ceaa224a0f7b30f16228cdf98e6ec14: Status 404 returned error can't find the container with id d97758c7a1bed262c7b4b3707d838af14ceaa224a0f7b30f16228cdf98e6ec14 Oct 30 17:11:02.658420 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:02.658395 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-0ae3fe-openshift-default-69fdb67df-sbgql"] Oct 30 17:11:02.661420 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:02.661382 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:11:02.661533 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:02.661473 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:11:02.661533 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:02.661517 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:11:03.320757 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:03.320710 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0ae3fe-openshift-default-69fdb67df-sbgql" event={"ID":"666e4b84-2d87-4557-a8b5-10f890a1740b","Type":"ContainerStarted","Data":"3e8dd9f08e58cf02aadc5e493e0f6bbb4b3d3a4f4e48c94664612a42fed64aa3"} Oct 30 17:11:03.321191 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:03.320818 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0ae3fe-openshift-default-69fdb67df-sbgql" event={"ID":"666e4b84-2d87-4557-a8b5-10f890a1740b","Type":"ContainerStarted","Data":"d97758c7a1bed262c7b4b3707d838af14ceaa224a0f7b30f16228cdf98e6ec14"} Oct 30 17:11:03.382847 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:03.382792 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-0ae3fe-openshift-default-69fdb67df-sbgql" podStartSLOduration=2.382778479 podStartE2EDuration="2.382778479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:11:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:11:03.381530635 +0000 UTC m=+3018.908865854" watchObservedRunningTime="2026-10-30 17:11:03.382778479 +0000 UTC m=+3018.910113697" Oct 30 17:11:04.243220 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:04.243180 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-0ae3fe-openshift-default-69fdb67df-sbgql" Oct 30 17:11:04.249909 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:04.249878 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-0ae3fe-openshift-default-69fdb67df-sbgql" Oct 30 17:11:04.328630 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:04.328560 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-0ae3fe-openshift-default-69fdb67df-sbgql" Oct 30 17:11:04.329639 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:04.329615 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-0ae3fe-openshift-default-69fdb67df-sbgql" Oct 30 17:11:07.536955 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:07.536922 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0ae3fe-5b445b79c8-cqkdf"] Oct 30 17:11:07.543096 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:07.543052 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-0ae3fe-5b445b79c8-cqkdf" Oct 30 17:11:07.560527 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:07.560498 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0ae3fe-5b445b79c8-cqkdf"] Oct 30 17:11:07.568551 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:07.568525 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-0ae3fe\"" Oct 30 17:11:07.589255 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:07.589225 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-0ae3fe-dockercfg-brgn6\"" Oct 30 17:11:07.600272 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:07.600241 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0ae3fe-598c75dd67-pjf8j"] Oct 30 17:11:07.606329 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:07.606308 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ae3fe-598c75dd67-pjf8j" Oct 30 17:11:07.631547 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:07.631495 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67f898fa-56b4-4f6d-8b02-3c9b24cedcf0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-0ae3fe-5b445b79c8-cqkdf\" (UID: \"67f898fa-56b4-4f6d-8b02-3c9b24cedcf0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0ae3fe-5b445b79c8-cqkdf" Oct 30 17:11:07.639426 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:07.639393 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0ae3fe-598c75dd67-pjf8j"] Oct 30 17:11:07.678388 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:07.675722 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9p8q\" (UniqueName: \"kubernetes.io/projected/67f898fa-56b4-4f6d-8b02-3c9b24cedcf0-kube-api-access-t9p8q\") pod \"payload-processing-e2e-ipp-b-0ae3fe-5b445b79c8-cqkdf\" (UID: \"67f898fa-56b4-4f6d-8b02-3c9b24cedcf0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0ae3fe-5b445b79c8-cqkdf" Oct 30 17:11:07.839659 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:07.839143 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67f898fa-56b4-4f6d-8b02-3c9b24cedcf0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-0ae3fe-5b445b79c8-cqkdf\" (UID: \"67f898fa-56b4-4f6d-8b02-3c9b24cedcf0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0ae3fe-5b445b79c8-cqkdf" Oct 30 17:11:07.839659 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:07.839272 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4739b888-b4ee-4912-99d8-cdf7b0d4abf6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-0ae3fe-598c75dd67-pjf8j\" (UID: \"4739b888-b4ee-4912-99d8-cdf7b0d4abf6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ae3fe-598c75dd67-pjf8j" Oct 30 17:11:07.839659 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:07.839318 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mlbz\" (UniqueName: \"kubernetes.io/projected/4739b888-b4ee-4912-99d8-cdf7b0d4abf6-kube-api-access-5mlbz\") pod \"payload-pre-processing-e2e-ipp-b-0ae3fe-598c75dd67-pjf8j\" (UID: \"4739b888-b4ee-4912-99d8-cdf7b0d4abf6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ae3fe-598c75dd67-pjf8j" Oct 30 17:11:07.839659 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:07.839403 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9p8q\" (UniqueName: \"kubernetes.io/projected/67f898fa-56b4-4f6d-8b02-3c9b24cedcf0-kube-api-access-t9p8q\") pod \"payload-processing-e2e-ipp-b-0ae3fe-5b445b79c8-cqkdf\" (UID: \"67f898fa-56b4-4f6d-8b02-3c9b24cedcf0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0ae3fe-5b445b79c8-cqkdf" Oct 30 17:11:07.876688 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:07.845678 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67f898fa-56b4-4f6d-8b02-3c9b24cedcf0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-0ae3fe-5b445b79c8-cqkdf\" (UID: \"67f898fa-56b4-4f6d-8b02-3c9b24cedcf0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0ae3fe-5b445b79c8-cqkdf" Oct 30 17:11:07.947563 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:07.942267 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4739b888-b4ee-4912-99d8-cdf7b0d4abf6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-0ae3fe-598c75dd67-pjf8j\" (UID: \"4739b888-b4ee-4912-99d8-cdf7b0d4abf6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ae3fe-598c75dd67-pjf8j" Oct 30 17:11:07.947563 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:07.942334 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mlbz\" (UniqueName: \"kubernetes.io/projected/4739b888-b4ee-4912-99d8-cdf7b0d4abf6-kube-api-access-5mlbz\") pod \"payload-pre-processing-e2e-ipp-b-0ae3fe-598c75dd67-pjf8j\" (UID: \"4739b888-b4ee-4912-99d8-cdf7b0d4abf6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ae3fe-598c75dd67-pjf8j" Oct 30 17:11:07.947563 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:07.944013 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4739b888-b4ee-4912-99d8-cdf7b0d4abf6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-0ae3fe-598c75dd67-pjf8j\" (UID: \"4739b888-b4ee-4912-99d8-cdf7b0d4abf6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ae3fe-598c75dd67-pjf8j" Oct 30 17:11:07.953039 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:07.952110 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9p8q\" (UniqueName: \"kubernetes.io/projected/67f898fa-56b4-4f6d-8b02-3c9b24cedcf0-kube-api-access-t9p8q\") pod \"payload-processing-e2e-ipp-b-0ae3fe-5b445b79c8-cqkdf\" (UID: \"67f898fa-56b4-4f6d-8b02-3c9b24cedcf0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0ae3fe-5b445b79c8-cqkdf" Oct 30 17:11:07.975036 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:07.972572 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mlbz\" (UniqueName: \"kubernetes.io/projected/4739b888-b4ee-4912-99d8-cdf7b0d4abf6-kube-api-access-5mlbz\") pod \"payload-pre-processing-e2e-ipp-b-0ae3fe-598c75dd67-pjf8j\" (UID: \"4739b888-b4ee-4912-99d8-cdf7b0d4abf6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ae3fe-598c75dd67-pjf8j" Oct 30 17:11:08.160909 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:08.160879 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-0ae3fe-5b445b79c8-cqkdf" Oct 30 17:11:08.241310 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:08.241242 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ae3fe-598c75dd67-pjf8j" Oct 30 17:11:08.472198 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:08.472157 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0ae3fe-5b445b79c8-cqkdf"] Oct 30 17:11:08.472330 ip-10-0-131-136 kubenswrapper[5052]: W1030 17:11:08.472307 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67f898fa_56b4_4f6d_8b02_3c9b24cedcf0.slice/crio-ce08a7b81e098f28f5f88a7c4b368711689aaec692e6729c445816f50ee65d69 WatchSource:0}: Error finding container ce08a7b81e098f28f5f88a7c4b368711689aaec692e6729c445816f50ee65d69: Status 404 returned error can't find the container with id ce08a7b81e098f28f5f88a7c4b368711689aaec692e6729c445816f50ee65d69 Oct 30 17:11:08.553788 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:08.553763 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0ae3fe-598c75dd67-pjf8j"] Oct 30 17:11:08.559020 ip-10-0-131-136 kubenswrapper[5052]: W1030 17:11:08.558991 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4739b888_b4ee_4912_99d8_cdf7b0d4abf6.slice/crio-36ecf1ee394e855f8230c7bff317a006e99a2f3dbf2738e3439e23f4ec03d5d4 WatchSource:0}: Error finding container 36ecf1ee394e855f8230c7bff317a006e99a2f3dbf2738e3439e23f4ec03d5d4: Status 404 returned error can't find the container with id 36ecf1ee394e855f8230c7bff317a006e99a2f3dbf2738e3439e23f4ec03d5d4 Oct 30 17:11:09.158576 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:09.158540 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-hwkgn"] Oct 30 17:11:09.158798 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:09.158774 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-hwkgn" podUID="96134e17-039f-4ea3-8522-ff54e3ab6797" containerName="istio-proxy" containerID="cri-o://554308610ee1f23add189bb1173774101b5b78e04fe078e83b3df4c8e85488dc" gracePeriod=30 Oct 30 17:11:09.376524 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:09.376487 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ae3fe-598c75dd67-pjf8j" event={"ID":"4739b888-b4ee-4912-99d8-cdf7b0d4abf6","Type":"ContainerStarted","Data":"9098f223120447b5115fade54ed8b94337042c62615495447d6ed32cc237851d"} Oct 30 17:11:09.376736 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:09.376543 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ae3fe-598c75dd67-pjf8j" event={"ID":"4739b888-b4ee-4912-99d8-cdf7b0d4abf6","Type":"ContainerStarted","Data":"36ecf1ee394e855f8230c7bff317a006e99a2f3dbf2738e3439e23f4ec03d5d4"} Oct 30 17:11:09.376736 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:09.376564 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ae3fe-598c75dd67-pjf8j" Oct 30 17:11:09.378368 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:09.378337 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-0ae3fe-5b445b79c8-cqkdf" event={"ID":"67f898fa-56b4-4f6d-8b02-3c9b24cedcf0","Type":"ContainerStarted","Data":"b5ea2bd3a524e7e634afbaf076904cb9f7eead38f1a0373069b73be26e58b3ea"} Oct 30 17:11:09.378479 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:09.378390 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-0ae3fe-5b445b79c8-cqkdf" event={"ID":"67f898fa-56b4-4f6d-8b02-3c9b24cedcf0","Type":"ContainerStarted","Data":"ce08a7b81e098f28f5f88a7c4b368711689aaec692e6729c445816f50ee65d69"} Oct 30 17:11:09.378479 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:09.378420 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-0ae3fe-5b445b79c8-cqkdf" Oct 30 17:11:09.421745 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:09.421655 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ae3fe-598c75dd67-pjf8j" podStartSLOduration=2.42164304 podStartE2EDuration="2.42164304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:11:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:11:09.418392281 +0000 UTC m=+3024.945727518" watchObservedRunningTime="2026-10-30 17:11:09.42164304 +0000 UTC m=+3024.948978258" Oct 30 17:11:09.461430 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:09.461376 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-0ae3fe-5b445b79c8-cqkdf" podStartSLOduration=2.461359237 podStartE2EDuration="2.461359237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:11:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:11:09.46011711 +0000 UTC m=+3024.987452328" watchObservedRunningTime="2026-10-30 17:11:09.461359237 +0000 UTC m=+3024.988694458" Oct 30 17:11:14.418968 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:14.418899 5052 generic.go:356] "Generic (PLEG): container finished" podID="96134e17-039f-4ea3-8522-ff54e3ab6797" containerID="554308610ee1f23add189bb1173774101b5b78e04fe078e83b3df4c8e85488dc" exitCode=0 Oct 30 17:11:14.419314 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:14.418975 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-hwkgn" event={"ID":"96134e17-039f-4ea3-8522-ff54e3ab6797","Type":"ContainerDied","Data":"554308610ee1f23add189bb1173774101b5b78e04fe078e83b3df4c8e85488dc"} Oct 30 17:11:14.452934 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:14.452911 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-hwkgn" Oct 30 17:11:14.558274 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:14.558216 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/96134e17-039f-4ea3-8522-ff54e3ab6797-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96134e17-039f-4ea3-8522-ff54e3ab6797-istio-podinfo\") pod \"96134e17-039f-4ea3-8522-ff54e3ab6797\" (UID: \"96134e17-039f-4ea3-8522-ff54e3ab6797\") " Oct 30 17:11:14.558472 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:14.558285 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96134e17-039f-4ea3-8522-ff54e3ab6797-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96134e17-039f-4ea3-8522-ff54e3ab6797-workload-socket\") pod \"96134e17-039f-4ea3-8522-ff54e3ab6797\" (UID: \"96134e17-039f-4ea3-8522-ff54e3ab6797\") " Oct 30 17:11:14.558472 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:14.558369 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96134e17-039f-4ea3-8522-ff54e3ab6797-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96134e17-039f-4ea3-8522-ff54e3ab6797-istiod-ca-cert\") pod \"96134e17-039f-4ea3-8522-ff54e3ab6797\" (UID: \"96134e17-039f-4ea3-8522-ff54e3ab6797\") " Oct 30 17:11:14.558472 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:14.558456 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96134e17-039f-4ea3-8522-ff54e3ab6797-istio-token\" (UniqueName: \"kubernetes.io/projected/96134e17-039f-4ea3-8522-ff54e3ab6797-istio-token\") pod \"96134e17-039f-4ea3-8522-ff54e3ab6797\" (UID: \"96134e17-039f-4ea3-8522-ff54e3ab6797\") " Oct 30 17:11:14.558750 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:14.558555 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96134e17-039f-4ea3-8522-ff54e3ab6797-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96134e17-039f-4ea3-8522-ff54e3ab6797-credential-socket\") pod \"96134e17-039f-4ea3-8522-ff54e3ab6797\" (UID: \"96134e17-039f-4ea3-8522-ff54e3ab6797\") " Oct 30 17:11:14.558750 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:14.558625 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96134e17-039f-4ea3-8522-ff54e3ab6797-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96134e17-039f-4ea3-8522-ff54e3ab6797-istio-envoy\") pod \"96134e17-039f-4ea3-8522-ff54e3ab6797\" (UID: \"96134e17-039f-4ea3-8522-ff54e3ab6797\") " Oct 30 17:11:14.558750 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:14.558672 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96134e17-039f-4ea3-8522-ff54e3ab6797-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96134e17-039f-4ea3-8522-ff54e3ab6797-istio-data\") pod \"96134e17-039f-4ea3-8522-ff54e3ab6797\" (UID: \"96134e17-039f-4ea3-8522-ff54e3ab6797\") " Oct 30 17:11:14.558920 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:14.558851 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96134e17-039f-4ea3-8522-ff54e3ab6797-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96134e17-039f-4ea3-8522-ff54e3ab6797-workload-certs\") pod \"96134e17-039f-4ea3-8522-ff54e3ab6797\" (UID: \"96134e17-039f-4ea3-8522-ff54e3ab6797\") " Oct 30 17:11:14.558970 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:14.558934 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96134e17-039f-4ea3-8522-ff54e3ab6797-kube-api-access-gg4j8\" (UniqueName: \"kubernetes.io/projected/96134e17-039f-4ea3-8522-ff54e3ab6797-kube-api-access-gg4j8\") pod \"96134e17-039f-4ea3-8522-ff54e3ab6797\" (UID: \"96134e17-039f-4ea3-8522-ff54e3ab6797\") " Oct 30 17:11:14.560560 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:14.560517 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96134e17-039f-4ea3-8522-ff54e3ab6797-workload-socket" pod "96134e17-039f-4ea3-8522-ff54e3ab6797" (UID: "96134e17-039f-4ea3-8522-ff54e3ab6797"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:11:14.561055 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:14.561003 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96134e17-039f-4ea3-8522-ff54e3ab6797-istiod-ca-cert" pod "96134e17-039f-4ea3-8522-ff54e3ab6797" (UID: "96134e17-039f-4ea3-8522-ff54e3ab6797"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:11:14.562007 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:14.561972 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96134e17-039f-4ea3-8522-ff54e3ab6797-credential-socket" pod "96134e17-039f-4ea3-8522-ff54e3ab6797" (UID: "96134e17-039f-4ea3-8522-ff54e3ab6797"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:11:14.562201 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:14.562179 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96134e17-039f-4ea3-8522-ff54e3ab6797-istio-data" pod "96134e17-039f-4ea3-8522-ff54e3ab6797" (UID: "96134e17-039f-4ea3-8522-ff54e3ab6797"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:11:14.562386 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:14.562366 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96134e17-039f-4ea3-8522-ff54e3ab6797-workload-certs" pod "96134e17-039f-4ea3-8522-ff54e3ab6797" (UID: "96134e17-039f-4ea3-8522-ff54e3ab6797"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:11:14.566669 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:14.565402 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/96134e17-039f-4ea3-8522-ff54e3ab6797-istio-podinfo" pod "96134e17-039f-4ea3-8522-ff54e3ab6797" (UID: "96134e17-039f-4ea3-8522-ff54e3ab6797"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:11:14.566784 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:14.566659 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96134e17-039f-4ea3-8522-ff54e3ab6797-istio-envoy" pod "96134e17-039f-4ea3-8522-ff54e3ab6797" (UID: "96134e17-039f-4ea3-8522-ff54e3ab6797"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:11:14.566856 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:14.566821 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96134e17-039f-4ea3-8522-ff54e3ab6797-istio-token" pod "96134e17-039f-4ea3-8522-ff54e3ab6797" (UID: "96134e17-039f-4ea3-8522-ff54e3ab6797"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:11:14.570423 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:14.568541 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96134e17-039f-4ea3-8522-ff54e3ab6797-kube-api-access-gg4j8" pod "96134e17-039f-4ea3-8522-ff54e3ab6797" (UID: "96134e17-039f-4ea3-8522-ff54e3ab6797"). InnerVolumeSpecName "kube-api-access-gg4j8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:11:14.661811 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:14.661782 5052 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96134e17-039f-4ea3-8522-ff54e3ab6797-istio-podinfo\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:11:14.661811 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:14.661810 5052 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96134e17-039f-4ea3-8522-ff54e3ab6797-workload-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:11:14.662011 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:14.661821 5052 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96134e17-039f-4ea3-8522-ff54e3ab6797-istiod-ca-cert\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:11:14.662011 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:14.661830 5052 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96134e17-039f-4ea3-8522-ff54e3ab6797-istio-token\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:11:14.662011 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:14.661838 5052 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96134e17-039f-4ea3-8522-ff54e3ab6797-credential-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:11:14.662011 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:14.661846 5052 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96134e17-039f-4ea3-8522-ff54e3ab6797-istio-envoy\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:11:14.662011 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:14.661856 5052 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96134e17-039f-4ea3-8522-ff54e3ab6797-istio-data\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:11:14.662011 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:14.661863 5052 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96134e17-039f-4ea3-8522-ff54e3ab6797-workload-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:11:14.662011 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:14.661871 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gg4j8\" (UniqueName: \"kubernetes.io/projected/96134e17-039f-4ea3-8522-ff54e3ab6797-kube-api-access-gg4j8\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:11:15.427911 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:15.427882 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-hwkgn" Oct 30 17:11:15.428314 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:15.427927 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-hwkgn" event={"ID":"96134e17-039f-4ea3-8522-ff54e3ab6797","Type":"ContainerDied","Data":"7076977a1cf44e6e3e985e186e9f234d6260570f5a0271746503b3dce62668d7"} Oct 30 17:11:15.428314 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:15.427991 5052 scope.go:122] "RemoveContainer" containerID="554308610ee1f23add189bb1173774101b5b78e04fe078e83b3df4c8e85488dc" Oct 30 17:11:15.499372 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:15.499338 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-hwkgn"] Oct 30 17:11:15.512421 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:15.512395 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-hwkgn"] Oct 30 17:11:16.600114 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:16.600079 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96134e17-039f-4ea3-8522-ff54e3ab6797" path="/var/lib/kubelet/pods/96134e17-039f-4ea3-8522-ff54e3ab6797/volumes" Oct 30 17:11:18.841736 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:11:18.841697 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 17:11:20.389138 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:20.389106 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-0ae3fe-5b445b79c8-cqkdf" Oct 30 17:11:20.389593 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:20.389451 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ae3fe-598c75dd67-pjf8j" Oct 30 17:11:23.085623 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:23.085284 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cdwcj"] Oct 30 17:11:32.042819 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:32.042776 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-e1e830-openshift-default-5949c998b5-8zw7j"] Oct 30 17:11:32.043479 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:32.043448 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-e1e830-openshift-default-5949c998b5-8zw7j" podUID="91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a" containerName="istio-proxy" containerID="cri-o://a3a1c5e9354a9f5efe4de2dd6e65a05a0413dbd42e6cc0bb5e13b7e7ff6f115d" gracePeriod=30 Oct 30 17:11:35.645439 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:35.645404 5052 generic.go:356] "Generic (PLEG): container finished" podID="505e6f2b-634b-40ae-b5db-6ced8e1bf322" containerID="0e8e12f2d38c3d2d9f6d112c53f873cd3f077945188e70180365ce297c14af94" exitCode=0 Oct 30 17:11:35.645895 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:35.645475 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-de18d5-openshift-default-748c669-n82k6" event={"ID":"505e6f2b-634b-40ae-b5db-6ced8e1bf322","Type":"ContainerDied","Data":"0e8e12f2d38c3d2d9f6d112c53f873cd3f077945188e70180365ce297c14af94"} Oct 30 17:11:35.645942 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:35.645905 5052 scope.go:122] "RemoveContainer" containerID="0e8e12f2d38c3d2d9f6d112c53f873cd3f077945188e70180365ce297c14af94" Oct 30 17:11:35.647806 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:35.647779 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:11:35.647869 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:35.647851 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:11:35.647906 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:35.647882 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:11:36.660827 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:36.660786 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-de18d5-openshift-default-748c669-n82k6" event={"ID":"505e6f2b-634b-40ae-b5db-6ced8e1bf322","Type":"ContainerStarted","Data":"f95be2dbdb5436562e544029b65fb86d2d4b9d9ed635360bd2caa4e9b379fdb7"} Oct 30 17:11:37.053668 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:37.053302 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-de18d5-openshift-default-748c669-n82k6" Oct 30 17:11:37.054801 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:37.054767 5052 patch_prober.go:29] interesting pod/e2e-ipp-a-de18d5-openshift-default-748c669-n82k6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body= Oct 30 17:11:37.054966 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:37.054826 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-de18d5-openshift-default-748c669-n82k6" podUID="505e6f2b-634b-40ae-b5db-6ced8e1bf322" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" Oct 30 17:11:37.468796 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:37.468769 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-e1e830-openshift-default-5949c998b5-8zw7j" Oct 30 17:11:37.525943 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:37.525906 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a-istio-data\") pod \"91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a\" (UID: \"91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a\") " Oct 30 17:11:37.526167 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:37.526026 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a-workload-socket\") pod \"91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a\" (UID: \"91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a\") " Oct 30 17:11:37.526167 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:37.526056 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a-workload-certs\") pod \"91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a\" (UID: \"91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a\") " Oct 30 17:11:37.526167 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:37.526089 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a-istio-envoy\") pod \"91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a\" (UID: \"91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a\") " Oct 30 17:11:37.526354 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:37.526255 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a-istio-data" pod "91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a" (UID: "91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:11:37.526354 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:37.526302 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a-credential-socket\") pod \"91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a\" (UID: \"91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a\") " Oct 30 17:11:37.526354 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:37.526337 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a-workload-socket" pod "91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a" (UID: "91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:11:37.526529 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:37.526379 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a-istio-podinfo\") pod \"91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a\" (UID: \"91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a\") " Oct 30 17:11:37.526529 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:37.526450 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a-kube-api-access-72mlx\" (UniqueName: \"kubernetes.io/projected/91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a-kube-api-access-72mlx\") pod \"91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a\" (UID: \"91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a\") " Oct 30 17:11:37.526529 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:37.526478 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a-credential-socket" pod "91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a" (UID: "91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:11:37.526529 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:37.526508 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a-istio-token\" (UniqueName: \"kubernetes.io/projected/91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a-istio-token\") pod \"91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a\" (UID: \"91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a\") " Oct 30 17:11:37.526807 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:37.526538 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a-workload-certs" pod "91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a" (UID: "91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:11:37.526807 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:37.526569 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a-istiod-ca-cert\") pod \"91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a\" (UID: \"91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a\") " Oct 30 17:11:37.526921 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:37.526845 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a-istiod-ca-cert" pod "91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a" (UID: "91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:11:37.527428 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:37.527399 5052 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a-credential-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:11:37.527647 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:37.527431 5052 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a-istiod-ca-cert\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:11:37.527647 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:37.527448 5052 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a-istio-data\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:11:37.527647 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:37.527461 5052 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a-workload-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:11:37.527647 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:37.527474 5052 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a-workload-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:11:37.560101 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:37.560070 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a-istio-envoy" pod "91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a" (UID: "91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:11:37.560210 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:37.560121 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a-istio-token" pod "91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a" (UID: "91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:11:37.563775 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:37.563711 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a-istio-podinfo" pod "91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a" (UID: "91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:11:37.567167 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:37.567133 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a-kube-api-access-72mlx" pod "91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a" (UID: "91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a"). InnerVolumeSpecName "kube-api-access-72mlx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:11:37.654637 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:37.654574 5052 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a-istio-podinfo\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:11:37.654637 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:37.654632 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72mlx\" (UniqueName: \"kubernetes.io/projected/91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a-kube-api-access-72mlx\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:11:37.654879 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:37.654647 5052 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a-istio-token\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:11:37.654879 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:37.654662 5052 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a-istio-envoy\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:11:37.670814 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:37.670777 5052 generic.go:356] "Generic (PLEG): container finished" podID="91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a" containerID="a3a1c5e9354a9f5efe4de2dd6e65a05a0413dbd42e6cc0bb5e13b7e7ff6f115d" exitCode=0 Oct 30 17:11:37.671311 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:37.670829 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-e1e830-openshift-default-5949c998b5-8zw7j" Oct 30 17:11:37.671311 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:37.670852 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e1e830-openshift-default-5949c998b5-8zw7j" event={"ID":"91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a","Type":"ContainerDied","Data":"a3a1c5e9354a9f5efe4de2dd6e65a05a0413dbd42e6cc0bb5e13b7e7ff6f115d"} Oct 30 17:11:37.671311 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:37.670911 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e1e830-openshift-default-5949c998b5-8zw7j" event={"ID":"91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a","Type":"ContainerDied","Data":"15304aaaceecda0be1665ba4e52406b5d55973080bd144092cf37f926b1acefe"} Oct 30 17:11:37.671311 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:37.670933 5052 scope.go:122] "RemoveContainer" containerID="a3a1c5e9354a9f5efe4de2dd6e65a05a0413dbd42e6cc0bb5e13b7e7ff6f115d" Oct 30 17:11:37.683361 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:37.683337 5052 scope.go:122] "RemoveContainer" containerID="607b3811a04d0437b68e538055a6316e57a386c63ffe58f35f134c1cf8009053" Oct 30 17:11:37.694316 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:37.694297 5052 scope.go:122] "RemoveContainer" containerID="a3a1c5e9354a9f5efe4de2dd6e65a05a0413dbd42e6cc0bb5e13b7e7ff6f115d" Oct 30 17:11:37.694641 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:11:37.694616 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3a1c5e9354a9f5efe4de2dd6e65a05a0413dbd42e6cc0bb5e13b7e7ff6f115d\": container with ID starting with a3a1c5e9354a9f5efe4de2dd6e65a05a0413dbd42e6cc0bb5e13b7e7ff6f115d not found: ID does not exist" containerID="a3a1c5e9354a9f5efe4de2dd6e65a05a0413dbd42e6cc0bb5e13b7e7ff6f115d" Oct 30 17:11:37.694707 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:37.694647 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3a1c5e9354a9f5efe4de2dd6e65a05a0413dbd42e6cc0bb5e13b7e7ff6f115d"} err="failed to get container status \"a3a1c5e9354a9f5efe4de2dd6e65a05a0413dbd42e6cc0bb5e13b7e7ff6f115d\": rpc error: code = NotFound desc = could not find container \"a3a1c5e9354a9f5efe4de2dd6e65a05a0413dbd42e6cc0bb5e13b7e7ff6f115d\": container with ID starting with a3a1c5e9354a9f5efe4de2dd6e65a05a0413dbd42e6cc0bb5e13b7e7ff6f115d not found: ID does not exist" Oct 30 17:11:37.694707 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:37.694670 5052 scope.go:122] "RemoveContainer" containerID="607b3811a04d0437b68e538055a6316e57a386c63ffe58f35f134c1cf8009053" Oct 30 17:11:37.694951 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:11:37.694931 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"607b3811a04d0437b68e538055a6316e57a386c63ffe58f35f134c1cf8009053\": container with ID starting with 607b3811a04d0437b68e538055a6316e57a386c63ffe58f35f134c1cf8009053 not found: ID does not exist" containerID="607b3811a04d0437b68e538055a6316e57a386c63ffe58f35f134c1cf8009053" Oct 30 17:11:37.694995 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:37.694958 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"607b3811a04d0437b68e538055a6316e57a386c63ffe58f35f134c1cf8009053"} err="failed to get container status \"607b3811a04d0437b68e538055a6316e57a386c63ffe58f35f134c1cf8009053\": rpc error: code = NotFound desc = could not find container \"607b3811a04d0437b68e538055a6316e57a386c63ffe58f35f134c1cf8009053\": container with ID starting with 607b3811a04d0437b68e538055a6316e57a386c63ffe58f35f134c1cf8009053 not found: ID does not exist" Oct 30 17:11:37.765201 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:37.765149 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-e1e830-openshift-default-5949c998b5-8zw7j"] Oct 30 17:11:37.769769 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:37.769746 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-e1e830-openshift-default-5949c998b5-8zw7j"] Oct 30 17:11:38.019973 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:38.019941 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-de18d5-openshift-default-748c669-n82k6" Oct 30 17:11:38.054705 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:38.054477 5052 patch_prober.go:29] interesting pod/e2e-ipp-a-de18d5-openshift-default-748c669-n82k6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body= Oct 30 17:11:38.054705 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:38.054536 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-de18d5-openshift-default-748c669-n82k6" podUID="505e6f2b-634b-40ae-b5db-6ced8e1bf322" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" Oct 30 17:11:38.602171 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:38.602045 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a" path="/var/lib/kubelet/pods/91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a/volumes" Oct 30 17:11:39.053833 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:39.053355 5052 patch_prober.go:29] interesting pod/e2e-ipp-a-de18d5-openshift-default-748c669-n82k6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body= Oct 30 17:11:39.053833 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:39.053417 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-de18d5-openshift-default-748c669-n82k6" podUID="505e6f2b-634b-40ae-b5db-6ced8e1bf322" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" Oct 30 17:11:40.057011 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:40.056964 5052 patch_prober.go:29] interesting pod/e2e-ipp-a-de18d5-openshift-default-748c669-n82k6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body= Oct 30 17:11:40.057560 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:40.057027 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-de18d5-openshift-default-748c669-n82k6" podUID="505e6f2b-634b-40ae-b5db-6ced8e1bf322" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" Oct 30 17:11:41.058606 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:41.057835 5052 patch_prober.go:29] interesting pod/e2e-ipp-a-de18d5-openshift-default-748c669-n82k6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body= Oct 30 17:11:41.058606 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:41.057900 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-de18d5-openshift-default-748c669-n82k6" podUID="505e6f2b-634b-40ae-b5db-6ced8e1bf322" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" Oct 30 17:11:42.059802 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:42.059768 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-de18d5-openshift-default-748c669-n82k6" Oct 30 17:11:42.060910 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:42.060884 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-de18d5-openshift-default-748c669-n82k6" Oct 30 17:11:46.336164 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:46.336119 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-de18d5-7cb866fd58-n9vgg"] Oct 30 17:11:46.366707 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:46.366677 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-de18d5-5fcc8b7f9f-sl9fh"] Oct 30 17:11:48.840808 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:11:48.840770 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 17:11:51.337078 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:51.337018 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-de18d5-7cb866fd58-n9vgg" podUID="c0c628c1-f21f-4a83-a007-98dc19cbf65b" containerName="payload-processing" containerID="cri-o://4e961c468e339a53b0fe38b272f744427df37bd95f412660636378dc4b7bdd61" gracePeriod=25 Oct 30 17:11:51.367559 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:51.367530 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-de18d5-5fcc8b7f9f-sl9fh" podUID="1da97b6c-928b-4f33-950e-065cf95c366e" containerName="payload-pre-processing" containerID="cri-o://3084d47fe4817d3526059122282e8f9583b115590c999ec375bb96764240acf2" gracePeriod=25 Oct 30 17:11:51.715382 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:51.715356 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-de18d5-7cb866fd58-n9vgg" Oct 30 17:11:51.757685 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:51.757652 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-de18d5-5fcc8b7f9f-sl9fh" Oct 30 17:11:51.830085 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:51.830045 5052 generic.go:356] "Generic (PLEG): container finished" podID="c0c628c1-f21f-4a83-a007-98dc19cbf65b" containerID="4e961c468e339a53b0fe38b272f744427df37bd95f412660636378dc4b7bdd61" exitCode=0 Oct 30 17:11:51.830343 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:51.830102 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-de18d5-7cb866fd58-n9vgg" Oct 30 17:11:51.830343 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:51.830131 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-de18d5-7cb866fd58-n9vgg" event={"ID":"c0c628c1-f21f-4a83-a007-98dc19cbf65b","Type":"ContainerDied","Data":"4e961c468e339a53b0fe38b272f744427df37bd95f412660636378dc4b7bdd61"} Oct 30 17:11:51.830343 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:51.830206 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-de18d5-7cb866fd58-n9vgg" event={"ID":"c0c628c1-f21f-4a83-a007-98dc19cbf65b","Type":"ContainerDied","Data":"e21404f86757e8227b80a49db1af523a9045a2d60abf8883402cd6293ec6d5ad"} Oct 30 17:11:51.830343 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:51.830241 5052 scope.go:122] "RemoveContainer" containerID="4e961c468e339a53b0fe38b272f744427df37bd95f412660636378dc4b7bdd61" Oct 30 17:11:51.831937 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:51.831908 5052 generic.go:356] "Generic (PLEG): container finished" podID="1da97b6c-928b-4f33-950e-065cf95c366e" containerID="3084d47fe4817d3526059122282e8f9583b115590c999ec375bb96764240acf2" exitCode=0 Oct 30 17:11:51.832078 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:51.831961 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-de18d5-5fcc8b7f9f-sl9fh" Oct 30 17:11:51.832212 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:51.832153 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-de18d5-5fcc8b7f9f-sl9fh" event={"ID":"1da97b6c-928b-4f33-950e-065cf95c366e","Type":"ContainerDied","Data":"3084d47fe4817d3526059122282e8f9583b115590c999ec375bb96764240acf2"} Oct 30 17:11:51.832212 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:51.832194 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-de18d5-5fcc8b7f9f-sl9fh" event={"ID":"1da97b6c-928b-4f33-950e-065cf95c366e","Type":"ContainerDied","Data":"7b9d02fb3fbe7e6e7942e052d4f5b6bab75a6095c0239e3518276c73069b1c4e"} Oct 30 17:11:51.842263 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:51.842243 5052 scope.go:122] "RemoveContainer" containerID="4e961c468e339a53b0fe38b272f744427df37bd95f412660636378dc4b7bdd61" Oct 30 17:11:51.842553 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:11:51.842537 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e961c468e339a53b0fe38b272f744427df37bd95f412660636378dc4b7bdd61\": container with ID starting with 4e961c468e339a53b0fe38b272f744427df37bd95f412660636378dc4b7bdd61 not found: ID does not exist" containerID="4e961c468e339a53b0fe38b272f744427df37bd95f412660636378dc4b7bdd61" Oct 30 17:11:51.842634 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:51.842562 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e961c468e339a53b0fe38b272f744427df37bd95f412660636378dc4b7bdd61"} err="failed to get container status \"4e961c468e339a53b0fe38b272f744427df37bd95f412660636378dc4b7bdd61\": rpc error: code = NotFound desc = could not find container \"4e961c468e339a53b0fe38b272f744427df37bd95f412660636378dc4b7bdd61\": container with ID starting with 4e961c468e339a53b0fe38b272f744427df37bd95f412660636378dc4b7bdd61 not found: ID does not exist" Oct 30 17:11:51.842634 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:51.842594 5052 scope.go:122] "RemoveContainer" containerID="3084d47fe4817d3526059122282e8f9583b115590c999ec375bb96764240acf2" Oct 30 17:11:51.853312 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:51.853287 5052 scope.go:122] "RemoveContainer" containerID="3084d47fe4817d3526059122282e8f9583b115590c999ec375bb96764240acf2" Oct 30 17:11:51.853691 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:11:51.853670 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3084d47fe4817d3526059122282e8f9583b115590c999ec375bb96764240acf2\": container with ID starting with 3084d47fe4817d3526059122282e8f9583b115590c999ec375bb96764240acf2 not found: ID does not exist" containerID="3084d47fe4817d3526059122282e8f9583b115590c999ec375bb96764240acf2" Oct 30 17:11:51.853789 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:51.853700 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3084d47fe4817d3526059122282e8f9583b115590c999ec375bb96764240acf2"} err="failed to get container status \"3084d47fe4817d3526059122282e8f9583b115590c999ec375bb96764240acf2\": rpc error: code = NotFound desc = could not find container \"3084d47fe4817d3526059122282e8f9583b115590c999ec375bb96764240acf2\": container with ID starting with 3084d47fe4817d3526059122282e8f9583b115590c999ec375bb96764240acf2 not found: ID does not exist" Oct 30 17:11:51.860665 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:51.860639 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0c628c1-f21f-4a83-a007-98dc19cbf65b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0c628c1-f21f-4a83-a007-98dc19cbf65b-plugins-config-volume\") pod \"c0c628c1-f21f-4a83-a007-98dc19cbf65b\" (UID: \"c0c628c1-f21f-4a83-a007-98dc19cbf65b\") " Oct 30 17:11:51.860841 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:51.860820 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0c628c1-f21f-4a83-a007-98dc19cbf65b-kube-api-access-r4wx7\" (UniqueName: \"kubernetes.io/projected/c0c628c1-f21f-4a83-a007-98dc19cbf65b-kube-api-access-r4wx7\") pod \"c0c628c1-f21f-4a83-a007-98dc19cbf65b\" (UID: \"c0c628c1-f21f-4a83-a007-98dc19cbf65b\") " Oct 30 17:11:51.860983 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:51.860967 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1da97b6c-928b-4f33-950e-065cf95c366e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1da97b6c-928b-4f33-950e-065cf95c366e-plugins-config-volume\") pod \"1da97b6c-928b-4f33-950e-065cf95c366e\" (UID: \"1da97b6c-928b-4f33-950e-065cf95c366e\") " Oct 30 17:11:51.861066 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:51.860980 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0c628c1-f21f-4a83-a007-98dc19cbf65b-plugins-config-volume" pod "c0c628c1-f21f-4a83-a007-98dc19cbf65b" (UID: "c0c628c1-f21f-4a83-a007-98dc19cbf65b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:11:51.861066 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:51.861018 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1da97b6c-928b-4f33-950e-065cf95c366e-kube-api-access-5zzm8\" (UniqueName: \"kubernetes.io/projected/1da97b6c-928b-4f33-950e-065cf95c366e-kube-api-access-5zzm8\") pod \"1da97b6c-928b-4f33-950e-065cf95c366e\" (UID: \"1da97b6c-928b-4f33-950e-065cf95c366e\") " Oct 30 17:11:51.861697 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:51.861652 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1da97b6c-928b-4f33-950e-065cf95c366e-plugins-config-volume" pod "1da97b6c-928b-4f33-950e-065cf95c366e" (UID: "1da97b6c-928b-4f33-950e-065cf95c366e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:11:51.861832 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:51.861808 5052 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0c628c1-f21f-4a83-a007-98dc19cbf65b-plugins-config-volume\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:11:51.861909 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:51.861838 5052 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1da97b6c-928b-4f33-950e-065cf95c366e-plugins-config-volume\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:11:51.863694 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:51.863670 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1da97b6c-928b-4f33-950e-065cf95c366e-kube-api-access-5zzm8" pod "1da97b6c-928b-4f33-950e-065cf95c366e" (UID: "1da97b6c-928b-4f33-950e-065cf95c366e"). InnerVolumeSpecName "kube-api-access-5zzm8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:11:51.864000 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:51.863969 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0c628c1-f21f-4a83-a007-98dc19cbf65b-kube-api-access-r4wx7" pod "c0c628c1-f21f-4a83-a007-98dc19cbf65b" (UID: "c0c628c1-f21f-4a83-a007-98dc19cbf65b"). InnerVolumeSpecName "kube-api-access-r4wx7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:11:51.963532 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:51.963449 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4wx7\" (UniqueName: \"kubernetes.io/projected/c0c628c1-f21f-4a83-a007-98dc19cbf65b-kube-api-access-r4wx7\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:11:51.963532 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:51.963480 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zzm8\" (UniqueName: \"kubernetes.io/projected/1da97b6c-928b-4f33-950e-065cf95c366e-kube-api-access-5zzm8\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:11:52.186380 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:52.186338 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-de18d5-5fcc8b7f9f-sl9fh"] Oct 30 17:11:52.199565 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:52.199538 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-de18d5-5fcc8b7f9f-sl9fh"] Oct 30 17:11:52.235494 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:52.235431 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-de18d5-7cb866fd58-n9vgg"] Oct 30 17:11:52.239108 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:52.239086 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-de18d5-7cb866fd58-n9vgg"] Oct 30 17:11:52.599564 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:52.599488 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1da97b6c-928b-4f33-950e-065cf95c366e" path="/var/lib/kubelet/pods/1da97b6c-928b-4f33-950e-065cf95c366e/volumes" Oct 30 17:11:52.599962 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:52.599947 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0c628c1-f21f-4a83-a007-98dc19cbf65b" path="/var/lib/kubelet/pods/c0c628c1-f21f-4a83-a007-98dc19cbf65b/volumes" Oct 30 17:11:52.953095 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:52.953062 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-e43400-5d9d988d86-hvrpq"] Oct 30 17:11:57.954226 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:57.954180 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-e43400-5d9d988d86-hvrpq" podUID="bd4ea853-d161-4a29-b561-cad7e8fe2951" containerName="payload-processing" containerID="cri-o://a145edb3fb03611ca73689d0fe871829ea76d234b167004b71a5a7c24f806c7c" gracePeriod=25 Oct 30 17:11:58.720008 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:58.719657 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-e43400-5d9d988d86-hvrpq" Oct 30 17:11:58.732437 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:58.732394 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd4ea853-d161-4a29-b561-cad7e8fe2951-kube-api-access-cd8v5\" (UniqueName: \"kubernetes.io/projected/bd4ea853-d161-4a29-b561-cad7e8fe2951-kube-api-access-cd8v5\") pod \"bd4ea853-d161-4a29-b561-cad7e8fe2951\" (UID: \"bd4ea853-d161-4a29-b561-cad7e8fe2951\") " Oct 30 17:11:58.733154 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:58.733119 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd4ea853-d161-4a29-b561-cad7e8fe2951-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd4ea853-d161-4a29-b561-cad7e8fe2951-plugins-config-volume\") pod \"bd4ea853-d161-4a29-b561-cad7e8fe2951\" (UID: \"bd4ea853-d161-4a29-b561-cad7e8fe2951\") " Oct 30 17:11:58.744669 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:58.744565 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd4ea853-d161-4a29-b561-cad7e8fe2951-plugins-config-volume" pod "bd4ea853-d161-4a29-b561-cad7e8fe2951" (UID: "bd4ea853-d161-4a29-b561-cad7e8fe2951"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:11:58.750680 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:58.750640 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd4ea853-d161-4a29-b561-cad7e8fe2951-kube-api-access-cd8v5" pod "bd4ea853-d161-4a29-b561-cad7e8fe2951" (UID: "bd4ea853-d161-4a29-b561-cad7e8fe2951"). InnerVolumeSpecName "kube-api-access-cd8v5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:11:58.837853 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:58.837809 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cd8v5\" (UniqueName: \"kubernetes.io/projected/bd4ea853-d161-4a29-b561-cad7e8fe2951-kube-api-access-cd8v5\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:11:58.837853 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:58.837855 5052 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd4ea853-d161-4a29-b561-cad7e8fe2951-plugins-config-volume\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:11:58.889895 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:58.889842 5052 generic.go:356] "Generic (PLEG): container finished" podID="bd4ea853-d161-4a29-b561-cad7e8fe2951" containerID="a145edb3fb03611ca73689d0fe871829ea76d234b167004b71a5a7c24f806c7c" exitCode=0 Oct 30 17:11:58.890096 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:58.889907 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-e43400-5d9d988d86-hvrpq" Oct 30 17:11:58.890096 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:58.889917 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-e43400-5d9d988d86-hvrpq" event={"ID":"bd4ea853-d161-4a29-b561-cad7e8fe2951","Type":"ContainerDied","Data":"a145edb3fb03611ca73689d0fe871829ea76d234b167004b71a5a7c24f806c7c"} Oct 30 17:11:58.890096 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:58.890011 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-e43400-5d9d988d86-hvrpq" event={"ID":"bd4ea853-d161-4a29-b561-cad7e8fe2951","Type":"ContainerDied","Data":"831f62f0faf77e7d2b691c7f67be02a7bd8c5a05ae52f23e7fd151fbe5173890"} Oct 30 17:11:58.890096 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:58.890042 5052 scope.go:122] "RemoveContainer" containerID="a145edb3fb03611ca73689d0fe871829ea76d234b167004b71a5a7c24f806c7c" Oct 30 17:11:58.917282 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:58.904558 5052 scope.go:122] "RemoveContainer" containerID="a145edb3fb03611ca73689d0fe871829ea76d234b167004b71a5a7c24f806c7c" Oct 30 17:11:58.917282 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:11:58.904983 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a145edb3fb03611ca73689d0fe871829ea76d234b167004b71a5a7c24f806c7c\": container with ID starting with a145edb3fb03611ca73689d0fe871829ea76d234b167004b71a5a7c24f806c7c not found: ID does not exist" containerID="a145edb3fb03611ca73689d0fe871829ea76d234b167004b71a5a7c24f806c7c" Oct 30 17:11:58.917282 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:58.905024 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a145edb3fb03611ca73689d0fe871829ea76d234b167004b71a5a7c24f806c7c"} err="failed to get container status \"a145edb3fb03611ca73689d0fe871829ea76d234b167004b71a5a7c24f806c7c\": rpc error: code = NotFound desc = could not find container \"a145edb3fb03611ca73689d0fe871829ea76d234b167004b71a5a7c24f806c7c\": container with ID starting with a145edb3fb03611ca73689d0fe871829ea76d234b167004b71a5a7c24f806c7c not found: ID does not exist" Oct 30 17:11:59.013735 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:59.013649 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-e43400-5d9d988d86-hvrpq"] Oct 30 17:11:59.015994 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:11:59.015965 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-e43400-5d9d988d86-hvrpq"] Oct 30 17:12:00.600111 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:00.600069 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd4ea853-d161-4a29-b561-cad7e8fe2951" path="/var/lib/kubelet/pods/bd4ea853-d161-4a29-b561-cad7e8fe2951/volumes" Oct 30 17:12:08.851695 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:08.851654 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-e43400-openshift-default-7cf9477785-5sxxd"] Oct 30 17:12:08.852198 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:08.851904 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w5-e43400-openshift-default-7cf9477785-5sxxd" podUID="050171fd-aa7a-47db-8cea-e7ed38828148" containerName="istio-proxy" containerID="cri-o://40a1f368d1a55fda00ebc57f9df1f081aaa213295279362ea35926bbc1ea789d" gracePeriod=30 Oct 30 17:12:09.617020 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:09.616983 5052 patch_prober.go:29] interesting pod/e2e-shared-a-w5-e43400-openshift-default-7cf9477785-5sxxd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 17:12:09.617203 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:09.617047 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w5-e43400-openshift-default-7cf9477785-5sxxd" podUID="050171fd-aa7a-47db-8cea-e7ed38828148" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 17:12:14.104091 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:14.104053 5052 generic.go:356] "Generic (PLEG): container finished" podID="050171fd-aa7a-47db-8cea-e7ed38828148" containerID="40a1f368d1a55fda00ebc57f9df1f081aaa213295279362ea35926bbc1ea789d" exitCode=0 Oct 30 17:12:14.105074 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:14.104184 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-e43400-openshift-default-7cf9477785-5sxxd" event={"ID":"050171fd-aa7a-47db-8cea-e7ed38828148","Type":"ContainerDied","Data":"40a1f368d1a55fda00ebc57f9df1f081aaa213295279362ea35926bbc1ea789d"} Oct 30 17:12:14.105227 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:14.105118 5052 scope.go:122] "RemoveContainer" containerID="789a1f1638487db7213badfc4c9b74afaf0324538c0e4019bfb4aefc2af45387" Oct 30 17:12:14.200374 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:14.200347 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-e43400-openshift-default-7cf9477785-5sxxd" Oct 30 17:12:14.254527 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:14.254496 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/050171fd-aa7a-47db-8cea-e7ed38828148-kube-api-access-75f49\" (UniqueName: \"kubernetes.io/projected/050171fd-aa7a-47db-8cea-e7ed38828148-kube-api-access-75f49\") pod \"050171fd-aa7a-47db-8cea-e7ed38828148\" (UID: \"050171fd-aa7a-47db-8cea-e7ed38828148\") " Oct 30 17:12:14.254685 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:14.254630 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/050171fd-aa7a-47db-8cea-e7ed38828148-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/050171fd-aa7a-47db-8cea-e7ed38828148-istio-data\") pod \"050171fd-aa7a-47db-8cea-e7ed38828148\" (UID: \"050171fd-aa7a-47db-8cea-e7ed38828148\") " Oct 30 17:12:14.254734 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:14.254699 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/050171fd-aa7a-47db-8cea-e7ed38828148-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/050171fd-aa7a-47db-8cea-e7ed38828148-istiod-ca-cert\") pod \"050171fd-aa7a-47db-8cea-e7ed38828148\" (UID: \"050171fd-aa7a-47db-8cea-e7ed38828148\") " Oct 30 17:12:14.254773 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:14.254735 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/050171fd-aa7a-47db-8cea-e7ed38828148-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/050171fd-aa7a-47db-8cea-e7ed38828148-credential-socket\") pod \"050171fd-aa7a-47db-8cea-e7ed38828148\" (UID: \"050171fd-aa7a-47db-8cea-e7ed38828148\") " Oct 30 17:12:14.254834 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:14.254818 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/050171fd-aa7a-47db-8cea-e7ed38828148-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/050171fd-aa7a-47db-8cea-e7ed38828148-workload-socket\") pod \"050171fd-aa7a-47db-8cea-e7ed38828148\" (UID: \"050171fd-aa7a-47db-8cea-e7ed38828148\") " Oct 30 17:12:14.254928 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:14.254911 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/050171fd-aa7a-47db-8cea-e7ed38828148-istio-token\" (UniqueName: \"kubernetes.io/projected/050171fd-aa7a-47db-8cea-e7ed38828148-istio-token\") pod \"050171fd-aa7a-47db-8cea-e7ed38828148\" (UID: \"050171fd-aa7a-47db-8cea-e7ed38828148\") " Oct 30 17:12:14.255040 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:14.254908 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/050171fd-aa7a-47db-8cea-e7ed38828148-istio-data" pod "050171fd-aa7a-47db-8cea-e7ed38828148" (UID: "050171fd-aa7a-47db-8cea-e7ed38828148"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:12:14.255162 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:14.255023 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/050171fd-aa7a-47db-8cea-e7ed38828148-credential-socket" pod "050171fd-aa7a-47db-8cea-e7ed38828148" (UID: "050171fd-aa7a-47db-8cea-e7ed38828148"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:12:14.255162 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:14.255095 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/050171fd-aa7a-47db-8cea-e7ed38828148-workload-socket" pod "050171fd-aa7a-47db-8cea-e7ed38828148" (UID: "050171fd-aa7a-47db-8cea-e7ed38828148"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:12:14.255162 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:14.255088 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/050171fd-aa7a-47db-8cea-e7ed38828148-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/050171fd-aa7a-47db-8cea-e7ed38828148-istio-envoy\") pod \"050171fd-aa7a-47db-8cea-e7ed38828148\" (UID: \"050171fd-aa7a-47db-8cea-e7ed38828148\") " Oct 30 17:12:14.255417 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:14.255131 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/050171fd-aa7a-47db-8cea-e7ed38828148-istiod-ca-cert" pod "050171fd-aa7a-47db-8cea-e7ed38828148" (UID: "050171fd-aa7a-47db-8cea-e7ed38828148"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:12:14.255417 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:14.255213 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/050171fd-aa7a-47db-8cea-e7ed38828148-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/050171fd-aa7a-47db-8cea-e7ed38828148-workload-certs\") pod \"050171fd-aa7a-47db-8cea-e7ed38828148\" (UID: \"050171fd-aa7a-47db-8cea-e7ed38828148\") " Oct 30 17:12:14.255417 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:14.255267 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/050171fd-aa7a-47db-8cea-e7ed38828148-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/050171fd-aa7a-47db-8cea-e7ed38828148-istio-podinfo\") pod \"050171fd-aa7a-47db-8cea-e7ed38828148\" (UID: \"050171fd-aa7a-47db-8cea-e7ed38828148\") " Oct 30 17:12:14.255612 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:14.255461 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/050171fd-aa7a-47db-8cea-e7ed38828148-workload-certs" pod "050171fd-aa7a-47db-8cea-e7ed38828148" (UID: "050171fd-aa7a-47db-8cea-e7ed38828148"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:12:14.255964 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:14.255943 5052 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/050171fd-aa7a-47db-8cea-e7ed38828148-istio-data\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:12:14.256053 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:14.255967 5052 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/050171fd-aa7a-47db-8cea-e7ed38828148-istiod-ca-cert\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:12:14.256053 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:14.255979 5052 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/050171fd-aa7a-47db-8cea-e7ed38828148-credential-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:12:14.256053 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:14.255988 5052 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/050171fd-aa7a-47db-8cea-e7ed38828148-workload-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:12:14.256053 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:14.255997 5052 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/050171fd-aa7a-47db-8cea-e7ed38828148-workload-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:12:14.257249 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:14.257220 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/050171fd-aa7a-47db-8cea-e7ed38828148-istio-token" pod "050171fd-aa7a-47db-8cea-e7ed38828148" (UID: "050171fd-aa7a-47db-8cea-e7ed38828148"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:12:14.257337 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:14.257248 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/050171fd-aa7a-47db-8cea-e7ed38828148-istio-podinfo" pod "050171fd-aa7a-47db-8cea-e7ed38828148" (UID: "050171fd-aa7a-47db-8cea-e7ed38828148"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:12:14.257381 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:14.257362 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/050171fd-aa7a-47db-8cea-e7ed38828148-kube-api-access-75f49" pod "050171fd-aa7a-47db-8cea-e7ed38828148" (UID: "050171fd-aa7a-47db-8cea-e7ed38828148"). InnerVolumeSpecName "kube-api-access-75f49". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:12:14.257430 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:14.257391 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/050171fd-aa7a-47db-8cea-e7ed38828148-istio-envoy" pod "050171fd-aa7a-47db-8cea-e7ed38828148" (UID: "050171fd-aa7a-47db-8cea-e7ed38828148"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:12:14.357656 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:14.357623 5052 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/050171fd-aa7a-47db-8cea-e7ed38828148-istio-token\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:12:14.357656 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:14.357655 5052 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/050171fd-aa7a-47db-8cea-e7ed38828148-istio-envoy\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:12:14.357870 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:14.357665 5052 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/050171fd-aa7a-47db-8cea-e7ed38828148-istio-podinfo\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:12:14.357870 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:14.357674 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75f49\" (UniqueName: \"kubernetes.io/projected/050171fd-aa7a-47db-8cea-e7ed38828148-kube-api-access-75f49\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:12:15.113132 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:15.113096 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-e43400-openshift-default-7cf9477785-5sxxd" Oct 30 17:12:15.113574 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:15.113125 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-e43400-openshift-default-7cf9477785-5sxxd" event={"ID":"050171fd-aa7a-47db-8cea-e7ed38828148","Type":"ContainerDied","Data":"490c45889a25474d2392f470f2c6922d37b118f68b374320c181306d773bbd86"} Oct 30 17:12:15.113574 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:15.113186 5052 scope.go:122] "RemoveContainer" containerID="40a1f368d1a55fda00ebc57f9df1f081aaa213295279362ea35926bbc1ea789d" Oct 30 17:12:15.187883 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:15.187852 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-e43400-openshift-default-7cf9477785-5sxxd"] Oct 30 17:12:15.201866 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:15.201835 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-e43400-openshift-default-7cf9477785-5sxxd"] Oct 30 17:12:16.601444 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:16.601412 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="050171fd-aa7a-47db-8cea-e7ed38828148" path="/var/lib/kubelet/pods/050171fd-aa7a-47db-8cea-e7ed38828148/volumes" Oct 30 17:12:16.868319 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:16.868231 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-g6lll"] Oct 30 17:12:16.869002 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:16.868982 5052 cpu_manager.go:414] "containerMap: removing container" podUID="96134e17-039f-4ea3-8522-ff54e3ab6797" containerName="istio-proxy" Oct 30 17:12:16.869002 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:16.869000 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96134e17-039f-4ea3-8522-ff54e3ab6797" containerName="istio-proxy" Oct 30 17:12:16.869187 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:16.869009 5052 cpu_manager.go:414] "containerMap: removing container" podUID="1da97b6c-928b-4f33-950e-065cf95c366e" containerName="payload-pre-processing" Oct 30 17:12:16.869187 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:16.869015 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1da97b6c-928b-4f33-950e-065cf95c366e" containerName="payload-pre-processing" Oct 30 17:12:16.869187 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:16.869027 5052 cpu_manager.go:414] "containerMap: removing container" podUID="050171fd-aa7a-47db-8cea-e7ed38828148" containerName="istio-proxy" Oct 30 17:12:16.869187 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:16.869032 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="050171fd-aa7a-47db-8cea-e7ed38828148" containerName="istio-proxy" Oct 30 17:12:16.869187 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:16.869041 5052 cpu_manager.go:414] "containerMap: removing container" podUID="91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a" containerName="istio-proxy" Oct 30 17:12:16.869187 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:16.869046 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a" containerName="istio-proxy" Oct 30 17:12:16.869187 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:16.869089 5052 cpu_manager.go:414] "containerMap: removing container" podUID="bd4ea853-d161-4a29-b561-cad7e8fe2951" containerName="payload-processing" Oct 30 17:12:16.869187 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:16.869094 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd4ea853-d161-4a29-b561-cad7e8fe2951" containerName="payload-processing" Oct 30 17:12:16.869187 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:16.869105 5052 cpu_manager.go:414] "containerMap: removing container" podUID="c0c628c1-f21f-4a83-a007-98dc19cbf65b" containerName="payload-processing" Oct 30 17:12:16.869187 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:16.869110 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0c628c1-f21f-4a83-a007-98dc19cbf65b" containerName="payload-processing" Oct 30 17:12:16.869651 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:16.869203 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a" containerName="istio-proxy" Oct 30 17:12:16.869651 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:16.869216 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd4ea853-d161-4a29-b561-cad7e8fe2951" containerName="payload-processing" Oct 30 17:12:16.869651 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:16.869223 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="050171fd-aa7a-47db-8cea-e7ed38828148" containerName="istio-proxy" Oct 30 17:12:16.869651 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:16.869228 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0c628c1-f21f-4a83-a007-98dc19cbf65b" containerName="payload-processing" Oct 30 17:12:16.869651 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:16.869236 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="96134e17-039f-4ea3-8522-ff54e3ab6797" containerName="istio-proxy" Oct 30 17:12:16.869651 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:16.869243 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="1da97b6c-928b-4f33-950e-065cf95c366e" containerName="payload-pre-processing" Oct 30 17:12:16.869651 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:16.869249 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="050171fd-aa7a-47db-8cea-e7ed38828148" containerName="istio-proxy" Oct 30 17:12:16.874438 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:16.874419 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-g6lll" Oct 30 17:12:16.898018 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:16.897986 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-g6lll"] Oct 30 17:12:16.979446 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:16.979416 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-55e83e-openshift-default-dockercfg-w5hdx\"" Oct 30 17:12:17.027928 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:17.027888 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0057b70d-3924-4e84-87f5-4cfc7e7babf5-istio-token\") pod \"e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-g6lll\" (UID: \"0057b70d-3924-4e84-87f5-4cfc7e7babf5\") " pod="openshift-ingress/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-g6lll" Oct 30 17:12:17.028157 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:17.028133 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0057b70d-3924-4e84-87f5-4cfc7e7babf5-istio-envoy\") pod \"e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-g6lll\" (UID: \"0057b70d-3924-4e84-87f5-4cfc7e7babf5\") " pod="openshift-ingress/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-g6lll" Oct 30 17:12:17.028234 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:17.028191 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0057b70d-3924-4e84-87f5-4cfc7e7babf5-istiod-ca-cert\") pod \"e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-g6lll\" (UID: \"0057b70d-3924-4e84-87f5-4cfc7e7babf5\") " pod="openshift-ingress/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-g6lll" Oct 30 17:12:17.028350 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:17.028326 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0057b70d-3924-4e84-87f5-4cfc7e7babf5-istio-podinfo\") pod \"e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-g6lll\" (UID: \"0057b70d-3924-4e84-87f5-4cfc7e7babf5\") " pod="openshift-ingress/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-g6lll" Oct 30 17:12:17.028427 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:17.028409 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0057b70d-3924-4e84-87f5-4cfc7e7babf5-istio-data\") pod \"e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-g6lll\" (UID: \"0057b70d-3924-4e84-87f5-4cfc7e7babf5\") " pod="openshift-ingress/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-g6lll" Oct 30 17:12:17.028493 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:17.028480 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zd5h2\" (UniqueName: \"kubernetes.io/projected/0057b70d-3924-4e84-87f5-4cfc7e7babf5-kube-api-access-zd5h2\") pod \"e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-g6lll\" (UID: \"0057b70d-3924-4e84-87f5-4cfc7e7babf5\") " pod="openshift-ingress/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-g6lll" Oct 30 17:12:17.028662 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:17.028645 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0057b70d-3924-4e84-87f5-4cfc7e7babf5-workload-socket\") pod \"e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-g6lll\" (UID: \"0057b70d-3924-4e84-87f5-4cfc7e7babf5\") " pod="openshift-ingress/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-g6lll" Oct 30 17:12:17.028753 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:17.028681 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0057b70d-3924-4e84-87f5-4cfc7e7babf5-credential-socket\") pod \"e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-g6lll\" (UID: \"0057b70d-3924-4e84-87f5-4cfc7e7babf5\") " pod="openshift-ingress/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-g6lll" Oct 30 17:12:17.028753 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:17.028705 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0057b70d-3924-4e84-87f5-4cfc7e7babf5-workload-certs\") pod \"e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-g6lll\" (UID: \"0057b70d-3924-4e84-87f5-4cfc7e7babf5\") " pod="openshift-ingress/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-g6lll" Oct 30 17:12:17.130492 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:17.130410 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0057b70d-3924-4e84-87f5-4cfc7e7babf5-istio-envoy\") pod \"e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-g6lll\" (UID: \"0057b70d-3924-4e84-87f5-4cfc7e7babf5\") " pod="openshift-ingress/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-g6lll" Oct 30 17:12:17.130492 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:17.130457 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0057b70d-3924-4e84-87f5-4cfc7e7babf5-istiod-ca-cert\") pod \"e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-g6lll\" (UID: \"0057b70d-3924-4e84-87f5-4cfc7e7babf5\") " pod="openshift-ingress/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-g6lll" Oct 30 17:12:17.130772 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:17.130498 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0057b70d-3924-4e84-87f5-4cfc7e7babf5-istio-podinfo\") pod \"e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-g6lll\" (UID: \"0057b70d-3924-4e84-87f5-4cfc7e7babf5\") " pod="openshift-ingress/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-g6lll" Oct 30 17:12:17.130772 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:17.130535 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0057b70d-3924-4e84-87f5-4cfc7e7babf5-istio-data\") pod \"e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-g6lll\" (UID: \"0057b70d-3924-4e84-87f5-4cfc7e7babf5\") " pod="openshift-ingress/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-g6lll" Oct 30 17:12:17.130772 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:17.130595 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zd5h2\" (UniqueName: \"kubernetes.io/projected/0057b70d-3924-4e84-87f5-4cfc7e7babf5-kube-api-access-zd5h2\") pod \"e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-g6lll\" (UID: \"0057b70d-3924-4e84-87f5-4cfc7e7babf5\") " pod="openshift-ingress/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-g6lll" Oct 30 17:12:17.130772 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:17.130683 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0057b70d-3924-4e84-87f5-4cfc7e7babf5-workload-socket\") pod \"e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-g6lll\" (UID: \"0057b70d-3924-4e84-87f5-4cfc7e7babf5\") " pod="openshift-ingress/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-g6lll" Oct 30 17:12:17.130772 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:17.130716 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0057b70d-3924-4e84-87f5-4cfc7e7babf5-credential-socket\") pod \"e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-g6lll\" (UID: \"0057b70d-3924-4e84-87f5-4cfc7e7babf5\") " pod="openshift-ingress/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-g6lll" Oct 30 17:12:17.130772 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:17.130741 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0057b70d-3924-4e84-87f5-4cfc7e7babf5-workload-certs\") pod \"e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-g6lll\" (UID: \"0057b70d-3924-4e84-87f5-4cfc7e7babf5\") " pod="openshift-ingress/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-g6lll" Oct 30 17:12:17.131081 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:17.130840 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0057b70d-3924-4e84-87f5-4cfc7e7babf5-istio-token\") pod \"e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-g6lll\" (UID: \"0057b70d-3924-4e84-87f5-4cfc7e7babf5\") " pod="openshift-ingress/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-g6lll" Oct 30 17:12:17.131134 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:17.131084 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0057b70d-3924-4e84-87f5-4cfc7e7babf5-workload-socket\") pod \"e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-g6lll\" (UID: \"0057b70d-3924-4e84-87f5-4cfc7e7babf5\") " pod="openshift-ingress/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-g6lll" Oct 30 17:12:17.131134 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:17.131104 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0057b70d-3924-4e84-87f5-4cfc7e7babf5-istio-data\") pod \"e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-g6lll\" (UID: \"0057b70d-3924-4e84-87f5-4cfc7e7babf5\") " pod="openshift-ingress/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-g6lll" Oct 30 17:12:17.131366 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:17.131343 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0057b70d-3924-4e84-87f5-4cfc7e7babf5-credential-socket\") pod \"e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-g6lll\" (UID: \"0057b70d-3924-4e84-87f5-4cfc7e7babf5\") " pod="openshift-ingress/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-g6lll" Oct 30 17:12:17.131366 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:17.131358 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0057b70d-3924-4e84-87f5-4cfc7e7babf5-workload-certs\") pod \"e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-g6lll\" (UID: \"0057b70d-3924-4e84-87f5-4cfc7e7babf5\") " pod="openshift-ingress/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-g6lll" Oct 30 17:12:17.131516 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:17.131490 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0057b70d-3924-4e84-87f5-4cfc7e7babf5-istiod-ca-cert\") pod \"e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-g6lll\" (UID: \"0057b70d-3924-4e84-87f5-4cfc7e7babf5\") " pod="openshift-ingress/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-g6lll" Oct 30 17:12:17.132953 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:17.132935 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0057b70d-3924-4e84-87f5-4cfc7e7babf5-istio-envoy\") pod \"e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-g6lll\" (UID: \"0057b70d-3924-4e84-87f5-4cfc7e7babf5\") " pod="openshift-ingress/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-g6lll" Oct 30 17:12:17.133611 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:17.133568 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0057b70d-3924-4e84-87f5-4cfc7e7babf5-istio-podinfo\") pod \"e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-g6lll\" (UID: \"0057b70d-3924-4e84-87f5-4cfc7e7babf5\") " pod="openshift-ingress/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-g6lll" Oct 30 17:12:17.166322 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:17.166292 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0057b70d-3924-4e84-87f5-4cfc7e7babf5-istio-token\") pod \"e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-g6lll\" (UID: \"0057b70d-3924-4e84-87f5-4cfc7e7babf5\") " pod="openshift-ingress/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-g6lll" Oct 30 17:12:17.166840 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:17.166817 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zd5h2\" (UniqueName: \"kubernetes.io/projected/0057b70d-3924-4e84-87f5-4cfc7e7babf5-kube-api-access-zd5h2\") pod \"e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-g6lll\" (UID: \"0057b70d-3924-4e84-87f5-4cfc7e7babf5\") " pod="openshift-ingress/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-g6lll" Oct 30 17:12:17.188774 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:17.188742 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-g6lll" Oct 30 17:12:17.516513 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:17.516486 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-g6lll"] Oct 30 17:12:17.517106 ip-10-0-131-136 kubenswrapper[5052]: W1030 17:12:17.517080 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0057b70d_3924_4e84_87f5_4cfc7e7babf5.slice/crio-241da2d4113cdfd665baa79a0b454d61186610d354f4e6555076577405bec0cf WatchSource:0}: Error finding container 241da2d4113cdfd665baa79a0b454d61186610d354f4e6555076577405bec0cf: Status 404 returned error can't find the container with id 241da2d4113cdfd665baa79a0b454d61186610d354f4e6555076577405bec0cf Oct 30 17:12:17.519981 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:17.519947 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:12:17.520052 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:17.520015 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:12:17.520052 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:17.520044 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:12:18.152604 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:18.152217 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-g6lll" event={"ID":"0057b70d-3924-4e84-87f5-4cfc7e7babf5","Type":"ContainerStarted","Data":"48313a13198f7585ade8b6eff18fc89e7ac1427dc5757db7a6e919cbeeedb20c"} Oct 30 17:12:18.152604 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:18.152290 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-g6lll" event={"ID":"0057b70d-3924-4e84-87f5-4cfc7e7babf5","Type":"ContainerStarted","Data":"241da2d4113cdfd665baa79a0b454d61186610d354f4e6555076577405bec0cf"} Oct 30 17:12:18.189548 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:18.189512 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-g6lll" Oct 30 17:12:18.191269 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:18.191240 5052 patch_prober.go:29] interesting pod/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-g6lll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body= Oct 30 17:12:18.191417 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:18.191303 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-g6lll" podUID="0057b70d-3924-4e84-87f5-4cfc7e7babf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" Oct 30 17:12:18.226388 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:18.226337 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-g6lll" podStartSLOduration=2.226325378 podStartE2EDuration="2.226325378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:12:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:12:18.213690872 +0000 UTC m=+3093.741026090" watchObservedRunningTime="2026-10-30 17:12:18.226325378 +0000 UTC m=+3093.753660661" Oct 30 17:12:18.867998 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:12:18.867961 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 17:12:19.190115 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:19.190081 5052 patch_prober.go:29] interesting pod/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-g6lll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body= Oct 30 17:12:19.190531 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:19.190134 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-g6lll" podUID="0057b70d-3924-4e84-87f5-4cfc7e7babf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" Oct 30 17:12:19.773237 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:19.773203 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-1b7bb3-7777f75787-27lnc"] Oct 30 17:12:20.189482 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:20.189446 5052 patch_prober.go:29] interesting pod/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-g6lll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body= Oct 30 17:12:20.189817 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:20.189527 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-g6lll" podUID="0057b70d-3924-4e84-87f5-4cfc7e7babf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" Oct 30 17:12:21.189717 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:21.189675 5052 patch_prober.go:29] interesting pod/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-g6lll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body= Oct 30 17:12:21.190157 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:21.189747 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-g6lll" podUID="0057b70d-3924-4e84-87f5-4cfc7e7babf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" Oct 30 17:12:22.194026 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:22.193983 5052 patch_prober.go:29] interesting pod/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-g6lll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body= Oct 30 17:12:22.194743 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:22.194707 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-g6lll" podUID="0057b70d-3924-4e84-87f5-4cfc7e7babf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" Oct 30 17:12:23.190037 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:23.190000 5052 patch_prober.go:29] interesting pod/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-g6lll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body= Oct 30 17:12:23.190243 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:23.190060 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-g6lll" podUID="0057b70d-3924-4e84-87f5-4cfc7e7babf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" Oct 30 17:12:24.269557 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:24.269519 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-g6lll" Oct 30 17:12:24.270250 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:24.270222 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-g6lll" Oct 30 17:12:24.270988 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:24.270964 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-g6lll" Oct 30 17:12:24.773483 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:24.773424 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-1b7bb3-7777f75787-27lnc" podUID="84028f94-3e19-4900-a29d-31b2d2576c8d" containerName="payload-pre-processing" containerID="cri-o://94e14c6fe7fd0c28e55867508aaf1ef31cc3be74d22c261cd53b03c5c67a9092" gracePeriod=25 Oct 30 17:12:25.073406 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:25.073376 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-1b7bb3-7777f75787-27lnc" Oct 30 17:12:25.157418 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:25.157386 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84028f94-3e19-4900-a29d-31b2d2576c8d-kube-api-access-htg8f\" (UniqueName: \"kubernetes.io/projected/84028f94-3e19-4900-a29d-31b2d2576c8d-kube-api-access-htg8f\") pod \"84028f94-3e19-4900-a29d-31b2d2576c8d\" (UID: \"84028f94-3e19-4900-a29d-31b2d2576c8d\") " Oct 30 17:12:25.157653 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:25.157485 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84028f94-3e19-4900-a29d-31b2d2576c8d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84028f94-3e19-4900-a29d-31b2d2576c8d-plugins-config-volume\") pod \"84028f94-3e19-4900-a29d-31b2d2576c8d\" (UID: \"84028f94-3e19-4900-a29d-31b2d2576c8d\") " Oct 30 17:12:25.157978 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:25.157944 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84028f94-3e19-4900-a29d-31b2d2576c8d-plugins-config-volume" pod "84028f94-3e19-4900-a29d-31b2d2576c8d" (UID: "84028f94-3e19-4900-a29d-31b2d2576c8d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:12:25.158303 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:25.158284 5052 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84028f94-3e19-4900-a29d-31b2d2576c8d-plugins-config-volume\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:12:25.159648 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:25.159621 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84028f94-3e19-4900-a29d-31b2d2576c8d-kube-api-access-htg8f" pod "84028f94-3e19-4900-a29d-31b2d2576c8d" (UID: "84028f94-3e19-4900-a29d-31b2d2576c8d"). InnerVolumeSpecName "kube-api-access-htg8f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:12:25.231573 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:25.231537 5052 generic.go:356] "Generic (PLEG): container finished" podID="84028f94-3e19-4900-a29d-31b2d2576c8d" containerID="94e14c6fe7fd0c28e55867508aaf1ef31cc3be74d22c261cd53b03c5c67a9092" exitCode=0 Oct 30 17:12:25.231757 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:25.231600 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-1b7bb3-7777f75787-27lnc" Oct 30 17:12:25.231757 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:25.231617 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-1b7bb3-7777f75787-27lnc" event={"ID":"84028f94-3e19-4900-a29d-31b2d2576c8d","Type":"ContainerDied","Data":"94e14c6fe7fd0c28e55867508aaf1ef31cc3be74d22c261cd53b03c5c67a9092"} Oct 30 17:12:25.231757 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:25.231681 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-1b7bb3-7777f75787-27lnc" event={"ID":"84028f94-3e19-4900-a29d-31b2d2576c8d","Type":"ContainerDied","Data":"c67cc0efa912d5bab2e7c2602ad171e37bb0b54863ab914f55c7be20d2021a60"} Oct 30 17:12:25.231757 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:25.231708 5052 scope.go:122] "RemoveContainer" containerID="94e14c6fe7fd0c28e55867508aaf1ef31cc3be74d22c261cd53b03c5c67a9092" Oct 30 17:12:25.246262 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:25.246240 5052 scope.go:122] "RemoveContainer" containerID="94e14c6fe7fd0c28e55867508aaf1ef31cc3be74d22c261cd53b03c5c67a9092" Oct 30 17:12:25.246535 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:12:25.246515 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94e14c6fe7fd0c28e55867508aaf1ef31cc3be74d22c261cd53b03c5c67a9092\": container with ID starting with 94e14c6fe7fd0c28e55867508aaf1ef31cc3be74d22c261cd53b03c5c67a9092 not found: ID does not exist" containerID="94e14c6fe7fd0c28e55867508aaf1ef31cc3be74d22c261cd53b03c5c67a9092" Oct 30 17:12:25.246634 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:25.246546 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94e14c6fe7fd0c28e55867508aaf1ef31cc3be74d22c261cd53b03c5c67a9092"} err="failed to get container status \"94e14c6fe7fd0c28e55867508aaf1ef31cc3be74d22c261cd53b03c5c67a9092\": rpc error: code = NotFound desc = could not find container \"94e14c6fe7fd0c28e55867508aaf1ef31cc3be74d22c261cd53b03c5c67a9092\": container with ID starting with 94e14c6fe7fd0c28e55867508aaf1ef31cc3be74d22c261cd53b03c5c67a9092 not found: ID does not exist" Oct 30 17:12:25.260337 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:25.260315 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htg8f\" (UniqueName: \"kubernetes.io/projected/84028f94-3e19-4900-a29d-31b2d2576c8d-kube-api-access-htg8f\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:12:25.289416 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:25.289356 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-1b7bb3-7777f75787-27lnc"] Oct 30 17:12:25.293077 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:25.293057 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-1b7bb3-7777f75787-27lnc"] Oct 30 17:12:26.599640 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:26.599567 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84028f94-3e19-4900-a29d-31b2d2576c8d" path="/var/lib/kubelet/pods/84028f94-3e19-4900-a29d-31b2d2576c8d/volumes" Oct 30 17:12:33.352604 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:33.336058 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cdwcj"] Oct 30 17:12:35.385760 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:35.385724 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4def7e-7d4b86c85b-46559"] Oct 30 17:12:35.419275 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:35.419241 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4def7e-6cd68b6f59-nkp59"] Oct 30 17:12:40.386598 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:40.386547 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-4def7e-7d4b86c85b-46559" podUID="f2400f0d-42a6-453a-8122-3ddd83396635" containerName="payload-processing" containerID="cri-o://2f89eae6b5d253a65dc97691897ef5e88f51d958f8ffd152ff34427129ef7641" gracePeriod=25 Oct 30 17:12:40.440691 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:40.440397 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4def7e-6cd68b6f59-nkp59" podUID="6ad13120-2103-4526-a234-93b2e4e852ea" containerName="payload-pre-processing" containerID="cri-o://de71e4eca834b8ba05f1943dbb5f7a3187be64111ba5d64eb8448ce5b2e75275" gracePeriod=25 Oct 30 17:12:40.937785 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:40.937756 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-4def7e-7d4b86c85b-46559" Oct 30 17:12:41.005901 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:41.005868 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4def7e-6cd68b6f59-nkp59" Oct 30 17:12:41.020689 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:41.020658 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2400f0d-42a6-453a-8122-3ddd83396635-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2400f0d-42a6-453a-8122-3ddd83396635-plugins-config-volume\") pod \"f2400f0d-42a6-453a-8122-3ddd83396635\" (UID: \"f2400f0d-42a6-453a-8122-3ddd83396635\") " Oct 30 17:12:41.020861 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:41.020832 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ad13120-2103-4526-a234-93b2e4e852ea-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ad13120-2103-4526-a234-93b2e4e852ea-plugins-config-volume\") pod \"6ad13120-2103-4526-a234-93b2e4e852ea\" (UID: \"6ad13120-2103-4526-a234-93b2e4e852ea\") " Oct 30 17:12:41.020942 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:41.020870 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ad13120-2103-4526-a234-93b2e4e852ea-kube-api-access-qsszn\" (UniqueName: \"kubernetes.io/projected/6ad13120-2103-4526-a234-93b2e4e852ea-kube-api-access-qsszn\") pod \"6ad13120-2103-4526-a234-93b2e4e852ea\" (UID: \"6ad13120-2103-4526-a234-93b2e4e852ea\") " Oct 30 17:12:41.021143 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:41.021123 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2400f0d-42a6-453a-8122-3ddd83396635-kube-api-access-6vkx9\" (UniqueName: \"kubernetes.io/projected/f2400f0d-42a6-453a-8122-3ddd83396635-kube-api-access-6vkx9\") pod \"f2400f0d-42a6-453a-8122-3ddd83396635\" (UID: \"f2400f0d-42a6-453a-8122-3ddd83396635\") " Oct 30 17:12:41.021246 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:41.021200 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2400f0d-42a6-453a-8122-3ddd83396635-plugins-config-volume" pod "f2400f0d-42a6-453a-8122-3ddd83396635" (UID: "f2400f0d-42a6-453a-8122-3ddd83396635"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:12:41.021313 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:41.021277 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ad13120-2103-4526-a234-93b2e4e852ea-plugins-config-volume" pod "6ad13120-2103-4526-a234-93b2e4e852ea" (UID: "6ad13120-2103-4526-a234-93b2e4e852ea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:12:41.022516 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:41.022482 5052 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ad13120-2103-4526-a234-93b2e4e852ea-plugins-config-volume\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:12:41.022516 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:41.022513 5052 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2400f0d-42a6-453a-8122-3ddd83396635-plugins-config-volume\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:12:41.023269 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:41.023243 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ad13120-2103-4526-a234-93b2e4e852ea-kube-api-access-qsszn" pod "6ad13120-2103-4526-a234-93b2e4e852ea" (UID: "6ad13120-2103-4526-a234-93b2e4e852ea"). InnerVolumeSpecName "kube-api-access-qsszn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:12:41.023371 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:41.023304 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2400f0d-42a6-453a-8122-3ddd83396635-kube-api-access-6vkx9" pod "f2400f0d-42a6-453a-8122-3ddd83396635" (UID: "f2400f0d-42a6-453a-8122-3ddd83396635"). InnerVolumeSpecName "kube-api-access-6vkx9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:12:41.124179 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:41.124141 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qsszn\" (UniqueName: \"kubernetes.io/projected/6ad13120-2103-4526-a234-93b2e4e852ea-kube-api-access-qsszn\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:12:41.124179 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:41.124174 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vkx9\" (UniqueName: \"kubernetes.io/projected/f2400f0d-42a6-453a-8122-3ddd83396635-kube-api-access-6vkx9\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:12:41.403867 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:41.403829 5052 generic.go:356] "Generic (PLEG): container finished" podID="f2400f0d-42a6-453a-8122-3ddd83396635" containerID="2f89eae6b5d253a65dc97691897ef5e88f51d958f8ffd152ff34427129ef7641" exitCode=0 Oct 30 17:12:41.404333 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:41.403878 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-4def7e-7d4b86c85b-46559" Oct 30 17:12:41.404333 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:41.403914 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-4def7e-7d4b86c85b-46559" event={"ID":"f2400f0d-42a6-453a-8122-3ddd83396635","Type":"ContainerDied","Data":"2f89eae6b5d253a65dc97691897ef5e88f51d958f8ffd152ff34427129ef7641"} Oct 30 17:12:41.404333 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:41.403980 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-4def7e-7d4b86c85b-46559" event={"ID":"f2400f0d-42a6-453a-8122-3ddd83396635","Type":"ContainerDied","Data":"8bb685ebf72a07970488dab4fc0ff377917a728451e70351a05fdd51a1298904"} Oct 30 17:12:41.404333 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:41.404018 5052 scope.go:122] "RemoveContainer" containerID="2f89eae6b5d253a65dc97691897ef5e88f51d958f8ffd152ff34427129ef7641" Oct 30 17:12:41.405490 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:41.405468 5052 generic.go:356] "Generic (PLEG): container finished" podID="6ad13120-2103-4526-a234-93b2e4e852ea" containerID="de71e4eca834b8ba05f1943dbb5f7a3187be64111ba5d64eb8448ce5b2e75275" exitCode=0 Oct 30 17:12:41.405566 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:41.405511 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4def7e-6cd68b6f59-nkp59" event={"ID":"6ad13120-2103-4526-a234-93b2e4e852ea","Type":"ContainerDied","Data":"de71e4eca834b8ba05f1943dbb5f7a3187be64111ba5d64eb8448ce5b2e75275"} Oct 30 17:12:41.405645 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:41.405557 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4def7e-6cd68b6f59-nkp59" event={"ID":"6ad13120-2103-4526-a234-93b2e4e852ea","Type":"ContainerDied","Data":"a8b64cd2952ac478a35e54ab2255104ec3128e7fded06997cc0dbb1ded613e1f"} Oct 30 17:12:41.405645 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:41.405522 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4def7e-6cd68b6f59-nkp59" Oct 30 17:12:41.416614 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:41.416596 5052 scope.go:122] "RemoveContainer" containerID="2f89eae6b5d253a65dc97691897ef5e88f51d958f8ffd152ff34427129ef7641" Oct 30 17:12:41.417017 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:12:41.416986 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f89eae6b5d253a65dc97691897ef5e88f51d958f8ffd152ff34427129ef7641\": container with ID starting with 2f89eae6b5d253a65dc97691897ef5e88f51d958f8ffd152ff34427129ef7641 not found: ID does not exist" containerID="2f89eae6b5d253a65dc97691897ef5e88f51d958f8ffd152ff34427129ef7641" Oct 30 17:12:41.417186 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:41.417152 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f89eae6b5d253a65dc97691897ef5e88f51d958f8ffd152ff34427129ef7641"} err="failed to get container status \"2f89eae6b5d253a65dc97691897ef5e88f51d958f8ffd152ff34427129ef7641\": rpc error: code = NotFound desc = could not find container \"2f89eae6b5d253a65dc97691897ef5e88f51d958f8ffd152ff34427129ef7641\": container with ID starting with 2f89eae6b5d253a65dc97691897ef5e88f51d958f8ffd152ff34427129ef7641 not found: ID does not exist" Oct 30 17:12:41.417337 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:41.417186 5052 scope.go:122] "RemoveContainer" containerID="de71e4eca834b8ba05f1943dbb5f7a3187be64111ba5d64eb8448ce5b2e75275" Oct 30 17:12:41.453671 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:41.450854 5052 scope.go:122] "RemoveContainer" containerID="de71e4eca834b8ba05f1943dbb5f7a3187be64111ba5d64eb8448ce5b2e75275" Oct 30 17:12:41.455391 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:12:41.455357 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de71e4eca834b8ba05f1943dbb5f7a3187be64111ba5d64eb8448ce5b2e75275\": container with ID starting with de71e4eca834b8ba05f1943dbb5f7a3187be64111ba5d64eb8448ce5b2e75275 not found: ID does not exist" containerID="de71e4eca834b8ba05f1943dbb5f7a3187be64111ba5d64eb8448ce5b2e75275" Oct 30 17:12:41.455510 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:41.455404 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de71e4eca834b8ba05f1943dbb5f7a3187be64111ba5d64eb8448ce5b2e75275"} err="failed to get container status \"de71e4eca834b8ba05f1943dbb5f7a3187be64111ba5d64eb8448ce5b2e75275\": rpc error: code = NotFound desc = could not find container \"de71e4eca834b8ba05f1943dbb5f7a3187be64111ba5d64eb8448ce5b2e75275\": container with ID starting with de71e4eca834b8ba05f1943dbb5f7a3187be64111ba5d64eb8448ce5b2e75275 not found: ID does not exist" Oct 30 17:12:41.558697 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:41.558652 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4def7e-6cd68b6f59-nkp59"] Oct 30 17:12:41.564551 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:41.564510 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4def7e-6cd68b6f59-nkp59"] Oct 30 17:12:41.569294 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:41.569270 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4def7e-7d4b86c85b-46559"] Oct 30 17:12:41.572835 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:41.572814 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4def7e-7d4b86c85b-46559"] Oct 30 17:12:42.600881 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:42.600839 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ad13120-2103-4526-a234-93b2e4e852ea" path="/var/lib/kubelet/pods/6ad13120-2103-4526-a234-93b2e4e852ea/volumes" Oct 30 17:12:42.601391 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:42.601274 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2400f0d-42a6-453a-8122-3ddd83396635" path="/var/lib/kubelet/pods/f2400f0d-42a6-453a-8122-3ddd83396635/volumes" Oct 30 17:12:46.447595 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:46.447557 5052 generic.go:356] "Generic (PLEG): container finished" podID="666e4b84-2d87-4557-a8b5-10f890a1740b" containerID="3e8dd9f08e58cf02aadc5e493e0f6bbb4b3d3a4f4e48c94664612a42fed64aa3" exitCode=0 Oct 30 17:12:46.448023 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:46.447632 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0ae3fe-openshift-default-69fdb67df-sbgql" event={"ID":"666e4b84-2d87-4557-a8b5-10f890a1740b","Type":"ContainerDied","Data":"3e8dd9f08e58cf02aadc5e493e0f6bbb4b3d3a4f4e48c94664612a42fed64aa3"} Oct 30 17:12:46.448105 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:46.448090 5052 scope.go:122] "RemoveContainer" containerID="3e8dd9f08e58cf02aadc5e493e0f6bbb4b3d3a4f4e48c94664612a42fed64aa3" Oct 30 17:12:46.450040 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:46.450011 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:12:46.450122 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:46.450082 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:12:46.450122 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:46.450114 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:12:47.462775 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:47.462738 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0ae3fe-openshift-default-69fdb67df-sbgql" event={"ID":"666e4b84-2d87-4557-a8b5-10f890a1740b","Type":"ContainerStarted","Data":"5e1bb34dcfa3906b934a7cf76809c8d2ad4cf7085c32d857441401563a4ec18a"} Oct 30 17:12:48.247601 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:48.243900 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-0ae3fe-openshift-default-69fdb67df-sbgql" Oct 30 17:12:48.250632 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:48.249817 5052 patch_prober.go:29] interesting pod/e2e-ipp-b-0ae3fe-openshift-default-69fdb67df-sbgql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body= Oct 30 17:12:48.250632 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:48.249891 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-0ae3fe-openshift-default-69fdb67df-sbgql" podUID="666e4b84-2d87-4557-a8b5-10f890a1740b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" Oct 30 17:12:48.867642 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:12:48.867575 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 17:12:49.246019 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:49.245979 5052 patch_prober.go:29] interesting pod/e2e-ipp-b-0ae3fe-openshift-default-69fdb67df-sbgql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body= Oct 30 17:12:49.246331 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:49.246297 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-0ae3fe-openshift-default-69fdb67df-sbgql" podUID="666e4b84-2d87-4557-a8b5-10f890a1740b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" Oct 30 17:12:49.338701 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:49.338666 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-0ae3fe-openshift-default-69fdb67df-sbgql" Oct 30 17:12:50.243601 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:50.243144 5052 patch_prober.go:29] interesting pod/e2e-ipp-b-0ae3fe-openshift-default-69fdb67df-sbgql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body= Oct 30 17:12:50.243601 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:50.243210 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-0ae3fe-openshift-default-69fdb67df-sbgql" podUID="666e4b84-2d87-4557-a8b5-10f890a1740b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" Oct 30 17:12:51.248607 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:51.245986 5052 patch_prober.go:29] interesting pod/e2e-ipp-b-0ae3fe-openshift-default-69fdb67df-sbgql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body= Oct 30 17:12:51.248607 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:51.246058 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-0ae3fe-openshift-default-69fdb67df-sbgql" podUID="666e4b84-2d87-4557-a8b5-10f890a1740b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" Oct 30 17:12:52.245940 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:52.245901 5052 patch_prober.go:29] interesting pod/e2e-ipp-b-0ae3fe-openshift-default-69fdb67df-sbgql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body= Oct 30 17:12:52.246283 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:52.246252 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-0ae3fe-openshift-default-69fdb67df-sbgql" podUID="666e4b84-2d87-4557-a8b5-10f890a1740b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" Oct 30 17:12:53.249054 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:53.249023 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-0ae3fe-openshift-default-69fdb67df-sbgql" Oct 30 17:12:53.250282 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:53.250258 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-0ae3fe-openshift-default-69fdb67df-sbgql" Oct 30 17:12:55.161849 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:55.161815 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-ddc10d-openshift-default-56bfdd795c-rjzgr"] Oct 30 17:12:55.162609 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:55.162575 5052 cpu_manager.go:414] "containerMap: removing container" podUID="f2400f0d-42a6-453a-8122-3ddd83396635" containerName="payload-processing" Oct 30 17:12:55.162609 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:55.162609 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2400f0d-42a6-453a-8122-3ddd83396635" containerName="payload-processing" Oct 30 17:12:55.162772 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:55.162625 5052 cpu_manager.go:414] "containerMap: removing container" podUID="6ad13120-2103-4526-a234-93b2e4e852ea" containerName="payload-pre-processing" Oct 30 17:12:55.162772 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:55.162631 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ad13120-2103-4526-a234-93b2e4e852ea" containerName="payload-pre-processing" Oct 30 17:12:55.162772 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:55.162649 5052 cpu_manager.go:414] "containerMap: removing container" podUID="84028f94-3e19-4900-a29d-31b2d2576c8d" containerName="payload-pre-processing" Oct 30 17:12:55.162772 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:55.162654 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84028f94-3e19-4900-a29d-31b2d2576c8d" containerName="payload-pre-processing" Oct 30 17:12:55.162772 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:55.162665 5052 cpu_manager.go:414] "containerMap: removing container" podUID="050171fd-aa7a-47db-8cea-e7ed38828148" containerName="istio-proxy" Oct 30 17:12:55.162772 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:55.162669 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="050171fd-aa7a-47db-8cea-e7ed38828148" containerName="istio-proxy" Oct 30 17:12:55.162772 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:55.162676 5052 cpu_manager.go:414] "containerMap: removing container" podUID="91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a" containerName="istio-proxy" Oct 30 17:12:55.162772 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:55.162680 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a" containerName="istio-proxy" Oct 30 17:12:55.163224 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:55.162869 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="84028f94-3e19-4900-a29d-31b2d2576c8d" containerName="payload-pre-processing" Oct 30 17:12:55.163224 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:55.162893 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="91887ee1-8aeb-46eb-aefa-a11fe7dd7f9a" containerName="istio-proxy" Oct 30 17:12:55.163224 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:55.162911 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2400f0d-42a6-453a-8122-3ddd83396635" containerName="payload-processing" Oct 30 17:12:55.163224 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:55.162922 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ad13120-2103-4526-a234-93b2e4e852ea" containerName="payload-pre-processing" Oct 30 17:12:55.168984 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:55.168959 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-ddc10d-openshift-default-56bfdd795c-rjzgr" Oct 30 17:12:55.197230 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:55.197197 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-ddc10d-openshift-default-56bfdd795c-rjzgr"] Oct 30 17:12:55.205985 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:55.205958 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-ddc10d-openshift-default-dockercfg-8vtdj\"" Oct 30 17:12:55.222266 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:55.222230 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rqqb\" (UniqueName: \"kubernetes.io/projected/0261ebf2-6dfd-4df5-b5de-4b7207ca761b-kube-api-access-8rqqb\") pod \"e2e-ipp-b-ddc10d-openshift-default-56bfdd795c-rjzgr\" (UID: \"0261ebf2-6dfd-4df5-b5de-4b7207ca761b\") " pod="openshift-ingress/e2e-ipp-b-ddc10d-openshift-default-56bfdd795c-rjzgr" Oct 30 17:12:55.222452 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:55.222404 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0261ebf2-6dfd-4df5-b5de-4b7207ca761b-istio-envoy\") pod \"e2e-ipp-b-ddc10d-openshift-default-56bfdd795c-rjzgr\" (UID: \"0261ebf2-6dfd-4df5-b5de-4b7207ca761b\") " pod="openshift-ingress/e2e-ipp-b-ddc10d-openshift-default-56bfdd795c-rjzgr" Oct 30 17:12:55.222528 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:55.222474 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0261ebf2-6dfd-4df5-b5de-4b7207ca761b-istio-podinfo\") pod \"e2e-ipp-b-ddc10d-openshift-default-56bfdd795c-rjzgr\" (UID: \"0261ebf2-6dfd-4df5-b5de-4b7207ca761b\") " pod="openshift-ingress/e2e-ipp-b-ddc10d-openshift-default-56bfdd795c-rjzgr" Oct 30 17:12:55.222528 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:55.222510 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0261ebf2-6dfd-4df5-b5de-4b7207ca761b-workload-socket\") pod \"e2e-ipp-b-ddc10d-openshift-default-56bfdd795c-rjzgr\" (UID: \"0261ebf2-6dfd-4df5-b5de-4b7207ca761b\") " pod="openshift-ingress/e2e-ipp-b-ddc10d-openshift-default-56bfdd795c-rjzgr" Oct 30 17:12:55.222655 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:55.222540 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0261ebf2-6dfd-4df5-b5de-4b7207ca761b-istio-data\") pod \"e2e-ipp-b-ddc10d-openshift-default-56bfdd795c-rjzgr\" (UID: \"0261ebf2-6dfd-4df5-b5de-4b7207ca761b\") " pod="openshift-ingress/e2e-ipp-b-ddc10d-openshift-default-56bfdd795c-rjzgr" Oct 30 17:12:55.222701 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:55.222671 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0261ebf2-6dfd-4df5-b5de-4b7207ca761b-workload-certs\") pod \"e2e-ipp-b-ddc10d-openshift-default-56bfdd795c-rjzgr\" (UID: \"0261ebf2-6dfd-4df5-b5de-4b7207ca761b\") " pod="openshift-ingress/e2e-ipp-b-ddc10d-openshift-default-56bfdd795c-rjzgr" Oct 30 17:12:55.222757 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:55.222743 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0261ebf2-6dfd-4df5-b5de-4b7207ca761b-istio-token\") pod \"e2e-ipp-b-ddc10d-openshift-default-56bfdd795c-rjzgr\" (UID: \"0261ebf2-6dfd-4df5-b5de-4b7207ca761b\") " pod="openshift-ingress/e2e-ipp-b-ddc10d-openshift-default-56bfdd795c-rjzgr" Oct 30 17:12:55.222853 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:55.222831 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0261ebf2-6dfd-4df5-b5de-4b7207ca761b-istiod-ca-cert\") pod \"e2e-ipp-b-ddc10d-openshift-default-56bfdd795c-rjzgr\" (UID: \"0261ebf2-6dfd-4df5-b5de-4b7207ca761b\") " pod="openshift-ingress/e2e-ipp-b-ddc10d-openshift-default-56bfdd795c-rjzgr" Oct 30 17:12:55.222909 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:55.222875 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0261ebf2-6dfd-4df5-b5de-4b7207ca761b-credential-socket\") pod \"e2e-ipp-b-ddc10d-openshift-default-56bfdd795c-rjzgr\" (UID: \"0261ebf2-6dfd-4df5-b5de-4b7207ca761b\") " pod="openshift-ingress/e2e-ipp-b-ddc10d-openshift-default-56bfdd795c-rjzgr" Oct 30 17:12:55.324788 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:55.324755 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rqqb\" (UniqueName: \"kubernetes.io/projected/0261ebf2-6dfd-4df5-b5de-4b7207ca761b-kube-api-access-8rqqb\") pod \"e2e-ipp-b-ddc10d-openshift-default-56bfdd795c-rjzgr\" (UID: \"0261ebf2-6dfd-4df5-b5de-4b7207ca761b\") " pod="openshift-ingress/e2e-ipp-b-ddc10d-openshift-default-56bfdd795c-rjzgr" Oct 30 17:12:55.324989 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:55.324823 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0261ebf2-6dfd-4df5-b5de-4b7207ca761b-istio-envoy\") pod \"e2e-ipp-b-ddc10d-openshift-default-56bfdd795c-rjzgr\" (UID: \"0261ebf2-6dfd-4df5-b5de-4b7207ca761b\") " pod="openshift-ingress/e2e-ipp-b-ddc10d-openshift-default-56bfdd795c-rjzgr" Oct 30 17:12:55.324989 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:55.324860 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0261ebf2-6dfd-4df5-b5de-4b7207ca761b-istio-podinfo\") pod \"e2e-ipp-b-ddc10d-openshift-default-56bfdd795c-rjzgr\" (UID: \"0261ebf2-6dfd-4df5-b5de-4b7207ca761b\") " pod="openshift-ingress/e2e-ipp-b-ddc10d-openshift-default-56bfdd795c-rjzgr" Oct 30 17:12:55.324989 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:55.324876 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0261ebf2-6dfd-4df5-b5de-4b7207ca761b-workload-socket\") pod \"e2e-ipp-b-ddc10d-openshift-default-56bfdd795c-rjzgr\" (UID: \"0261ebf2-6dfd-4df5-b5de-4b7207ca761b\") " pod="openshift-ingress/e2e-ipp-b-ddc10d-openshift-default-56bfdd795c-rjzgr" Oct 30 17:12:55.324989 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:55.324892 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0261ebf2-6dfd-4df5-b5de-4b7207ca761b-istio-data\") pod \"e2e-ipp-b-ddc10d-openshift-default-56bfdd795c-rjzgr\" (UID: \"0261ebf2-6dfd-4df5-b5de-4b7207ca761b\") " pod="openshift-ingress/e2e-ipp-b-ddc10d-openshift-default-56bfdd795c-rjzgr" Oct 30 17:12:55.324989 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:55.324930 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0261ebf2-6dfd-4df5-b5de-4b7207ca761b-workload-certs\") pod \"e2e-ipp-b-ddc10d-openshift-default-56bfdd795c-rjzgr\" (UID: \"0261ebf2-6dfd-4df5-b5de-4b7207ca761b\") " pod="openshift-ingress/e2e-ipp-b-ddc10d-openshift-default-56bfdd795c-rjzgr" Oct 30 17:12:55.324989 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:55.324983 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0261ebf2-6dfd-4df5-b5de-4b7207ca761b-istio-token\") pod \"e2e-ipp-b-ddc10d-openshift-default-56bfdd795c-rjzgr\" (UID: \"0261ebf2-6dfd-4df5-b5de-4b7207ca761b\") " pod="openshift-ingress/e2e-ipp-b-ddc10d-openshift-default-56bfdd795c-rjzgr" Oct 30 17:12:55.325336 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:55.325041 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0261ebf2-6dfd-4df5-b5de-4b7207ca761b-istiod-ca-cert\") pod \"e2e-ipp-b-ddc10d-openshift-default-56bfdd795c-rjzgr\" (UID: \"0261ebf2-6dfd-4df5-b5de-4b7207ca761b\") " pod="openshift-ingress/e2e-ipp-b-ddc10d-openshift-default-56bfdd795c-rjzgr" Oct 30 17:12:55.325336 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:55.325076 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0261ebf2-6dfd-4df5-b5de-4b7207ca761b-credential-socket\") pod \"e2e-ipp-b-ddc10d-openshift-default-56bfdd795c-rjzgr\" (UID: \"0261ebf2-6dfd-4df5-b5de-4b7207ca761b\") " pod="openshift-ingress/e2e-ipp-b-ddc10d-openshift-default-56bfdd795c-rjzgr" Oct 30 17:12:55.325500 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:55.325389 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0261ebf2-6dfd-4df5-b5de-4b7207ca761b-workload-socket\") pod \"e2e-ipp-b-ddc10d-openshift-default-56bfdd795c-rjzgr\" (UID: \"0261ebf2-6dfd-4df5-b5de-4b7207ca761b\") " pod="openshift-ingress/e2e-ipp-b-ddc10d-openshift-default-56bfdd795c-rjzgr" Oct 30 17:12:55.325500 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:55.325439 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0261ebf2-6dfd-4df5-b5de-4b7207ca761b-credential-socket\") pod \"e2e-ipp-b-ddc10d-openshift-default-56bfdd795c-rjzgr\" (UID: \"0261ebf2-6dfd-4df5-b5de-4b7207ca761b\") " pod="openshift-ingress/e2e-ipp-b-ddc10d-openshift-default-56bfdd795c-rjzgr" Oct 30 17:12:55.325652 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:55.325510 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0261ebf2-6dfd-4df5-b5de-4b7207ca761b-workload-certs\") pod \"e2e-ipp-b-ddc10d-openshift-default-56bfdd795c-rjzgr\" (UID: \"0261ebf2-6dfd-4df5-b5de-4b7207ca761b\") " pod="openshift-ingress/e2e-ipp-b-ddc10d-openshift-default-56bfdd795c-rjzgr" Oct 30 17:12:55.325886 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:55.325861 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0261ebf2-6dfd-4df5-b5de-4b7207ca761b-istio-data\") pod \"e2e-ipp-b-ddc10d-openshift-default-56bfdd795c-rjzgr\" (UID: \"0261ebf2-6dfd-4df5-b5de-4b7207ca761b\") " pod="openshift-ingress/e2e-ipp-b-ddc10d-openshift-default-56bfdd795c-rjzgr" Oct 30 17:12:55.325956 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:55.325921 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0261ebf2-6dfd-4df5-b5de-4b7207ca761b-istiod-ca-cert\") pod \"e2e-ipp-b-ddc10d-openshift-default-56bfdd795c-rjzgr\" (UID: \"0261ebf2-6dfd-4df5-b5de-4b7207ca761b\") " pod="openshift-ingress/e2e-ipp-b-ddc10d-openshift-default-56bfdd795c-rjzgr" Oct 30 17:12:55.327243 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:55.327225 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0261ebf2-6dfd-4df5-b5de-4b7207ca761b-istio-envoy\") pod \"e2e-ipp-b-ddc10d-openshift-default-56bfdd795c-rjzgr\" (UID: \"0261ebf2-6dfd-4df5-b5de-4b7207ca761b\") " pod="openshift-ingress/e2e-ipp-b-ddc10d-openshift-default-56bfdd795c-rjzgr" Oct 30 17:12:55.327695 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:55.327674 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0261ebf2-6dfd-4df5-b5de-4b7207ca761b-istio-podinfo\") pod \"e2e-ipp-b-ddc10d-openshift-default-56bfdd795c-rjzgr\" (UID: \"0261ebf2-6dfd-4df5-b5de-4b7207ca761b\") " pod="openshift-ingress/e2e-ipp-b-ddc10d-openshift-default-56bfdd795c-rjzgr" Oct 30 17:12:55.355268 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:55.354453 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rqqb\" (UniqueName: \"kubernetes.io/projected/0261ebf2-6dfd-4df5-b5de-4b7207ca761b-kube-api-access-8rqqb\") pod \"e2e-ipp-b-ddc10d-openshift-default-56bfdd795c-rjzgr\" (UID: \"0261ebf2-6dfd-4df5-b5de-4b7207ca761b\") " pod="openshift-ingress/e2e-ipp-b-ddc10d-openshift-default-56bfdd795c-rjzgr" Oct 30 17:12:55.355268 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:55.354980 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0261ebf2-6dfd-4df5-b5de-4b7207ca761b-istio-token\") pod \"e2e-ipp-b-ddc10d-openshift-default-56bfdd795c-rjzgr\" (UID: \"0261ebf2-6dfd-4df5-b5de-4b7207ca761b\") " pod="openshift-ingress/e2e-ipp-b-ddc10d-openshift-default-56bfdd795c-rjzgr" Oct 30 17:12:55.502657 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:55.497470 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-ddc10d-openshift-default-56bfdd795c-rjzgr" Oct 30 17:12:56.054677 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:56.054047 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-ddc10d-openshift-default-56bfdd795c-rjzgr"] Oct 30 17:12:56.175816 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:56.174625 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:12:56.175816 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:56.174723 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:12:56.175816 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:56.174765 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:12:56.637998 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:56.637965 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ddc10d-openshift-default-56bfdd795c-rjzgr" event={"ID":"0261ebf2-6dfd-4df5-b5de-4b7207ca761b","Type":"ContainerStarted","Data":"679ff824e61208b26c583fee45e391f0e33d81c23c74fcb9f7cc6f1a87d566ab"} Oct 30 17:12:56.638185 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:56.638016 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ddc10d-openshift-default-56bfdd795c-rjzgr" event={"ID":"0261ebf2-6dfd-4df5-b5de-4b7207ca761b","Type":"ContainerStarted","Data":"b5ef86695b30809c68ab785bd492d34cbeacb52fd5175c944c2454cb41e50e7f"} Oct 30 17:12:56.686703 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:56.686647 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-ddc10d-openshift-default-56bfdd795c-rjzgr" podStartSLOduration=1.686634266 podStartE2EDuration="1.686634266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:12:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:12:56.685519917 +0000 UTC m=+3132.212855137" watchObservedRunningTime="2026-10-30 17:12:56.686634266 +0000 UTC m=+3132.213969484" Oct 30 17:12:57.498386 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:57.498350 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-ddc10d-openshift-default-56bfdd795c-rjzgr" Oct 30 17:12:57.504559 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:57.504535 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-ddc10d-openshift-default-56bfdd795c-rjzgr" Oct 30 17:12:57.645329 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:57.645297 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-ddc10d-openshift-default-56bfdd795c-rjzgr" Oct 30 17:12:57.646245 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:57.646226 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-ddc10d-openshift-default-56bfdd795c-rjzgr" Oct 30 17:12:57.959830 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:57.959793 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0ae3fe-5b445b79c8-cqkdf"] Oct 30 17:12:57.992347 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:12:57.992317 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0ae3fe-598c75dd67-pjf8j"] Oct 30 17:13:01.136561 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:01.136524 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ddc10d-55c9674d94-hwbjl"] Oct 30 17:13:01.141778 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:01.141754 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ddc10d-55c9674d94-hwbjl" Oct 30 17:13:01.160768 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:01.160739 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ddc10d-55c9674d94-hwbjl"] Oct 30 17:13:01.169277 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:01.169246 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-ddc10d\"" Oct 30 17:13:01.169277 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:01.169249 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-ddc10d-dockercfg-dwknv\"" Oct 30 17:13:01.197989 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:01.197957 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ddc10d-55f45f45f-4dt5v"] Oct 30 17:13:01.202905 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:01.202884 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ddc10d-55f45f45f-4dt5v" Oct 30 17:13:01.412673 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:01.307162 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgfbq\" (UniqueName: \"kubernetes.io/projected/ec94c66b-d7c7-43cb-8df7-e7cf32251abb-kube-api-access-pgfbq\") pod \"payload-pre-processing-e2e-ipp-b-ddc10d-55f45f45f-4dt5v\" (UID: \"ec94c66b-d7c7-43cb-8df7-e7cf32251abb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ddc10d-55f45f45f-4dt5v" Oct 30 17:13:01.412673 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:01.307231 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6227d9b1-b976-4afd-ab68-c7e051272d11-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ddc10d-55c9674d94-hwbjl\" (UID: \"6227d9b1-b976-4afd-ab68-c7e051272d11\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ddc10d-55c9674d94-hwbjl" Oct 30 17:13:01.412673 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:01.307372 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p25qf\" (UniqueName: \"kubernetes.io/projected/6227d9b1-b976-4afd-ab68-c7e051272d11-kube-api-access-p25qf\") pod \"payload-processing-e2e-ipp-b-ddc10d-55c9674d94-hwbjl\" (UID: \"6227d9b1-b976-4afd-ab68-c7e051272d11\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ddc10d-55c9674d94-hwbjl" Oct 30 17:13:01.412673 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:01.307453 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec94c66b-d7c7-43cb-8df7-e7cf32251abb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ddc10d-55f45f45f-4dt5v\" (UID: \"ec94c66b-d7c7-43cb-8df7-e7cf32251abb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ddc10d-55f45f45f-4dt5v" Oct 30 17:13:01.478607 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:01.419164 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgfbq\" (UniqueName: \"kubernetes.io/projected/ec94c66b-d7c7-43cb-8df7-e7cf32251abb-kube-api-access-pgfbq\") pod \"payload-pre-processing-e2e-ipp-b-ddc10d-55f45f45f-4dt5v\" (UID: \"ec94c66b-d7c7-43cb-8df7-e7cf32251abb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ddc10d-55f45f45f-4dt5v" Oct 30 17:13:01.478607 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:01.419231 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6227d9b1-b976-4afd-ab68-c7e051272d11-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ddc10d-55c9674d94-hwbjl\" (UID: \"6227d9b1-b976-4afd-ab68-c7e051272d11\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ddc10d-55c9674d94-hwbjl" Oct 30 17:13:01.478607 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:01.419421 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p25qf\" (UniqueName: \"kubernetes.io/projected/6227d9b1-b976-4afd-ab68-c7e051272d11-kube-api-access-p25qf\") pod \"payload-processing-e2e-ipp-b-ddc10d-55c9674d94-hwbjl\" (UID: \"6227d9b1-b976-4afd-ab68-c7e051272d11\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ddc10d-55c9674d94-hwbjl" Oct 30 17:13:01.478607 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:01.419523 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec94c66b-d7c7-43cb-8df7-e7cf32251abb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ddc10d-55f45f45f-4dt5v\" (UID: \"ec94c66b-d7c7-43cb-8df7-e7cf32251abb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ddc10d-55f45f45f-4dt5v" Oct 30 17:13:01.478607 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:01.420388 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec94c66b-d7c7-43cb-8df7-e7cf32251abb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ddc10d-55f45f45f-4dt5v\" (UID: \"ec94c66b-d7c7-43cb-8df7-e7cf32251abb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ddc10d-55f45f45f-4dt5v" Oct 30 17:13:01.478607 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:01.462047 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ddc10d-55f45f45f-4dt5v"] Oct 30 17:13:01.495700 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:01.486082 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgfbq\" (UniqueName: \"kubernetes.io/projected/ec94c66b-d7c7-43cb-8df7-e7cf32251abb-kube-api-access-pgfbq\") pod \"payload-pre-processing-e2e-ipp-b-ddc10d-55f45f45f-4dt5v\" (UID: \"ec94c66b-d7c7-43cb-8df7-e7cf32251abb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ddc10d-55f45f45f-4dt5v" Oct 30 17:13:01.526648 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:01.496471 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6227d9b1-b976-4afd-ab68-c7e051272d11-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ddc10d-55c9674d94-hwbjl\" (UID: \"6227d9b1-b976-4afd-ab68-c7e051272d11\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ddc10d-55c9674d94-hwbjl" Oct 30 17:13:01.531229 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:01.531187 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ddc10d-55f45f45f-4dt5v" Oct 30 17:13:01.544221 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:01.544142 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p25qf\" (UniqueName: \"kubernetes.io/projected/6227d9b1-b976-4afd-ab68-c7e051272d11-kube-api-access-p25qf\") pod \"payload-processing-e2e-ipp-b-ddc10d-55c9674d94-hwbjl\" (UID: \"6227d9b1-b976-4afd-ab68-c7e051272d11\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ddc10d-55c9674d94-hwbjl" Oct 30 17:13:01.757721 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:01.757621 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ddc10d-55c9674d94-hwbjl" Oct 30 17:13:01.867206 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:01.867169 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ddc10d-55f45f45f-4dt5v"] Oct 30 17:13:01.868309 ip-10-0-131-136 kubenswrapper[5052]: W1030 17:13:01.868275 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec94c66b_d7c7_43cb_8df7_e7cf32251abb.slice/crio-e2bccb87c105481f636610000b99281e0d37e661ed7873d430829560b5c17c43 WatchSource:0}: Error finding container e2bccb87c105481f636610000b99281e0d37e661ed7873d430829560b5c17c43: Status 404 returned error can't find the container with id e2bccb87c105481f636610000b99281e0d37e661ed7873d430829560b5c17c43 Oct 30 17:13:02.071600 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:02.071553 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ddc10d-55c9674d94-hwbjl"] Oct 30 17:13:02.072033 ip-10-0-131-136 kubenswrapper[5052]: W1030 17:13:02.072005 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6227d9b1_b976_4afd_ab68_c7e051272d11.slice/crio-ce575eaf5027410a2dc4aac3fa842e8ee6058f1259ef7ae0c48431bf7cad2bcb WatchSource:0}: Error finding container ce575eaf5027410a2dc4aac3fa842e8ee6058f1259ef7ae0c48431bf7cad2bcb: Status 404 returned error can't find the container with id ce575eaf5027410a2dc4aac3fa842e8ee6058f1259ef7ae0c48431bf7cad2bcb Oct 30 17:13:02.733890 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:02.733847 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ddc10d-55c9674d94-hwbjl" event={"ID":"6227d9b1-b976-4afd-ab68-c7e051272d11","Type":"ContainerStarted","Data":"84bba1d9b88b1de3c7da69e2306744cebced1f9497c2edea37aed83e39164172"} Oct 30 17:13:02.734383 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:02.733910 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ddc10d-55c9674d94-hwbjl" event={"ID":"6227d9b1-b976-4afd-ab68-c7e051272d11","Type":"ContainerStarted","Data":"ce575eaf5027410a2dc4aac3fa842e8ee6058f1259ef7ae0c48431bf7cad2bcb"} Oct 30 17:13:02.734383 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:02.733942 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-ddc10d-55c9674d94-hwbjl" Oct 30 17:13:02.735736 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:02.735708 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ddc10d-55f45f45f-4dt5v" event={"ID":"ec94c66b-d7c7-43cb-8df7-e7cf32251abb","Type":"ContainerStarted","Data":"79bda2629a22bcf1ce22d67a5f8309983baa31e0758f90fbb2361c669bc042f2"} Oct 30 17:13:02.735856 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:02.735754 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ddc10d-55f45f45f-4dt5v" event={"ID":"ec94c66b-d7c7-43cb-8df7-e7cf32251abb","Type":"ContainerStarted","Data":"e2bccb87c105481f636610000b99281e0d37e661ed7873d430829560b5c17c43"} Oct 30 17:13:02.735856 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:02.735795 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ddc10d-55f45f45f-4dt5v" Oct 30 17:13:02.817749 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:02.816787 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-ddc10d-55c9674d94-hwbjl" podStartSLOduration=1.816767448 podStartE2EDuration="1.816767448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:13:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:13:02.790667133 +0000 UTC m=+3138.318002352" watchObservedRunningTime="2026-10-30 17:13:02.816767448 +0000 UTC m=+3138.344102669" Oct 30 17:13:02.839522 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:02.839460 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ddc10d-55f45f45f-4dt5v" podStartSLOduration=1.8394417490000001 podStartE2EDuration="1.839441749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:13:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:13:02.838818906 +0000 UTC m=+3138.366154127" watchObservedRunningTime="2026-10-30 17:13:02.839441749 +0000 UTC m=+3138.366776969" Oct 30 17:13:02.976626 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:02.973172 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-0ae3fe-5b445b79c8-cqkdf" podUID="67f898fa-56b4-4f6d-8b02-3c9b24cedcf0" containerName="payload-processing" containerID="cri-o://b5ea2bd3a524e7e634afbaf076904cb9f7eead38f1a0373069b73be26e58b3ea" gracePeriod=25 Oct 30 17:13:02.992819 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:02.992619 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ae3fe-598c75dd67-pjf8j" podUID="4739b888-b4ee-4912-99d8-cdf7b0d4abf6" containerName="payload-pre-processing" containerID="cri-o://9098f223120447b5115fade54ed8b94337042c62615495447d6ed32cc237851d" gracePeriod=25 Oct 30 17:13:03.289641 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:03.289615 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-0ae3fe-5b445b79c8-cqkdf" Oct 30 17:13:03.310465 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:03.310444 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ae3fe-598c75dd67-pjf8j" Oct 30 17:13:03.366695 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:03.366664 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4739b888-b4ee-4912-99d8-cdf7b0d4abf6-kube-api-access-5mlbz\" (UniqueName: \"kubernetes.io/projected/4739b888-b4ee-4912-99d8-cdf7b0d4abf6-kube-api-access-5mlbz\") pod \"4739b888-b4ee-4912-99d8-cdf7b0d4abf6\" (UID: \"4739b888-b4ee-4912-99d8-cdf7b0d4abf6\") " Oct 30 17:13:03.366845 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:03.366783 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67f898fa-56b4-4f6d-8b02-3c9b24cedcf0-kube-api-access-t9p8q\" (UniqueName: \"kubernetes.io/projected/67f898fa-56b4-4f6d-8b02-3c9b24cedcf0-kube-api-access-t9p8q\") pod \"67f898fa-56b4-4f6d-8b02-3c9b24cedcf0\" (UID: \"67f898fa-56b4-4f6d-8b02-3c9b24cedcf0\") " Oct 30 17:13:03.366904 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:03.366859 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4739b888-b4ee-4912-99d8-cdf7b0d4abf6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4739b888-b4ee-4912-99d8-cdf7b0d4abf6-plugins-config-volume\") pod \"4739b888-b4ee-4912-99d8-cdf7b0d4abf6\" (UID: \"4739b888-b4ee-4912-99d8-cdf7b0d4abf6\") " Oct 30 17:13:03.367090 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:03.367067 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67f898fa-56b4-4f6d-8b02-3c9b24cedcf0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67f898fa-56b4-4f6d-8b02-3c9b24cedcf0-plugins-config-volume\") pod \"67f898fa-56b4-4f6d-8b02-3c9b24cedcf0\" (UID: \"67f898fa-56b4-4f6d-8b02-3c9b24cedcf0\") " Oct 30 17:13:03.367690 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:03.367650 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67f898fa-56b4-4f6d-8b02-3c9b24cedcf0-plugins-config-volume" pod "67f898fa-56b4-4f6d-8b02-3c9b24cedcf0" (UID: "67f898fa-56b4-4f6d-8b02-3c9b24cedcf0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:13:03.367931 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:03.367910 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4739b888-b4ee-4912-99d8-cdf7b0d4abf6-plugins-config-volume" pod "4739b888-b4ee-4912-99d8-cdf7b0d4abf6" (UID: "4739b888-b4ee-4912-99d8-cdf7b0d4abf6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:13:03.369656 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:03.369631 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4739b888-b4ee-4912-99d8-cdf7b0d4abf6-kube-api-access-5mlbz" pod "4739b888-b4ee-4912-99d8-cdf7b0d4abf6" (UID: "4739b888-b4ee-4912-99d8-cdf7b0d4abf6"). InnerVolumeSpecName "kube-api-access-5mlbz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:13:03.369735 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:03.369670 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67f898fa-56b4-4f6d-8b02-3c9b24cedcf0-kube-api-access-t9p8q" pod "67f898fa-56b4-4f6d-8b02-3c9b24cedcf0" (UID: "67f898fa-56b4-4f6d-8b02-3c9b24cedcf0"). InnerVolumeSpecName "kube-api-access-t9p8q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:13:03.469368 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:03.469331 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mlbz\" (UniqueName: \"kubernetes.io/projected/4739b888-b4ee-4912-99d8-cdf7b0d4abf6-kube-api-access-5mlbz\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:13:03.469368 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:03.469369 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9p8q\" (UniqueName: \"kubernetes.io/projected/67f898fa-56b4-4f6d-8b02-3c9b24cedcf0-kube-api-access-t9p8q\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:13:03.469570 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:03.469385 5052 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4739b888-b4ee-4912-99d8-cdf7b0d4abf6-plugins-config-volume\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:13:03.469570 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:03.469400 5052 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67f898fa-56b4-4f6d-8b02-3c9b24cedcf0-plugins-config-volume\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:13:03.744434 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:03.744396 5052 generic.go:356] "Generic (PLEG): container finished" podID="67f898fa-56b4-4f6d-8b02-3c9b24cedcf0" containerID="b5ea2bd3a524e7e634afbaf076904cb9f7eead38f1a0373069b73be26e58b3ea" exitCode=0 Oct 30 17:13:03.744939 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:03.744437 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-0ae3fe-5b445b79c8-cqkdf" Oct 30 17:13:03.744939 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:03.744477 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-0ae3fe-5b445b79c8-cqkdf" event={"ID":"67f898fa-56b4-4f6d-8b02-3c9b24cedcf0","Type":"ContainerDied","Data":"b5ea2bd3a524e7e634afbaf076904cb9f7eead38f1a0373069b73be26e58b3ea"} Oct 30 17:13:03.744939 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:03.744530 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-0ae3fe-5b445b79c8-cqkdf" event={"ID":"67f898fa-56b4-4f6d-8b02-3c9b24cedcf0","Type":"ContainerDied","Data":"ce08a7b81e098f28f5f88a7c4b368711689aaec692e6729c445816f50ee65d69"} Oct 30 17:13:03.744939 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:03.744562 5052 scope.go:122] "RemoveContainer" containerID="b5ea2bd3a524e7e634afbaf076904cb9f7eead38f1a0373069b73be26e58b3ea" Oct 30 17:13:03.746065 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:03.746033 5052 generic.go:356] "Generic (PLEG): container finished" podID="4739b888-b4ee-4912-99d8-cdf7b0d4abf6" containerID="9098f223120447b5115fade54ed8b94337042c62615495447d6ed32cc237851d" exitCode=0 Oct 30 17:13:03.746189 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:03.746085 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ae3fe-598c75dd67-pjf8j" Oct 30 17:13:03.746189 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:03.746093 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ae3fe-598c75dd67-pjf8j" event={"ID":"4739b888-b4ee-4912-99d8-cdf7b0d4abf6","Type":"ContainerDied","Data":"9098f223120447b5115fade54ed8b94337042c62615495447d6ed32cc237851d"} Oct 30 17:13:03.746189 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:03.746134 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ae3fe-598c75dd67-pjf8j" event={"ID":"4739b888-b4ee-4912-99d8-cdf7b0d4abf6","Type":"ContainerDied","Data":"36ecf1ee394e855f8230c7bff317a006e99a2f3dbf2738e3439e23f4ec03d5d4"} Oct 30 17:13:03.755572 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:03.755557 5052 scope.go:122] "RemoveContainer" containerID="b5ea2bd3a524e7e634afbaf076904cb9f7eead38f1a0373069b73be26e58b3ea" Oct 30 17:13:03.755848 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:13:03.755829 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5ea2bd3a524e7e634afbaf076904cb9f7eead38f1a0373069b73be26e58b3ea\": container with ID starting with b5ea2bd3a524e7e634afbaf076904cb9f7eead38f1a0373069b73be26e58b3ea not found: ID does not exist" containerID="b5ea2bd3a524e7e634afbaf076904cb9f7eead38f1a0373069b73be26e58b3ea" Oct 30 17:13:03.755923 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:03.755850 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5ea2bd3a524e7e634afbaf076904cb9f7eead38f1a0373069b73be26e58b3ea"} err="failed to get container status \"b5ea2bd3a524e7e634afbaf076904cb9f7eead38f1a0373069b73be26e58b3ea\": rpc error: code = NotFound desc = could not find container \"b5ea2bd3a524e7e634afbaf076904cb9f7eead38f1a0373069b73be26e58b3ea\": container with ID starting with b5ea2bd3a524e7e634afbaf076904cb9f7eead38f1a0373069b73be26e58b3ea not found: ID does not exist" Oct 30 17:13:03.755923 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:03.755866 5052 scope.go:122] "RemoveContainer" containerID="9098f223120447b5115fade54ed8b94337042c62615495447d6ed32cc237851d" Oct 30 17:13:03.767700 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:03.767684 5052 scope.go:122] "RemoveContainer" containerID="9098f223120447b5115fade54ed8b94337042c62615495447d6ed32cc237851d" Oct 30 17:13:03.767992 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:13:03.767969 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9098f223120447b5115fade54ed8b94337042c62615495447d6ed32cc237851d\": container with ID starting with 9098f223120447b5115fade54ed8b94337042c62615495447d6ed32cc237851d not found: ID does not exist" containerID="9098f223120447b5115fade54ed8b94337042c62615495447d6ed32cc237851d" Oct 30 17:13:03.768065 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:03.768001 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9098f223120447b5115fade54ed8b94337042c62615495447d6ed32cc237851d"} err="failed to get container status \"9098f223120447b5115fade54ed8b94337042c62615495447d6ed32cc237851d\": rpc error: code = NotFound desc = could not find container \"9098f223120447b5115fade54ed8b94337042c62615495447d6ed32cc237851d\": container with ID starting with 9098f223120447b5115fade54ed8b94337042c62615495447d6ed32cc237851d not found: ID does not exist" Oct 30 17:13:03.801813 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:03.801791 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0ae3fe-598c75dd67-pjf8j"] Oct 30 17:13:03.808766 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:03.808740 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0ae3fe-598c75dd67-pjf8j"] Oct 30 17:13:03.847040 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:03.847017 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0ae3fe-5b445b79c8-cqkdf"] Oct 30 17:13:03.854267 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:03.854242 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0ae3fe-5b445b79c8-cqkdf"] Oct 30 17:13:04.600456 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:04.600423 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4739b888-b4ee-4912-99d8-cdf7b0d4abf6" path="/var/lib/kubelet/pods/4739b888-b4ee-4912-99d8-cdf7b0d4abf6/volumes" Oct 30 17:13:04.601123 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:04.601004 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67f898fa-56b4-4f6d-8b02-3c9b24cedcf0" path="/var/lib/kubelet/pods/67f898fa-56b4-4f6d-8b02-3c9b24cedcf0/volumes" Oct 30 17:13:10.632064 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:10.632024 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-1b7bb3-7c468f545b-nlwlv"] Oct 30 17:13:10.633064 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:10.632228 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-1b7bb3-7c468f545b-nlwlv" podUID="7700cd37-52fa-4210-afa9-7771f758a8ba" containerName="maas-api" containerID="cri-o://913bb27ba70458c0c723c4fc44d50cc7add23bd167caca6c4d2cd5bb28f1240b" gracePeriod=30 Oct 30 17:13:10.835530 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:10.831971 5052 generic.go:356] "Generic (PLEG): container finished" podID="7700cd37-52fa-4210-afa9-7771f758a8ba" containerID="913bb27ba70458c0c723c4fc44d50cc7add23bd167caca6c4d2cd5bb28f1240b" exitCode=0 Oct 30 17:13:10.835530 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:10.832034 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-1b7bb3-7c468f545b-nlwlv" event={"ID":"7700cd37-52fa-4210-afa9-7771f758a8ba","Type":"ContainerDied","Data":"913bb27ba70458c0c723c4fc44d50cc7add23bd167caca6c4d2cd5bb28f1240b"} Oct 30 17:13:11.047086 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:11.047049 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-1b7bb3-7c468f545b-nlwlv" Oct 30 17:13:11.274141 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:11.272963 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7700cd37-52fa-4210-afa9-7771f758a8ba-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7700cd37-52fa-4210-afa9-7771f758a8ba-maas-api-tls\") pod \"7700cd37-52fa-4210-afa9-7771f758a8ba\" (UID: \"7700cd37-52fa-4210-afa9-7771f758a8ba\") " Oct 30 17:13:11.274141 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:11.273106 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7700cd37-52fa-4210-afa9-7771f758a8ba-kube-api-access-crkc6\" (UniqueName: \"kubernetes.io/projected/7700cd37-52fa-4210-afa9-7771f758a8ba-kube-api-access-crkc6\") pod \"7700cd37-52fa-4210-afa9-7771f758a8ba\" (UID: \"7700cd37-52fa-4210-afa9-7771f758a8ba\") " Oct 30 17:13:11.274141 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:11.273145 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7700cd37-52fa-4210-afa9-7771f758a8ba-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7700cd37-52fa-4210-afa9-7771f758a8ba-openshift-service-ca\") pod \"7700cd37-52fa-4210-afa9-7771f758a8ba\" (UID: \"7700cd37-52fa-4210-afa9-7771f758a8ba\") " Oct 30 17:13:11.274141 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:11.273433 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7700cd37-52fa-4210-afa9-7771f758a8ba-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7700cd37-52fa-4210-afa9-7771f758a8ba-metrics-certs\") pod \"7700cd37-52fa-4210-afa9-7771f758a8ba\" (UID: \"7700cd37-52fa-4210-afa9-7771f758a8ba\") " Oct 30 17:13:11.274679 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:11.274651 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7700cd37-52fa-4210-afa9-7771f758a8ba-openshift-service-ca" pod "7700cd37-52fa-4210-afa9-7771f758a8ba" (UID: "7700cd37-52fa-4210-afa9-7771f758a8ba"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:13:11.355602 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:11.351897 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7700cd37-52fa-4210-afa9-7771f758a8ba-metrics-certs" pod "7700cd37-52fa-4210-afa9-7771f758a8ba" (UID: "7700cd37-52fa-4210-afa9-7771f758a8ba"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:13:11.355602 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:11.352439 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7700cd37-52fa-4210-afa9-7771f758a8ba-maas-api-tls" pod "7700cd37-52fa-4210-afa9-7771f758a8ba" (UID: "7700cd37-52fa-4210-afa9-7771f758a8ba"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:13:11.355602 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:11.353059 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7700cd37-52fa-4210-afa9-7771f758a8ba-kube-api-access-crkc6" pod "7700cd37-52fa-4210-afa9-7771f758a8ba" (UID: "7700cd37-52fa-4210-afa9-7771f758a8ba"). InnerVolumeSpecName "kube-api-access-crkc6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:13:11.375745 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:11.375712 5052 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7700cd37-52fa-4210-afa9-7771f758a8ba-metrics-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:13:11.375745 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:11.375740 5052 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7700cd37-52fa-4210-afa9-7771f758a8ba-maas-api-tls\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:13:11.375745 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:11.375749 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crkc6\" (UniqueName: \"kubernetes.io/projected/7700cd37-52fa-4210-afa9-7771f758a8ba-kube-api-access-crkc6\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:13:11.375974 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:11.375759 5052 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7700cd37-52fa-4210-afa9-7771f758a8ba-openshift-service-ca\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:13:11.840614 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:11.840557 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-1b7bb3-7c468f545b-nlwlv" Oct 30 17:13:11.840614 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:11.840566 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-1b7bb3-7c468f545b-nlwlv" event={"ID":"7700cd37-52fa-4210-afa9-7771f758a8ba","Type":"ContainerDied","Data":"4efabf9fd01572b6c6a721c60106e12a1282beae3c5a851cb7a737f991376a60"} Oct 30 17:13:11.841215 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:11.840666 5052 scope.go:122] "RemoveContainer" containerID="913bb27ba70458c0c723c4fc44d50cc7add23bd167caca6c4d2cd5bb28f1240b" Oct 30 17:13:11.898108 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:11.898078 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-1b7bb3-7c468f545b-nlwlv"] Oct 30 17:13:11.911477 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:11.911450 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-1b7bb3-7c468f545b-nlwlv"] Oct 30 17:13:12.377737 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:12.377703 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cdwcj"] Oct 30 17:13:12.607589 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:12.607542 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7700cd37-52fa-4210-afa9-7771f758a8ba" path="/var/lib/kubelet/pods/7700cd37-52fa-4210-afa9-7771f758a8ba/volumes" Oct 30 17:13:13.809603 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:13.797781 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ddc10d-55f45f45f-4dt5v" Oct 30 17:13:13.820602 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:13.819764 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-ddc10d-55c9674d94-hwbjl" Oct 30 17:13:18.506922 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:18.506886 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-hmw2l"] Oct 30 17:13:18.507735 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:18.507693 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-hmw2l" podUID="952ac6ad-c9ff-4078-b361-201e411053de" containerName="istio-proxy" containerID="cri-o://5299695c33208f41d7994a89ac6c857de3b4177d02fb5c5b37705b4a012ce689" gracePeriod=30 Oct 30 17:13:18.785816 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:18.785669 5052 patch_prober.go:29] interesting pod/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-hmw2l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 17:13:18.787714 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:18.786220 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-hmw2l" podUID="952ac6ad-c9ff-4078-b361-201e411053de" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 17:13:18.941659 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:13:18.941615 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 17:13:20.901785 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:20.901757 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-4ptgl_973d1ea4-3981-4350-b719-edb7911c23ab/payload-processing/0.log" Oct 30 17:13:21.098519 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:21.098483 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-4ptgl_973d1ea4-3981-4350-b719-edb7911c23ab/payload-processing/0.log" Oct 30 17:13:23.805094 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:23.805063 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-hmw2l" Oct 30 17:13:23.905333 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:23.905299 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/952ac6ad-c9ff-4078-b361-201e411053de-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/952ac6ad-c9ff-4078-b361-201e411053de-workload-socket\") pod \"952ac6ad-c9ff-4078-b361-201e411053de\" (UID: \"952ac6ad-c9ff-4078-b361-201e411053de\") " Oct 30 17:13:23.905530 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:23.905351 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/952ac6ad-c9ff-4078-b361-201e411053de-istio-token\" (UniqueName: \"kubernetes.io/projected/952ac6ad-c9ff-4078-b361-201e411053de-istio-token\") pod \"952ac6ad-c9ff-4078-b361-201e411053de\" (UID: \"952ac6ad-c9ff-4078-b361-201e411053de\") " Oct 30 17:13:23.905530 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:23.905387 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/952ac6ad-c9ff-4078-b361-201e411053de-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/952ac6ad-c9ff-4078-b361-201e411053de-istio-podinfo\") pod \"952ac6ad-c9ff-4078-b361-201e411053de\" (UID: \"952ac6ad-c9ff-4078-b361-201e411053de\") " Oct 30 17:13:23.905530 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:23.905434 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/952ac6ad-c9ff-4078-b361-201e411053de-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/952ac6ad-c9ff-4078-b361-201e411053de-istiod-ca-cert\") pod \"952ac6ad-c9ff-4078-b361-201e411053de\" (UID: \"952ac6ad-c9ff-4078-b361-201e411053de\") " Oct 30 17:13:23.905530 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:23.905473 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/952ac6ad-c9ff-4078-b361-201e411053de-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/952ac6ad-c9ff-4078-b361-201e411053de-istio-data\") pod \"952ac6ad-c9ff-4078-b361-201e411053de\" (UID: \"952ac6ad-c9ff-4078-b361-201e411053de\") " Oct 30 17:13:23.905842 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:23.905534 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/952ac6ad-c9ff-4078-b361-201e411053de-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/952ac6ad-c9ff-4078-b361-201e411053de-istio-envoy\") pod \"952ac6ad-c9ff-4078-b361-201e411053de\" (UID: \"952ac6ad-c9ff-4078-b361-201e411053de\") " Oct 30 17:13:23.905842 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:23.905560 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/952ac6ad-c9ff-4078-b361-201e411053de-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/952ac6ad-c9ff-4078-b361-201e411053de-credential-socket\") pod \"952ac6ad-c9ff-4078-b361-201e411053de\" (UID: \"952ac6ad-c9ff-4078-b361-201e411053de\") " Oct 30 17:13:23.905842 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:23.905651 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/952ac6ad-c9ff-4078-b361-201e411053de-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/952ac6ad-c9ff-4078-b361-201e411053de-workload-certs\") pod \"952ac6ad-c9ff-4078-b361-201e411053de\" (UID: \"952ac6ad-c9ff-4078-b361-201e411053de\") " Oct 30 17:13:23.905842 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:23.905683 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/952ac6ad-c9ff-4078-b361-201e411053de-workload-socket" pod "952ac6ad-c9ff-4078-b361-201e411053de" (UID: "952ac6ad-c9ff-4078-b361-201e411053de"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:13:23.905842 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:23.905705 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/952ac6ad-c9ff-4078-b361-201e411053de-kube-api-access-vsvmf\" (UniqueName: \"kubernetes.io/projected/952ac6ad-c9ff-4078-b361-201e411053de-kube-api-access-vsvmf\") pod \"952ac6ad-c9ff-4078-b361-201e411053de\" (UID: \"952ac6ad-c9ff-4078-b361-201e411053de\") " Oct 30 17:13:23.906129 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:23.905960 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/952ac6ad-c9ff-4078-b361-201e411053de-istiod-ca-cert" pod "952ac6ad-c9ff-4078-b361-201e411053de" (UID: "952ac6ad-c9ff-4078-b361-201e411053de"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:13:23.906129 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:23.905947 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/952ac6ad-c9ff-4078-b361-201e411053de-credential-socket" pod "952ac6ad-c9ff-4078-b361-201e411053de" (UID: "952ac6ad-c9ff-4078-b361-201e411053de"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:13:23.906254 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:23.906130 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/952ac6ad-c9ff-4078-b361-201e411053de-istio-data" pod "952ac6ad-c9ff-4078-b361-201e411053de" (UID: "952ac6ad-c9ff-4078-b361-201e411053de"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:13:23.906254 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:23.906192 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/952ac6ad-c9ff-4078-b361-201e411053de-workload-certs" pod "952ac6ad-c9ff-4078-b361-201e411053de" (UID: "952ac6ad-c9ff-4078-b361-201e411053de"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:13:23.907060 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:23.907033 5052 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/952ac6ad-c9ff-4078-b361-201e411053de-istio-data\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:13:23.907236 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:23.907221 5052 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/952ac6ad-c9ff-4078-b361-201e411053de-credential-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:13:23.907360 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:23.907341 5052 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/952ac6ad-c9ff-4078-b361-201e411053de-workload-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:13:23.907508 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:23.907486 5052 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/952ac6ad-c9ff-4078-b361-201e411053de-workload-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:13:23.907720 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:23.907509 5052 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/952ac6ad-c9ff-4078-b361-201e411053de-istiod-ca-cert\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:13:23.908423 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:23.908400 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/952ac6ad-c9ff-4078-b361-201e411053de-istio-envoy" pod "952ac6ad-c9ff-4078-b361-201e411053de" (UID: "952ac6ad-c9ff-4078-b361-201e411053de"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:13:23.908619 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:23.908571 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/952ac6ad-c9ff-4078-b361-201e411053de-kube-api-access-vsvmf" pod "952ac6ad-c9ff-4078-b361-201e411053de" (UID: "952ac6ad-c9ff-4078-b361-201e411053de"). InnerVolumeSpecName "kube-api-access-vsvmf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:13:23.908695 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:23.908663 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/952ac6ad-c9ff-4078-b361-201e411053de-istio-token" pod "952ac6ad-c9ff-4078-b361-201e411053de" (UID: "952ac6ad-c9ff-4078-b361-201e411053de"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:13:23.908806 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:23.908782 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/952ac6ad-c9ff-4078-b361-201e411053de-istio-podinfo" pod "952ac6ad-c9ff-4078-b361-201e411053de" (UID: "952ac6ad-c9ff-4078-b361-201e411053de"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:13:23.964965 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:23.964934 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-967fac/distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx"] Oct 30 17:13:23.970173 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:23.970144 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-967fac/distinct-w5-967fac-kserve-69cbcccd8b-b85bn"] Oct 30 17:13:24.006849 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:24.006811 5052 generic.go:356] "Generic (PLEG): container finished" podID="952ac6ad-c9ff-4078-b361-201e411053de" containerID="5299695c33208f41d7994a89ac6c857de3b4177d02fb5c5b37705b4a012ce689" exitCode=0 Oct 30 17:13:24.007014 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:24.006865 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-hmw2l" Oct 30 17:13:24.007014 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:24.006864 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-hmw2l" event={"ID":"952ac6ad-c9ff-4078-b361-201e411053de","Type":"ContainerDied","Data":"5299695c33208f41d7994a89ac6c857de3b4177d02fb5c5b37705b4a012ce689"} Oct 30 17:13:24.007127 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:24.007007 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-hmw2l" event={"ID":"952ac6ad-c9ff-4078-b361-201e411053de","Type":"ContainerDied","Data":"02bf8af5b14b9f869522fab434a6cdacda3b7258d150632f042a61d77f8b7849"} Oct 30 17:13:24.007127 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:24.007048 5052 scope.go:122] "RemoveContainer" containerID="5299695c33208f41d7994a89ac6c857de3b4177d02fb5c5b37705b4a012ce689" Oct 30 17:13:24.008805 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:24.008780 5052 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/952ac6ad-c9ff-4078-b361-201e411053de-istio-envoy\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:13:24.008921 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:24.008809 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsvmf\" (UniqueName: \"kubernetes.io/projected/952ac6ad-c9ff-4078-b361-201e411053de-kube-api-access-vsvmf\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:13:24.008921 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:24.008826 5052 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/952ac6ad-c9ff-4078-b361-201e411053de-istio-token\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:13:24.008921 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:24.008841 5052 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/952ac6ad-c9ff-4078-b361-201e411053de-istio-podinfo\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:13:24.021799 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:24.019182 5052 scope.go:122] "RemoveContainer" containerID="5299695c33208f41d7994a89ac6c857de3b4177d02fb5c5b37705b4a012ce689" Oct 30 17:13:24.021799 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:13:24.019810 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5299695c33208f41d7994a89ac6c857de3b4177d02fb5c5b37705b4a012ce689\": container with ID starting with 5299695c33208f41d7994a89ac6c857de3b4177d02fb5c5b37705b4a012ce689 not found: ID does not exist" containerID="5299695c33208f41d7994a89ac6c857de3b4177d02fb5c5b37705b4a012ce689" Oct 30 17:13:24.021799 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:24.019836 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5299695c33208f41d7994a89ac6c857de3b4177d02fb5c5b37705b4a012ce689"} err="failed to get container status \"5299695c33208f41d7994a89ac6c857de3b4177d02fb5c5b37705b4a012ce689\": rpc error: code = NotFound desc = could not find container \"5299695c33208f41d7994a89ac6c857de3b4177d02fb5c5b37705b4a012ce689\": container with ID starting with 5299695c33208f41d7994a89ac6c857de3b4177d02fb5c5b37705b4a012ce689 not found: ID does not exist" Oct 30 17:13:24.113994 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:24.113893 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-hmw2l"] Oct 30 17:13:24.118053 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:24.118023 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-hmw2l"] Oct 30 17:13:24.599606 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:24.599561 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="952ac6ad-c9ff-4078-b361-201e411053de" path="/var/lib/kubelet/pods/952ac6ad-c9ff-4078-b361-201e411053de/volumes" Oct 30 17:13:27.235768 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:27.230728 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4def7e-6d5bf4d5c8-62jp6"] Oct 30 17:13:27.235768 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:27.231908 5052 cpu_manager.go:414] "containerMap: removing container" podUID="952ac6ad-c9ff-4078-b361-201e411053de" containerName="istio-proxy" Oct 30 17:13:27.235768 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:27.231931 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="952ac6ad-c9ff-4078-b361-201e411053de" containerName="istio-proxy" Oct 30 17:13:27.235768 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:27.231967 5052 cpu_manager.go:414] "containerMap: removing container" podUID="7700cd37-52fa-4210-afa9-7771f758a8ba" containerName="maas-api" Oct 30 17:13:27.235768 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:27.231976 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7700cd37-52fa-4210-afa9-7771f758a8ba" containerName="maas-api" Oct 30 17:13:27.235768 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:27.232020 5052 cpu_manager.go:414] "containerMap: removing container" podUID="67f898fa-56b4-4f6d-8b02-3c9b24cedcf0" containerName="payload-processing" Oct 30 17:13:27.235768 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:27.232028 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67f898fa-56b4-4f6d-8b02-3c9b24cedcf0" containerName="payload-processing" Oct 30 17:13:27.235768 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:27.232065 5052 cpu_manager.go:414] "containerMap: removing container" podUID="4739b888-b4ee-4912-99d8-cdf7b0d4abf6" containerName="payload-pre-processing" Oct 30 17:13:27.235768 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:27.232075 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4739b888-b4ee-4912-99d8-cdf7b0d4abf6" containerName="payload-pre-processing" Oct 30 17:13:27.235768 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:27.232253 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="952ac6ad-c9ff-4078-b361-201e411053de" containerName="istio-proxy" Oct 30 17:13:27.235768 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:27.232273 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="7700cd37-52fa-4210-afa9-7771f758a8ba" containerName="maas-api" Oct 30 17:13:27.235768 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:27.232292 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="67f898fa-56b4-4f6d-8b02-3c9b24cedcf0" containerName="payload-processing" Oct 30 17:13:27.235768 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:27.232306 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="4739b888-b4ee-4912-99d8-cdf7b0d4abf6" containerName="payload-pre-processing" Oct 30 17:13:27.238930 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:27.238892 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4def7e-6d5bf4d5c8-62jp6"] Oct 30 17:13:27.239296 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:27.239195 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-4def7e-6d5bf4d5c8-62jp6" Oct 30 17:13:27.276635 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:27.269966 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-4def7e\"" Oct 30 17:13:27.276635 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:27.270081 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-4def7e-dockercfg-cfbgg\"" Oct 30 17:13:27.415560 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:27.415525 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/358f6b76-be1e-474d-bf65-28c808a052ce-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-4def7e-6d5bf4d5c8-62jp6\" (UID: \"358f6b76-be1e-474d-bf65-28c808a052ce\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4def7e-6d5bf4d5c8-62jp6" Oct 30 17:13:27.415765 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:27.415608 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snj7t\" (UniqueName: \"kubernetes.io/projected/358f6b76-be1e-474d-bf65-28c808a052ce-kube-api-access-snj7t\") pod \"payload-processing-e2e-ipp-b-4def7e-6d5bf4d5c8-62jp6\" (UID: \"358f6b76-be1e-474d-bf65-28c808a052ce\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4def7e-6d5bf4d5c8-62jp6" Oct 30 17:13:27.517137 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:27.517045 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/358f6b76-be1e-474d-bf65-28c808a052ce-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-4def7e-6d5bf4d5c8-62jp6\" (UID: \"358f6b76-be1e-474d-bf65-28c808a052ce\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4def7e-6d5bf4d5c8-62jp6" Oct 30 17:13:27.517307 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:27.517137 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snj7t\" (UniqueName: \"kubernetes.io/projected/358f6b76-be1e-474d-bf65-28c808a052ce-kube-api-access-snj7t\") pod \"payload-processing-e2e-ipp-b-4def7e-6d5bf4d5c8-62jp6\" (UID: \"358f6b76-be1e-474d-bf65-28c808a052ce\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4def7e-6d5bf4d5c8-62jp6" Oct 30 17:13:27.517880 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:27.517858 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/358f6b76-be1e-474d-bf65-28c808a052ce-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-4def7e-6d5bf4d5c8-62jp6\" (UID: \"358f6b76-be1e-474d-bf65-28c808a052ce\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4def7e-6d5bf4d5c8-62jp6" Oct 30 17:13:27.542249 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:27.542217 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snj7t\" (UniqueName: \"kubernetes.io/projected/358f6b76-be1e-474d-bf65-28c808a052ce-kube-api-access-snj7t\") pod \"payload-processing-e2e-ipp-b-4def7e-6d5bf4d5c8-62jp6\" (UID: \"358f6b76-be1e-474d-bf65-28c808a052ce\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4def7e-6d5bf4d5c8-62jp6" Oct 30 17:13:27.566849 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:27.566814 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-4def7e-6d5bf4d5c8-62jp6" Oct 30 17:13:27.901591 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:27.901546 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4def7e-6d5bf4d5c8-62jp6"] Oct 30 17:13:27.902218 ip-10-0-131-136 kubenswrapper[5052]: W1030 17:13:27.902193 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod358f6b76_be1e_474d_bf65_28c808a052ce.slice/crio-f3fac603f8c1f585ddb9920dc3a0270c862a9539e28ece915db1aa9836eac682 WatchSource:0}: Error finding container f3fac603f8c1f585ddb9920dc3a0270c862a9539e28ece915db1aa9836eac682: Status 404 returned error can't find the container with id f3fac603f8c1f585ddb9920dc3a0270c862a9539e28ece915db1aa9836eac682 Oct 30 17:13:28.243180 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:28.243073 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-4def7e-6d5bf4d5c8-62jp6" event={"ID":"358f6b76-be1e-474d-bf65-28c808a052ce","Type":"ContainerStarted","Data":"f3fac603f8c1f585ddb9920dc3a0270c862a9539e28ece915db1aa9836eac682"} Oct 30 17:13:30.407037 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:30.406989 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-4def7e-6d5bf4d5c8-62jp6" event={"ID":"358f6b76-be1e-474d-bf65-28c808a052ce","Type":"ContainerStarted","Data":"8af1e80341c6fe659a966431b25a75bc29e36fed602850289582fc8cc9d7ef48"} Oct 30 17:13:30.408567 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:30.408535 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-4def7e-6d5bf4d5c8-62jp6" Oct 30 17:13:30.482603 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:30.481305 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-4def7e-6d5bf4d5c8-62jp6" podStartSLOduration=1.743223277 podStartE2EDuration="3.481282026s" totalImagesPullingTime="1.738058749s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:13:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:13:30.460456467 +0000 UTC m=+3165.987791688" watchObservedRunningTime="2026-10-30 17:13:30.481282026 +0000 UTC m=+3166.008617247" Oct 30 17:13:38.994725 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:38.994687 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-967fac/distinct-w5-967fac-kserve-69cbcccd8b-b85bn" podUID="8f6b9354-378f-4451-9ff7-be7f7fec12f2" containerName="main" containerID="cri-o://9494a34003b003d9843e8c7adfb410a88e495c6f69d79d636b12d3a550be0fba" gracePeriod=45 Oct 30 17:13:38.995164 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:38.994743 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-967fac/distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx" podUID="bd662f3a-659f-477c-b8d5-94c64288c0c1" containerName="main" containerID="cri-o://ad5251608a3f1bb102f143be62b37a74a7d717543190a34039207ec335cf92a9" gracePeriod=45 Oct 30 17:13:39.048448 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:39.048415 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/distinct-w5-967fac-kserve-69cbcccd8b-b85bn" podUID="8f6b9354-378f-4451-9ff7-be7f7fec12f2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": dial tcp 10.132.0.76:8000: connect: connection refused" Oct 30 17:13:39.052655 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:39.052627 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx" podUID="bd662f3a-659f-477c-b8d5-94c64288c0c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": dial tcp 10.132.0.78:8000: connect: connection refused" Oct 30 17:13:39.281991 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:39.281959 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-967fac/distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx" Oct 30 17:13:39.322274 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:39.322251 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-967fac/distinct-w5-967fac-kserve-69cbcccd8b-b85bn" Oct 30 17:13:39.330911 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:39.330885 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd662f3a-659f-477c-b8d5-94c64288c0c1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd662f3a-659f-477c-b8d5-94c64288c0c1-dshm\") pod \"bd662f3a-659f-477c-b8d5-94c64288c0c1\" (UID: \"bd662f3a-659f-477c-b8d5-94c64288c0c1\") " Oct 30 17:13:39.331043 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:39.330964 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd662f3a-659f-477c-b8d5-94c64288c0c1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd662f3a-659f-477c-b8d5-94c64288c0c1-model-cache\") pod \"bd662f3a-659f-477c-b8d5-94c64288c0c1\" (UID: \"bd662f3a-659f-477c-b8d5-94c64288c0c1\") " Oct 30 17:13:39.331043 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:39.330998 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd662f3a-659f-477c-b8d5-94c64288c0c1-home\" (UniqueName: \"kubernetes.io/empty-dir/bd662f3a-659f-477c-b8d5-94c64288c0c1-home\") pod \"bd662f3a-659f-477c-b8d5-94c64288c0c1\" (UID: \"bd662f3a-659f-477c-b8d5-94c64288c0c1\") " Oct 30 17:13:39.331199 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:39.331173 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd662f3a-659f-477c-b8d5-94c64288c0c1-kube-api-access-vkjvr\" (UniqueName: \"kubernetes.io/projected/bd662f3a-659f-477c-b8d5-94c64288c0c1-kube-api-access-vkjvr\") pod \"bd662f3a-659f-477c-b8d5-94c64288c0c1\" (UID: \"bd662f3a-659f-477c-b8d5-94c64288c0c1\") " Oct 30 17:13:39.331259 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:39.331217 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd662f3a-659f-477c-b8d5-94c64288c0c1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd662f3a-659f-477c-b8d5-94c64288c0c1-tmp-dir\") pod \"bd662f3a-659f-477c-b8d5-94c64288c0c1\" (UID: \"bd662f3a-659f-477c-b8d5-94c64288c0c1\") " Oct 30 17:13:39.331259 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:39.331248 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd662f3a-659f-477c-b8d5-94c64288c0c1-tls-certs\" (UniqueName: \"kubernetes.io/secret/bd662f3a-659f-477c-b8d5-94c64288c0c1-tls-certs\") pod \"bd662f3a-659f-477c-b8d5-94c64288c0c1\" (UID: \"bd662f3a-659f-477c-b8d5-94c64288c0c1\") " Oct 30 17:13:39.331368 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:39.331274 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd662f3a-659f-477c-b8d5-94c64288c0c1-model-cache" pod "bd662f3a-659f-477c-b8d5-94c64288c0c1" (UID: "bd662f3a-659f-477c-b8d5-94c64288c0c1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:13:39.331368 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:39.331312 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd662f3a-659f-477c-b8d5-94c64288c0c1-home" pod "bd662f3a-659f-477c-b8d5-94c64288c0c1" (UID: "bd662f3a-659f-477c-b8d5-94c64288c0c1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:13:39.331519 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:39.331499 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd662f3a-659f-477c-b8d5-94c64288c0c1-tmp-dir" pod "bd662f3a-659f-477c-b8d5-94c64288c0c1" (UID: "bd662f3a-659f-477c-b8d5-94c64288c0c1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:13:39.332203 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:39.332178 5052 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd662f3a-659f-477c-b8d5-94c64288c0c1-tmp-dir\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:13:39.332319 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:39.332208 5052 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd662f3a-659f-477c-b8d5-94c64288c0c1-model-cache\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:13:39.332319 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:39.332223 5052 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd662f3a-659f-477c-b8d5-94c64288c0c1-home\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:13:39.333573 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:39.333547 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd662f3a-659f-477c-b8d5-94c64288c0c1-dshm" pod "bd662f3a-659f-477c-b8d5-94c64288c0c1" (UID: "bd662f3a-659f-477c-b8d5-94c64288c0c1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:13:39.333721 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:39.333675 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd662f3a-659f-477c-b8d5-94c64288c0c1-tls-certs" pod "bd662f3a-659f-477c-b8d5-94c64288c0c1" (UID: "bd662f3a-659f-477c-b8d5-94c64288c0c1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:13:39.333859 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:39.333836 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd662f3a-659f-477c-b8d5-94c64288c0c1-kube-api-access-vkjvr" pod "bd662f3a-659f-477c-b8d5-94c64288c0c1" (UID: "bd662f3a-659f-477c-b8d5-94c64288c0c1"). InnerVolumeSpecName "kube-api-access-vkjvr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:13:39.433651 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:39.433565 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f6b9354-378f-4451-9ff7-be7f7fec12f2-kube-api-access-5g7b2\" (UniqueName: \"kubernetes.io/projected/8f6b9354-378f-4451-9ff7-be7f7fec12f2-kube-api-access-5g7b2\") pod \"8f6b9354-378f-4451-9ff7-be7f7fec12f2\" (UID: \"8f6b9354-378f-4451-9ff7-be7f7fec12f2\") " Oct 30 17:13:39.433850 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:39.433752 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f6b9354-378f-4451-9ff7-be7f7fec12f2-home\" (UniqueName: \"kubernetes.io/empty-dir/8f6b9354-378f-4451-9ff7-be7f7fec12f2-home\") pod \"8f6b9354-378f-4451-9ff7-be7f7fec12f2\" (UID: \"8f6b9354-378f-4451-9ff7-be7f7fec12f2\") " Oct 30 17:13:39.433850 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:39.433793 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f6b9354-378f-4451-9ff7-be7f7fec12f2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f6b9354-378f-4451-9ff7-be7f7fec12f2-model-cache\") pod \"8f6b9354-378f-4451-9ff7-be7f7fec12f2\" (UID: \"8f6b9354-378f-4451-9ff7-be7f7fec12f2\") " Oct 30 17:13:39.434038 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:39.434021 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f6b9354-378f-4451-9ff7-be7f7fec12f2-tls-certs\" (UniqueName: \"kubernetes.io/secret/8f6b9354-378f-4451-9ff7-be7f7fec12f2-tls-certs\") pod \"8f6b9354-378f-4451-9ff7-be7f7fec12f2\" (UID: \"8f6b9354-378f-4451-9ff7-be7f7fec12f2\") " Oct 30 17:13:39.434125 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:39.434109 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f6b9354-378f-4451-9ff7-be7f7fec12f2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f6b9354-378f-4451-9ff7-be7f7fec12f2-dshm\") pod \"8f6b9354-378f-4451-9ff7-be7f7fec12f2\" (UID: \"8f6b9354-378f-4451-9ff7-be7f7fec12f2\") " Oct 30 17:13:39.434213 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:39.434199 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f6b9354-378f-4451-9ff7-be7f7fec12f2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f6b9354-378f-4451-9ff7-be7f7fec12f2-tmp-dir\") pod \"8f6b9354-378f-4451-9ff7-be7f7fec12f2\" (UID: \"8f6b9354-378f-4451-9ff7-be7f7fec12f2\") " Oct 30 17:13:39.434292 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:39.434013 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f6b9354-378f-4451-9ff7-be7f7fec12f2-home" pod "8f6b9354-378f-4451-9ff7-be7f7fec12f2" (UID: "8f6b9354-378f-4451-9ff7-be7f7fec12f2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:13:39.434292 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:39.434208 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f6b9354-378f-4451-9ff7-be7f7fec12f2-model-cache" pod "8f6b9354-378f-4451-9ff7-be7f7fec12f2" (UID: "8f6b9354-378f-4451-9ff7-be7f7fec12f2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:13:39.434808 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:39.434761 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f6b9354-378f-4451-9ff7-be7f7fec12f2-tmp-dir" pod "8f6b9354-378f-4451-9ff7-be7f7fec12f2" (UID: "8f6b9354-378f-4451-9ff7-be7f7fec12f2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:13:39.434936 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:39.434920 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkjvr\" (UniqueName: \"kubernetes.io/projected/bd662f3a-659f-477c-b8d5-94c64288c0c1-kube-api-access-vkjvr\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:13:39.435005 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:39.434943 5052 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd662f3a-659f-477c-b8d5-94c64288c0c1-tls-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:13:39.435005 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:39.434957 5052 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd662f3a-659f-477c-b8d5-94c64288c0c1-dshm\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:13:39.435005 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:39.434972 5052 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f6b9354-378f-4451-9ff7-be7f7fec12f2-tmp-dir\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:13:39.435005 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:39.434984 5052 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f6b9354-378f-4451-9ff7-be7f7fec12f2-home\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:13:39.435005 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:39.434996 5052 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f6b9354-378f-4451-9ff7-be7f7fec12f2-model-cache\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:13:39.436755 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:39.436727 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f6b9354-378f-4451-9ff7-be7f7fec12f2-kube-api-access-5g7b2" pod "8f6b9354-378f-4451-9ff7-be7f7fec12f2" (UID: "8f6b9354-378f-4451-9ff7-be7f7fec12f2"). InnerVolumeSpecName "kube-api-access-5g7b2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:13:39.436861 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:39.436757 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f6b9354-378f-4451-9ff7-be7f7fec12f2-dshm" pod "8f6b9354-378f-4451-9ff7-be7f7fec12f2" (UID: "8f6b9354-378f-4451-9ff7-be7f7fec12f2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:13:39.437040 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:39.437019 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f6b9354-378f-4451-9ff7-be7f7fec12f2-tls-certs" pod "8f6b9354-378f-4451-9ff7-be7f7fec12f2" (UID: "8f6b9354-378f-4451-9ff7-be7f7fec12f2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:13:39.477077 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:39.477050 5052 generic.go:356] "Generic (PLEG): container finished" podID="8f6b9354-378f-4451-9ff7-be7f7fec12f2" containerID="9494a34003b003d9843e8c7adfb410a88e495c6f69d79d636b12d3a550be0fba" exitCode=0 Oct 30 17:13:39.477230 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:39.477095 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-967fac/distinct-w5-967fac-kserve-69cbcccd8b-b85bn" Oct 30 17:13:39.477230 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:39.477132 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-967fac/distinct-w5-967fac-kserve-69cbcccd8b-b85bn" event={"ID":"8f6b9354-378f-4451-9ff7-be7f7fec12f2","Type":"ContainerDied","Data":"9494a34003b003d9843e8c7adfb410a88e495c6f69d79d636b12d3a550be0fba"} Oct 30 17:13:39.477230 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:39.477179 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-967fac/distinct-w5-967fac-kserve-69cbcccd8b-b85bn" event={"ID":"8f6b9354-378f-4451-9ff7-be7f7fec12f2","Type":"ContainerDied","Data":"e9d2fd09af0ecc8dbede5c3ad921446c488ec3268a14e9a13fe8dc0c6d5c8b78"} Oct 30 17:13:39.477416 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:39.477230 5052 scope.go:122] "RemoveContainer" containerID="9494a34003b003d9843e8c7adfb410a88e495c6f69d79d636b12d3a550be0fba" Oct 30 17:13:39.478636 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:39.478616 5052 generic.go:356] "Generic (PLEG): container finished" podID="bd662f3a-659f-477c-b8d5-94c64288c0c1" containerID="ad5251608a3f1bb102f143be62b37a74a7d717543190a34039207ec335cf92a9" exitCode=0 Oct 30 17:13:39.478751 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:39.478673 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-967fac/distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx" Oct 30 17:13:39.478799 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:39.478672 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-967fac/distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx" event={"ID":"bd662f3a-659f-477c-b8d5-94c64288c0c1","Type":"ContainerDied","Data":"ad5251608a3f1bb102f143be62b37a74a7d717543190a34039207ec335cf92a9"} Oct 30 17:13:39.478799 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:39.478782 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-967fac/distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx" event={"ID":"bd662f3a-659f-477c-b8d5-94c64288c0c1","Type":"ContainerDied","Data":"654c0ad8a3784f9c6ba80cbd68181439d0451cd32849d3279ae8ec2e452a92ad"} Oct 30 17:13:39.504695 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:39.504673 5052 scope.go:122] "RemoveContainer" containerID="9494a34003b003d9843e8c7adfb410a88e495c6f69d79d636b12d3a550be0fba" Oct 30 17:13:39.504999 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:13:39.504978 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9494a34003b003d9843e8c7adfb410a88e495c6f69d79d636b12d3a550be0fba\": container with ID starting with 9494a34003b003d9843e8c7adfb410a88e495c6f69d79d636b12d3a550be0fba not found: ID does not exist" containerID="9494a34003b003d9843e8c7adfb410a88e495c6f69d79d636b12d3a550be0fba" Oct 30 17:13:39.505050 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:39.505010 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9494a34003b003d9843e8c7adfb410a88e495c6f69d79d636b12d3a550be0fba"} err="failed to get container status \"9494a34003b003d9843e8c7adfb410a88e495c6f69d79d636b12d3a550be0fba\": rpc error: code = NotFound desc = could not find container \"9494a34003b003d9843e8c7adfb410a88e495c6f69d79d636b12d3a550be0fba\": container with ID starting with 9494a34003b003d9843e8c7adfb410a88e495c6f69d79d636b12d3a550be0fba not found: ID does not exist" Oct 30 17:13:39.505050 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:39.505028 5052 scope.go:122] "RemoveContainer" containerID="ad5251608a3f1bb102f143be62b37a74a7d717543190a34039207ec335cf92a9" Oct 30 17:13:39.515293 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:39.515270 5052 scope.go:122] "RemoveContainer" containerID="ad5251608a3f1bb102f143be62b37a74a7d717543190a34039207ec335cf92a9" Oct 30 17:13:39.515543 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:13:39.515526 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad5251608a3f1bb102f143be62b37a74a7d717543190a34039207ec335cf92a9\": container with ID starting with ad5251608a3f1bb102f143be62b37a74a7d717543190a34039207ec335cf92a9 not found: ID does not exist" containerID="ad5251608a3f1bb102f143be62b37a74a7d717543190a34039207ec335cf92a9" Oct 30 17:13:39.515622 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:39.515549 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad5251608a3f1bb102f143be62b37a74a7d717543190a34039207ec335cf92a9"} err="failed to get container status \"ad5251608a3f1bb102f143be62b37a74a7d717543190a34039207ec335cf92a9\": rpc error: code = NotFound desc = could not find container \"ad5251608a3f1bb102f143be62b37a74a7d717543190a34039207ec335cf92a9\": container with ID starting with ad5251608a3f1bb102f143be62b37a74a7d717543190a34039207ec335cf92a9 not found: ID does not exist" Oct 30 17:13:39.536099 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:39.536073 5052 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f6b9354-378f-4451-9ff7-be7f7fec12f2-dshm\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:13:39.536217 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:39.536103 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5g7b2\" (UniqueName: \"kubernetes.io/projected/8f6b9354-378f-4451-9ff7-be7f7fec12f2-kube-api-access-5g7b2\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:13:39.536217 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:39.536118 5052 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f6b9354-378f-4451-9ff7-be7f7fec12f2-tls-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:13:39.552017 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:39.551994 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-967fac/distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx"] Oct 30 17:13:39.555927 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:39.555902 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-967fac/distinct-2-w5-967fac-kserve-6cd5476b49-dxjkx"] Oct 30 17:13:39.593960 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:39.593930 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-967fac/distinct-w5-967fac-kserve-69cbcccd8b-b85bn"] Oct 30 17:13:39.599913 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:39.599638 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-967fac/distinct-w5-967fac-kserve-69cbcccd8b-b85bn"] Oct 30 17:13:40.602015 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:40.601978 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f6b9354-378f-4451-9ff7-be7f7fec12f2" path="/var/lib/kubelet/pods/8f6b9354-378f-4451-9ff7-be7f7fec12f2/volumes" Oct 30 17:13:40.602511 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:40.602495 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd662f3a-659f-477c-b8d5-94c64288c0c1" path="/var/lib/kubelet/pods/bd662f3a-659f-477c-b8d5-94c64288c0c1/volumes" Oct 30 17:13:42.425793 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:42.425713 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-4def7e-6d5bf4d5c8-62jp6" Oct 30 17:13:48.841682 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:13:48.841648 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 17:13:55.879157 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:55.879099 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-4frq9"] Oct 30 17:13:55.879797 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:55.879367 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-4frq9" podUID="8451a193-e5be-47eb-9e5b-75c5214d3abc" containerName="istio-proxy" containerID="cri-o://a5603fd958409a6b510ee54e6488ae161277b39817552e58289ff5907c5d62b0" gracePeriod=30 Oct 30 17:13:55.983343 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:55.983305 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-lhnp2"] Oct 30 17:13:55.983539 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:55.983515 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-lhnp2" podUID="523c6f22-3610-4d8f-a048-acb661bb23b5" containerName="istio-proxy" containerID="cri-o://84641a29cb8d4fbba4808b2e6538c65d21414b19dcdf929c0de0923de41a3b57" gracePeriod=30 Oct 30 17:13:56.932209 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:56.931432 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-330282-openshift-default-5784687855-s68x6"] Oct 30 17:13:56.932209 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:56.931707 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-330282-openshift-default-5784687855-s68x6" podUID="ce11f38f-0ae0-4881-aa0f-346a0dd513c4" containerName="istio-proxy" containerID="cri-o://b09cf2b2468112f122ac51f5ee4d6dc72dbe40920fc903d4c8c2a3b30abb5c8b" gracePeriod=30 Oct 30 17:13:59.902270 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:59.902229 5052 patch_prober.go:29] interesting pod/e2e-worker-w4-263f00-openshift-default-b969d55bc-4frq9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 17:13:59.902678 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:13:59.902301 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-4frq9" podUID="8451a193-e5be-47eb-9e5b-75c5214d3abc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 17:14:00.420680 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:00.420643 5052 patch_prober.go:29] interesting pod/e2e-ipp-a-330282-openshift-default-5784687855-s68x6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 17:14:00.420870 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:00.420705 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-330282-openshift-default-5784687855-s68x6" podUID="ce11f38f-0ae0-4881-aa0f-346a0dd513c4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 17:14:01.184551 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:01.184481 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-4frq9" Oct 30 17:14:01.298318 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:01.298288 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-lhnp2" Oct 30 17:14:01.326773 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:01.326736 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8451a193-e5be-47eb-9e5b-75c5214d3abc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8451a193-e5be-47eb-9e5b-75c5214d3abc-istio-podinfo\") pod \"8451a193-e5be-47eb-9e5b-75c5214d3abc\" (UID: \"8451a193-e5be-47eb-9e5b-75c5214d3abc\") " Oct 30 17:14:01.326954 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:01.326858 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8451a193-e5be-47eb-9e5b-75c5214d3abc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8451a193-e5be-47eb-9e5b-75c5214d3abc-istio-data\") pod \"8451a193-e5be-47eb-9e5b-75c5214d3abc\" (UID: \"8451a193-e5be-47eb-9e5b-75c5214d3abc\") " Oct 30 17:14:01.326954 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:01.326933 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8451a193-e5be-47eb-9e5b-75c5214d3abc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8451a193-e5be-47eb-9e5b-75c5214d3abc-workload-socket\") pod \"8451a193-e5be-47eb-9e5b-75c5214d3abc\" (UID: \"8451a193-e5be-47eb-9e5b-75c5214d3abc\") " Oct 30 17:14:01.326954 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:01.326952 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8451a193-e5be-47eb-9e5b-75c5214d3abc-kube-api-access-vlrrt\" (UniqueName: \"kubernetes.io/projected/8451a193-e5be-47eb-9e5b-75c5214d3abc-kube-api-access-vlrrt\") pod \"8451a193-e5be-47eb-9e5b-75c5214d3abc\" (UID: \"8451a193-e5be-47eb-9e5b-75c5214d3abc\") " Oct 30 17:14:01.327196 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:01.326991 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8451a193-e5be-47eb-9e5b-75c5214d3abc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8451a193-e5be-47eb-9e5b-75c5214d3abc-istiod-ca-cert\") pod \"8451a193-e5be-47eb-9e5b-75c5214d3abc\" (UID: \"8451a193-e5be-47eb-9e5b-75c5214d3abc\") " Oct 30 17:14:01.327196 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:01.327014 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8451a193-e5be-47eb-9e5b-75c5214d3abc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8451a193-e5be-47eb-9e5b-75c5214d3abc-workload-certs\") pod \"8451a193-e5be-47eb-9e5b-75c5214d3abc\" (UID: \"8451a193-e5be-47eb-9e5b-75c5214d3abc\") " Oct 30 17:14:01.327196 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:01.327158 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8451a193-e5be-47eb-9e5b-75c5214d3abc-istio-token\" (UniqueName: \"kubernetes.io/projected/8451a193-e5be-47eb-9e5b-75c5214d3abc-istio-token\") pod \"8451a193-e5be-47eb-9e5b-75c5214d3abc\" (UID: \"8451a193-e5be-47eb-9e5b-75c5214d3abc\") " Oct 30 17:14:01.327196 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:01.327187 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8451a193-e5be-47eb-9e5b-75c5214d3abc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8451a193-e5be-47eb-9e5b-75c5214d3abc-istio-envoy\") pod \"8451a193-e5be-47eb-9e5b-75c5214d3abc\" (UID: \"8451a193-e5be-47eb-9e5b-75c5214d3abc\") " Oct 30 17:14:01.327419 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:01.327176 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8451a193-e5be-47eb-9e5b-75c5214d3abc-istio-data" pod "8451a193-e5be-47eb-9e5b-75c5214d3abc" (UID: "8451a193-e5be-47eb-9e5b-75c5214d3abc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:14:01.327419 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:01.327212 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8451a193-e5be-47eb-9e5b-75c5214d3abc-workload-socket" pod "8451a193-e5be-47eb-9e5b-75c5214d3abc" (UID: "8451a193-e5be-47eb-9e5b-75c5214d3abc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:14:01.327419 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:01.327263 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8451a193-e5be-47eb-9e5b-75c5214d3abc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8451a193-e5be-47eb-9e5b-75c5214d3abc-credential-socket\") pod \"8451a193-e5be-47eb-9e5b-75c5214d3abc\" (UID: \"8451a193-e5be-47eb-9e5b-75c5214d3abc\") " Oct 30 17:14:01.327419 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:01.327386 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8451a193-e5be-47eb-9e5b-75c5214d3abc-workload-certs" pod "8451a193-e5be-47eb-9e5b-75c5214d3abc" (UID: "8451a193-e5be-47eb-9e5b-75c5214d3abc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:14:01.327701 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:01.327416 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8451a193-e5be-47eb-9e5b-75c5214d3abc-istiod-ca-cert" pod "8451a193-e5be-47eb-9e5b-75c5214d3abc" (UID: "8451a193-e5be-47eb-9e5b-75c5214d3abc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:14:01.327701 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:01.327650 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8451a193-e5be-47eb-9e5b-75c5214d3abc-credential-socket" pod "8451a193-e5be-47eb-9e5b-75c5214d3abc" (UID: "8451a193-e5be-47eb-9e5b-75c5214d3abc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:14:01.328045 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:01.328021 5052 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8451a193-e5be-47eb-9e5b-75c5214d3abc-workload-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:14:01.328146 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:01.328048 5052 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8451a193-e5be-47eb-9e5b-75c5214d3abc-istiod-ca-cert\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:14:01.328146 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:01.328063 5052 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8451a193-e5be-47eb-9e5b-75c5214d3abc-workload-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:14:01.328146 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:01.328079 5052 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8451a193-e5be-47eb-9e5b-75c5214d3abc-credential-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:14:01.328146 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:01.328111 5052 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8451a193-e5be-47eb-9e5b-75c5214d3abc-istio-data\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:14:01.329174 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:01.329148 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8451a193-e5be-47eb-9e5b-75c5214d3abc-istio-podinfo" pod "8451a193-e5be-47eb-9e5b-75c5214d3abc" (UID: "8451a193-e5be-47eb-9e5b-75c5214d3abc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:14:01.329630 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:01.329599 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8451a193-e5be-47eb-9e5b-75c5214d3abc-kube-api-access-vlrrt" pod "8451a193-e5be-47eb-9e5b-75c5214d3abc" (UID: "8451a193-e5be-47eb-9e5b-75c5214d3abc"). InnerVolumeSpecName "kube-api-access-vlrrt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:14:01.329742 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:01.329719 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8451a193-e5be-47eb-9e5b-75c5214d3abc-istio-token" pod "8451a193-e5be-47eb-9e5b-75c5214d3abc" (UID: "8451a193-e5be-47eb-9e5b-75c5214d3abc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:14:01.330255 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:01.330230 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8451a193-e5be-47eb-9e5b-75c5214d3abc-istio-envoy" pod "8451a193-e5be-47eb-9e5b-75c5214d3abc" (UID: "8451a193-e5be-47eb-9e5b-75c5214d3abc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:14:01.428694 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:01.428652 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/523c6f22-3610-4d8f-a048-acb661bb23b5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/523c6f22-3610-4d8f-a048-acb661bb23b5-workload-socket\") pod \"523c6f22-3610-4d8f-a048-acb661bb23b5\" (UID: \"523c6f22-3610-4d8f-a048-acb661bb23b5\") " Oct 30 17:14:01.430104 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:01.428704 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/523c6f22-3610-4d8f-a048-acb661bb23b5-istio-token\" (UniqueName: \"kubernetes.io/projected/523c6f22-3610-4d8f-a048-acb661bb23b5-istio-token\") pod \"523c6f22-3610-4d8f-a048-acb661bb23b5\" (UID: \"523c6f22-3610-4d8f-a048-acb661bb23b5\") " Oct 30 17:14:01.430104 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:01.428769 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/523c6f22-3610-4d8f-a048-acb661bb23b5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/523c6f22-3610-4d8f-a048-acb661bb23b5-istio-envoy\") pod \"523c6f22-3610-4d8f-a048-acb661bb23b5\" (UID: \"523c6f22-3610-4d8f-a048-acb661bb23b5\") " Oct 30 17:14:01.430104 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:01.428869 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/523c6f22-3610-4d8f-a048-acb661bb23b5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/523c6f22-3610-4d8f-a048-acb661bb23b5-credential-socket\") pod \"523c6f22-3610-4d8f-a048-acb661bb23b5\" (UID: \"523c6f22-3610-4d8f-a048-acb661bb23b5\") " Oct 30 17:14:01.430104 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:01.428959 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/523c6f22-3610-4d8f-a048-acb661bb23b5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/523c6f22-3610-4d8f-a048-acb661bb23b5-istiod-ca-cert\") pod \"523c6f22-3610-4d8f-a048-acb661bb23b5\" (UID: \"523c6f22-3610-4d8f-a048-acb661bb23b5\") " Oct 30 17:14:01.430104 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:01.429095 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/523c6f22-3610-4d8f-a048-acb661bb23b5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/523c6f22-3610-4d8f-a048-acb661bb23b5-workload-certs\") pod \"523c6f22-3610-4d8f-a048-acb661bb23b5\" (UID: \"523c6f22-3610-4d8f-a048-acb661bb23b5\") " Oct 30 17:14:01.430104 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:01.429165 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/523c6f22-3610-4d8f-a048-acb661bb23b5-kube-api-access-7mbcw\" (UniqueName: \"kubernetes.io/projected/523c6f22-3610-4d8f-a048-acb661bb23b5-kube-api-access-7mbcw\") pod \"523c6f22-3610-4d8f-a048-acb661bb23b5\" (UID: \"523c6f22-3610-4d8f-a048-acb661bb23b5\") " Oct 30 17:14:01.430104 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:01.429252 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/523c6f22-3610-4d8f-a048-acb661bb23b5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/523c6f22-3610-4d8f-a048-acb661bb23b5-istio-podinfo\") pod \"523c6f22-3610-4d8f-a048-acb661bb23b5\" (UID: \"523c6f22-3610-4d8f-a048-acb661bb23b5\") " Oct 30 17:14:01.430104 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:01.429365 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/523c6f22-3610-4d8f-a048-acb661bb23b5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/523c6f22-3610-4d8f-a048-acb661bb23b5-istio-data\") pod \"523c6f22-3610-4d8f-a048-acb661bb23b5\" (UID: \"523c6f22-3610-4d8f-a048-acb661bb23b5\") " Oct 30 17:14:01.430104 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:01.429629 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/523c6f22-3610-4d8f-a048-acb661bb23b5-istiod-ca-cert" pod "523c6f22-3610-4d8f-a048-acb661bb23b5" (UID: "523c6f22-3610-4d8f-a048-acb661bb23b5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:14:01.430104 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:01.429668 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/523c6f22-3610-4d8f-a048-acb661bb23b5-workload-certs" pod "523c6f22-3610-4d8f-a048-acb661bb23b5" (UID: "523c6f22-3610-4d8f-a048-acb661bb23b5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:14:01.430104 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:01.429824 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/523c6f22-3610-4d8f-a048-acb661bb23b5-workload-socket" pod "523c6f22-3610-4d8f-a048-acb661bb23b5" (UID: "523c6f22-3610-4d8f-a048-acb661bb23b5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:14:01.430104 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:01.429920 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/523c6f22-3610-4d8f-a048-acb661bb23b5-credential-socket" pod "523c6f22-3610-4d8f-a048-acb661bb23b5" (UID: "523c6f22-3610-4d8f-a048-acb661bb23b5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:14:01.430628 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:01.430124 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/523c6f22-3610-4d8f-a048-acb661bb23b5-istio-data" pod "523c6f22-3610-4d8f-a048-acb661bb23b5" (UID: "523c6f22-3610-4d8f-a048-acb661bb23b5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:14:01.430628 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:01.430301 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlrrt\" (UniqueName: \"kubernetes.io/projected/8451a193-e5be-47eb-9e5b-75c5214d3abc-kube-api-access-vlrrt\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:14:01.430628 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:01.430322 5052 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/523c6f22-3610-4d8f-a048-acb661bb23b5-istio-data\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:14:01.430628 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:01.430336 5052 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/523c6f22-3610-4d8f-a048-acb661bb23b5-workload-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:14:01.430628 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:01.430350 5052 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8451a193-e5be-47eb-9e5b-75c5214d3abc-istio-token\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:14:01.430628 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:01.430364 5052 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8451a193-e5be-47eb-9e5b-75c5214d3abc-istio-envoy\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:14:01.430628 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:01.430377 5052 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8451a193-e5be-47eb-9e5b-75c5214d3abc-istio-podinfo\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:14:01.430628 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:01.430391 5052 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/523c6f22-3610-4d8f-a048-acb661bb23b5-credential-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:14:01.430628 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:01.430405 5052 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/523c6f22-3610-4d8f-a048-acb661bb23b5-istiod-ca-cert\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:14:01.430628 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:01.430420 5052 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/523c6f22-3610-4d8f-a048-acb661bb23b5-workload-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:14:01.432188 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:01.432163 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/523c6f22-3610-4d8f-a048-acb661bb23b5-istio-podinfo" pod "523c6f22-3610-4d8f-a048-acb661bb23b5" (UID: "523c6f22-3610-4d8f-a048-acb661bb23b5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:14:01.432315 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:01.432223 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/523c6f22-3610-4d8f-a048-acb661bb23b5-istio-token" pod "523c6f22-3610-4d8f-a048-acb661bb23b5" (UID: "523c6f22-3610-4d8f-a048-acb661bb23b5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:14:01.432315 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:01.432259 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/523c6f22-3610-4d8f-a048-acb661bb23b5-kube-api-access-7mbcw" pod "523c6f22-3610-4d8f-a048-acb661bb23b5" (UID: "523c6f22-3610-4d8f-a048-acb661bb23b5"). InnerVolumeSpecName "kube-api-access-7mbcw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:14:01.432402 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:01.432391 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/523c6f22-3610-4d8f-a048-acb661bb23b5-istio-envoy" pod "523c6f22-3610-4d8f-a048-acb661bb23b5" (UID: "523c6f22-3610-4d8f-a048-acb661bb23b5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:14:01.532122 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:01.532026 5052 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/523c6f22-3610-4d8f-a048-acb661bb23b5-istio-podinfo\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:14:01.532122 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:01.532062 5052 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/523c6f22-3610-4d8f-a048-acb661bb23b5-istio-token\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:14:01.532122 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:01.532076 5052 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/523c6f22-3610-4d8f-a048-acb661bb23b5-istio-envoy\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:14:01.532122 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:01.532089 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mbcw\" (UniqueName: \"kubernetes.io/projected/523c6f22-3610-4d8f-a048-acb661bb23b5-kube-api-access-7mbcw\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:14:01.801710 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:01.801619 5052 generic.go:356] "Generic (PLEG): container finished" podID="8451a193-e5be-47eb-9e5b-75c5214d3abc" containerID="a5603fd958409a6b510ee54e6488ae161277b39817552e58289ff5907c5d62b0" exitCode=0 Oct 30 17:14:01.801878 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:01.801728 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-4frq9" Oct 30 17:14:01.801878 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:01.801744 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-4frq9" event={"ID":"8451a193-e5be-47eb-9e5b-75c5214d3abc","Type":"ContainerDied","Data":"a5603fd958409a6b510ee54e6488ae161277b39817552e58289ff5907c5d62b0"} Oct 30 17:14:01.801878 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:01.801808 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-4frq9" event={"ID":"8451a193-e5be-47eb-9e5b-75c5214d3abc","Type":"ContainerDied","Data":"1cf17567e27e0368e714f1934f7236aefe195f2718488e24e6196d744b55b2b7"} Oct 30 17:14:01.801878 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:01.801835 5052 scope.go:122] "RemoveContainer" containerID="a5603fd958409a6b510ee54e6488ae161277b39817552e58289ff5907c5d62b0" Oct 30 17:14:01.803414 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:01.803386 5052 generic.go:356] "Generic (PLEG): container finished" podID="523c6f22-3610-4d8f-a048-acb661bb23b5" containerID="84641a29cb8d4fbba4808b2e6538c65d21414b19dcdf929c0de0923de41a3b57" exitCode=0 Oct 30 17:14:01.803534 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:01.803428 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-lhnp2" Oct 30 17:14:01.803534 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:01.803455 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-lhnp2" event={"ID":"523c6f22-3610-4d8f-a048-acb661bb23b5","Type":"ContainerDied","Data":"84641a29cb8d4fbba4808b2e6538c65d21414b19dcdf929c0de0923de41a3b57"} Oct 30 17:14:01.803534 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:01.803498 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-lhnp2" event={"ID":"523c6f22-3610-4d8f-a048-acb661bb23b5","Type":"ContainerDied","Data":"ec959e07938e0cf97fd08ff8aa55404910a2e49ccc71acac6ca8fd3e40cfea07"} Oct 30 17:14:01.816839 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:01.816635 5052 scope.go:122] "RemoveContainer" containerID="a5603fd958409a6b510ee54e6488ae161277b39817552e58289ff5907c5d62b0" Oct 30 17:14:01.817068 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:14:01.817032 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5603fd958409a6b510ee54e6488ae161277b39817552e58289ff5907c5d62b0\": container with ID starting with a5603fd958409a6b510ee54e6488ae161277b39817552e58289ff5907c5d62b0 not found: ID does not exist" containerID="a5603fd958409a6b510ee54e6488ae161277b39817552e58289ff5907c5d62b0" Oct 30 17:14:01.817163 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:01.817078 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5603fd958409a6b510ee54e6488ae161277b39817552e58289ff5907c5d62b0"} err="failed to get container status \"a5603fd958409a6b510ee54e6488ae161277b39817552e58289ff5907c5d62b0\": rpc error: code = NotFound desc = could not find container \"a5603fd958409a6b510ee54e6488ae161277b39817552e58289ff5907c5d62b0\": container with ID starting with a5603fd958409a6b510ee54e6488ae161277b39817552e58289ff5907c5d62b0 not found: ID does not exist" Oct 30 17:14:01.817163 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:01.817107 5052 scope.go:122] "RemoveContainer" containerID="84641a29cb8d4fbba4808b2e6538c65d21414b19dcdf929c0de0923de41a3b57" Oct 30 17:14:01.830670 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:01.830621 5052 scope.go:122] "RemoveContainer" containerID="84641a29cb8d4fbba4808b2e6538c65d21414b19dcdf929c0de0923de41a3b57" Oct 30 17:14:01.831006 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:14:01.830977 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84641a29cb8d4fbba4808b2e6538c65d21414b19dcdf929c0de0923de41a3b57\": container with ID starting with 84641a29cb8d4fbba4808b2e6538c65d21414b19dcdf929c0de0923de41a3b57 not found: ID does not exist" containerID="84641a29cb8d4fbba4808b2e6538c65d21414b19dcdf929c0de0923de41a3b57" Oct 30 17:14:01.831109 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:01.831009 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84641a29cb8d4fbba4808b2e6538c65d21414b19dcdf929c0de0923de41a3b57"} err="failed to get container status \"84641a29cb8d4fbba4808b2e6538c65d21414b19dcdf929c0de0923de41a3b57\": rpc error: code = NotFound desc = could not find container \"84641a29cb8d4fbba4808b2e6538c65d21414b19dcdf929c0de0923de41a3b57\": container with ID starting with 84641a29cb8d4fbba4808b2e6538c65d21414b19dcdf929c0de0923de41a3b57 not found: ID does not exist" Oct 30 17:14:01.872801 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:01.872684 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-lhnp2"] Oct 30 17:14:01.886758 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:01.886732 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-lhnp2"] Oct 30 17:14:01.913642 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:01.913606 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-4frq9"] Oct 30 17:14:01.921326 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:01.921300 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-4frq9"] Oct 30 17:14:02.468460 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:02.468431 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-330282-openshift-default-5784687855-s68x6" Oct 30 17:14:02.673878 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:02.673841 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce11f38f-0ae0-4881-aa0f-346a0dd513c4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce11f38f-0ae0-4881-aa0f-346a0dd513c4-credential-socket\") pod \"ce11f38f-0ae0-4881-aa0f-346a0dd513c4\" (UID: \"ce11f38f-0ae0-4881-aa0f-346a0dd513c4\") " Oct 30 17:14:02.675974 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:02.673977 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce11f38f-0ae0-4881-aa0f-346a0dd513c4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce11f38f-0ae0-4881-aa0f-346a0dd513c4-workload-socket\") pod \"ce11f38f-0ae0-4881-aa0f-346a0dd513c4\" (UID: \"ce11f38f-0ae0-4881-aa0f-346a0dd513c4\") " Oct 30 17:14:02.675974 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:02.674062 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce11f38f-0ae0-4881-aa0f-346a0dd513c4-kube-api-access-7v2b9\" (UniqueName: \"kubernetes.io/projected/ce11f38f-0ae0-4881-aa0f-346a0dd513c4-kube-api-access-7v2b9\") pod \"ce11f38f-0ae0-4881-aa0f-346a0dd513c4\" (UID: \"ce11f38f-0ae0-4881-aa0f-346a0dd513c4\") " Oct 30 17:14:02.675974 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:02.674122 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce11f38f-0ae0-4881-aa0f-346a0dd513c4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce11f38f-0ae0-4881-aa0f-346a0dd513c4-istio-data\") pod \"ce11f38f-0ae0-4881-aa0f-346a0dd513c4\" (UID: \"ce11f38f-0ae0-4881-aa0f-346a0dd513c4\") " Oct 30 17:14:02.675974 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:02.674156 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce11f38f-0ae0-4881-aa0f-346a0dd513c4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce11f38f-0ae0-4881-aa0f-346a0dd513c4-workload-certs\") pod \"ce11f38f-0ae0-4881-aa0f-346a0dd513c4\" (UID: \"ce11f38f-0ae0-4881-aa0f-346a0dd513c4\") " Oct 30 17:14:02.675974 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:02.674257 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce11f38f-0ae0-4881-aa0f-346a0dd513c4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce11f38f-0ae0-4881-aa0f-346a0dd513c4-istio-envoy\") pod \"ce11f38f-0ae0-4881-aa0f-346a0dd513c4\" (UID: \"ce11f38f-0ae0-4881-aa0f-346a0dd513c4\") " Oct 30 17:14:02.675974 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:02.674291 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ce11f38f-0ae0-4881-aa0f-346a0dd513c4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce11f38f-0ae0-4881-aa0f-346a0dd513c4-istio-podinfo\") pod \"ce11f38f-0ae0-4881-aa0f-346a0dd513c4\" (UID: \"ce11f38f-0ae0-4881-aa0f-346a0dd513c4\") " Oct 30 17:14:02.675974 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:02.674389 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce11f38f-0ae0-4881-aa0f-346a0dd513c4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce11f38f-0ae0-4881-aa0f-346a0dd513c4-istiod-ca-cert\") pod \"ce11f38f-0ae0-4881-aa0f-346a0dd513c4\" (UID: \"ce11f38f-0ae0-4881-aa0f-346a0dd513c4\") " Oct 30 17:14:02.675974 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:02.674435 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce11f38f-0ae0-4881-aa0f-346a0dd513c4-istio-token\" (UniqueName: \"kubernetes.io/projected/ce11f38f-0ae0-4881-aa0f-346a0dd513c4-istio-token\") pod \"ce11f38f-0ae0-4881-aa0f-346a0dd513c4\" (UID: \"ce11f38f-0ae0-4881-aa0f-346a0dd513c4\") " Oct 30 17:14:02.676492 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:02.676271 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce11f38f-0ae0-4881-aa0f-346a0dd513c4-credential-socket" pod "ce11f38f-0ae0-4881-aa0f-346a0dd513c4" (UID: "ce11f38f-0ae0-4881-aa0f-346a0dd513c4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:14:02.676616 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:02.676549 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce11f38f-0ae0-4881-aa0f-346a0dd513c4-workload-socket" pod "ce11f38f-0ae0-4881-aa0f-346a0dd513c4" (UID: "ce11f38f-0ae0-4881-aa0f-346a0dd513c4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:14:02.677658 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:02.677317 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce11f38f-0ae0-4881-aa0f-346a0dd513c4-istio-data" pod "ce11f38f-0ae0-4881-aa0f-346a0dd513c4" (UID: "ce11f38f-0ae0-4881-aa0f-346a0dd513c4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:14:02.678268 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:02.678218 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce11f38f-0ae0-4881-aa0f-346a0dd513c4-workload-certs" pod "ce11f38f-0ae0-4881-aa0f-346a0dd513c4" (UID: "ce11f38f-0ae0-4881-aa0f-346a0dd513c4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:14:02.680480 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:02.678973 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce11f38f-0ae0-4881-aa0f-346a0dd513c4-istiod-ca-cert" pod "ce11f38f-0ae0-4881-aa0f-346a0dd513c4" (UID: "ce11f38f-0ae0-4881-aa0f-346a0dd513c4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:14:02.680480 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:02.680011 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce11f38f-0ae0-4881-aa0f-346a0dd513c4-kube-api-access-7v2b9" pod "ce11f38f-0ae0-4881-aa0f-346a0dd513c4" (UID: "ce11f38f-0ae0-4881-aa0f-346a0dd513c4"). InnerVolumeSpecName "kube-api-access-7v2b9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:14:02.680967 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:02.680890 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce11f38f-0ae0-4881-aa0f-346a0dd513c4-istio-envoy" pod "ce11f38f-0ae0-4881-aa0f-346a0dd513c4" (UID: "ce11f38f-0ae0-4881-aa0f-346a0dd513c4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:14:02.681383 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:02.681349 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ce11f38f-0ae0-4881-aa0f-346a0dd513c4-istio-podinfo" pod "ce11f38f-0ae0-4881-aa0f-346a0dd513c4" (UID: "ce11f38f-0ae0-4881-aa0f-346a0dd513c4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:14:02.684918 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:02.684898 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce11f38f-0ae0-4881-aa0f-346a0dd513c4-istio-token" pod "ce11f38f-0ae0-4881-aa0f-346a0dd513c4" (UID: "ce11f38f-0ae0-4881-aa0f-346a0dd513c4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:14:02.686301 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:02.685977 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="523c6f22-3610-4d8f-a048-acb661bb23b5" path="/var/lib/kubelet/pods/523c6f22-3610-4d8f-a048-acb661bb23b5/volumes" Oct 30 17:14:02.687089 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:02.686489 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8451a193-e5be-47eb-9e5b-75c5214d3abc" path="/var/lib/kubelet/pods/8451a193-e5be-47eb-9e5b-75c5214d3abc/volumes" Oct 30 17:14:02.776370 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:02.776342 5052 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce11f38f-0ae0-4881-aa0f-346a0dd513c4-istio-envoy\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:14:02.776370 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:02.776370 5052 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce11f38f-0ae0-4881-aa0f-346a0dd513c4-istio-podinfo\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:14:02.776370 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:02.776379 5052 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce11f38f-0ae0-4881-aa0f-346a0dd513c4-istiod-ca-cert\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:14:02.776657 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:02.776388 5052 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce11f38f-0ae0-4881-aa0f-346a0dd513c4-istio-token\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:14:02.776657 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:02.776396 5052 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce11f38f-0ae0-4881-aa0f-346a0dd513c4-credential-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:14:02.776657 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:02.776405 5052 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce11f38f-0ae0-4881-aa0f-346a0dd513c4-workload-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:14:02.776657 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:02.776415 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7v2b9\" (UniqueName: \"kubernetes.io/projected/ce11f38f-0ae0-4881-aa0f-346a0dd513c4-kube-api-access-7v2b9\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:14:02.776657 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:02.776425 5052 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce11f38f-0ae0-4881-aa0f-346a0dd513c4-istio-data\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:14:02.776657 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:02.776437 5052 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce11f38f-0ae0-4881-aa0f-346a0dd513c4-workload-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:14:02.814113 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:02.814074 5052 generic.go:356] "Generic (PLEG): container finished" podID="ce11f38f-0ae0-4881-aa0f-346a0dd513c4" containerID="b09cf2b2468112f122ac51f5ee4d6dc72dbe40920fc903d4c8c2a3b30abb5c8b" exitCode=0 Oct 30 17:14:02.814308 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:02.814122 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-330282-openshift-default-5784687855-s68x6" Oct 30 17:14:02.814308 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:02.814150 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-330282-openshift-default-5784687855-s68x6" event={"ID":"ce11f38f-0ae0-4881-aa0f-346a0dd513c4","Type":"ContainerDied","Data":"b09cf2b2468112f122ac51f5ee4d6dc72dbe40920fc903d4c8c2a3b30abb5c8b"} Oct 30 17:14:02.814308 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:02.814213 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-330282-openshift-default-5784687855-s68x6" event={"ID":"ce11f38f-0ae0-4881-aa0f-346a0dd513c4","Type":"ContainerDied","Data":"2deda4d44343e1cf5c40a5775bacd0137dec6c3757a86c32285abb8f562cbb36"} Oct 30 17:14:02.814308 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:02.814237 5052 scope.go:122] "RemoveContainer" containerID="b09cf2b2468112f122ac51f5ee4d6dc72dbe40920fc903d4c8c2a3b30abb5c8b" Oct 30 17:14:02.827505 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:02.827485 5052 scope.go:122] "RemoveContainer" containerID="2e348d887319104ed3f97de52fb4effecebdb81eab193bcd2a22ac7b0b96a325" Oct 30 17:14:02.837604 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:02.837560 5052 scope.go:122] "RemoveContainer" containerID="b09cf2b2468112f122ac51f5ee4d6dc72dbe40920fc903d4c8c2a3b30abb5c8b" Oct 30 17:14:02.837857 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:14:02.837834 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b09cf2b2468112f122ac51f5ee4d6dc72dbe40920fc903d4c8c2a3b30abb5c8b\": container with ID starting with b09cf2b2468112f122ac51f5ee4d6dc72dbe40920fc903d4c8c2a3b30abb5c8b not found: ID does not exist" containerID="b09cf2b2468112f122ac51f5ee4d6dc72dbe40920fc903d4c8c2a3b30abb5c8b" Oct 30 17:14:02.837973 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:02.837862 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b09cf2b2468112f122ac51f5ee4d6dc72dbe40920fc903d4c8c2a3b30abb5c8b"} err="failed to get container status \"b09cf2b2468112f122ac51f5ee4d6dc72dbe40920fc903d4c8c2a3b30abb5c8b\": rpc error: code = NotFound desc = could not find container \"b09cf2b2468112f122ac51f5ee4d6dc72dbe40920fc903d4c8c2a3b30abb5c8b\": container with ID starting with b09cf2b2468112f122ac51f5ee4d6dc72dbe40920fc903d4c8c2a3b30abb5c8b not found: ID does not exist" Oct 30 17:14:02.837973 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:02.837879 5052 scope.go:122] "RemoveContainer" containerID="2e348d887319104ed3f97de52fb4effecebdb81eab193bcd2a22ac7b0b96a325" Oct 30 17:14:02.838114 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:14:02.838097 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e348d887319104ed3f97de52fb4effecebdb81eab193bcd2a22ac7b0b96a325\": container with ID starting with 2e348d887319104ed3f97de52fb4effecebdb81eab193bcd2a22ac7b0b96a325 not found: ID does not exist" containerID="2e348d887319104ed3f97de52fb4effecebdb81eab193bcd2a22ac7b0b96a325" Oct 30 17:14:02.838164 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:02.838119 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e348d887319104ed3f97de52fb4effecebdb81eab193bcd2a22ac7b0b96a325"} err="failed to get container status \"2e348d887319104ed3f97de52fb4effecebdb81eab193bcd2a22ac7b0b96a325\": rpc error: code = NotFound desc = could not find container \"2e348d887319104ed3f97de52fb4effecebdb81eab193bcd2a22ac7b0b96a325\": container with ID starting with 2e348d887319104ed3f97de52fb4effecebdb81eab193bcd2a22ac7b0b96a325 not found: ID does not exist" Oct 30 17:14:02.876878 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:02.876851 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-330282-openshift-default-5784687855-s68x6"] Oct 30 17:14:02.890115 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:02.890086 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-330282-openshift-default-5784687855-s68x6"] Oct 30 17:14:04.600394 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:04.600361 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce11f38f-0ae0-4881-aa0f-346a0dd513c4" path="/var/lib/kubelet/pods/ce11f38f-0ae0-4881-aa0f-346a0dd513c4/volumes" Oct 30 17:14:07.482201 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:07.482088 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cdwcj"] Oct 30 17:14:13.173042 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:13.169639 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-6jzqj"] Oct 30 17:14:13.173042 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:13.169894 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-6jzqj" podUID="1fb4a801-fd82-4036-ba89-79fdf32d3be3" containerName="istio-proxy" containerID="cri-o://b9e3fbca33f82af327ba9d2612b8bfb6b0763849081e591fa80f05e22433665a" gracePeriod=30 Oct 30 17:14:13.185657 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:13.185084 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-kgzxp"] Oct 30 17:14:13.185657 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:13.185307 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-kgzxp" podUID="025e9d83-1372-496c-b75e-4f5205d48dcd" containerName="istio-proxy" containerID="cri-o://538de255129a6d08fdad51382b6432e2f8d70e4102e68fef4706550d0780ae26" gracePeriod=30 Oct 30 17:14:18.528254 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:18.528223 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-kgzxp" Oct 30 17:14:18.582338 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:18.582307 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/025e9d83-1372-496c-b75e-4f5205d48dcd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/025e9d83-1372-496c-b75e-4f5205d48dcd-workload-certs\") pod \"025e9d83-1372-496c-b75e-4f5205d48dcd\" (UID: \"025e9d83-1372-496c-b75e-4f5205d48dcd\") " Oct 30 17:14:18.582513 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:18.582425 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/025e9d83-1372-496c-b75e-4f5205d48dcd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/025e9d83-1372-496c-b75e-4f5205d48dcd-istio-podinfo\") pod \"025e9d83-1372-496c-b75e-4f5205d48dcd\" (UID: \"025e9d83-1372-496c-b75e-4f5205d48dcd\") " Oct 30 17:14:18.582513 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:18.582476 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/025e9d83-1372-496c-b75e-4f5205d48dcd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/025e9d83-1372-496c-b75e-4f5205d48dcd-workload-socket\") pod \"025e9d83-1372-496c-b75e-4f5205d48dcd\" (UID: \"025e9d83-1372-496c-b75e-4f5205d48dcd\") " Oct 30 17:14:18.582757 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:18.582601 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/025e9d83-1372-496c-b75e-4f5205d48dcd-workload-certs" pod "025e9d83-1372-496c-b75e-4f5205d48dcd" (UID: "025e9d83-1372-496c-b75e-4f5205d48dcd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:14:18.582757 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:18.582631 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/025e9d83-1372-496c-b75e-4f5205d48dcd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/025e9d83-1372-496c-b75e-4f5205d48dcd-credential-socket\") pod \"025e9d83-1372-496c-b75e-4f5205d48dcd\" (UID: \"025e9d83-1372-496c-b75e-4f5205d48dcd\") " Oct 30 17:14:18.582757 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:18.582744 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/025e9d83-1372-496c-b75e-4f5205d48dcd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/025e9d83-1372-496c-b75e-4f5205d48dcd-istio-envoy\") pod \"025e9d83-1372-496c-b75e-4f5205d48dcd\" (UID: \"025e9d83-1372-496c-b75e-4f5205d48dcd\") " Oct 30 17:14:18.582935 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:18.582757 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/025e9d83-1372-496c-b75e-4f5205d48dcd-workload-socket" pod "025e9d83-1372-496c-b75e-4f5205d48dcd" (UID: "025e9d83-1372-496c-b75e-4f5205d48dcd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:14:18.582935 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:18.582819 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/025e9d83-1372-496c-b75e-4f5205d48dcd-kube-api-access-72vht\" (UniqueName: \"kubernetes.io/projected/025e9d83-1372-496c-b75e-4f5205d48dcd-kube-api-access-72vht\") pod \"025e9d83-1372-496c-b75e-4f5205d48dcd\" (UID: \"025e9d83-1372-496c-b75e-4f5205d48dcd\") " Oct 30 17:14:18.582935 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:18.582884 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/025e9d83-1372-496c-b75e-4f5205d48dcd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/025e9d83-1372-496c-b75e-4f5205d48dcd-istio-data\") pod \"025e9d83-1372-496c-b75e-4f5205d48dcd\" (UID: \"025e9d83-1372-496c-b75e-4f5205d48dcd\") " Oct 30 17:14:18.582935 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:18.582892 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/025e9d83-1372-496c-b75e-4f5205d48dcd-credential-socket" pod "025e9d83-1372-496c-b75e-4f5205d48dcd" (UID: "025e9d83-1372-496c-b75e-4f5205d48dcd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:14:18.583349 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:18.583026 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/025e9d83-1372-496c-b75e-4f5205d48dcd-istio-token\" (UniqueName: \"kubernetes.io/projected/025e9d83-1372-496c-b75e-4f5205d48dcd-istio-token\") pod \"025e9d83-1372-496c-b75e-4f5205d48dcd\" (UID: \"025e9d83-1372-496c-b75e-4f5205d48dcd\") " Oct 30 17:14:18.583349 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:18.583073 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/025e9d83-1372-496c-b75e-4f5205d48dcd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/025e9d83-1372-496c-b75e-4f5205d48dcd-istiod-ca-cert\") pod \"025e9d83-1372-496c-b75e-4f5205d48dcd\" (UID: \"025e9d83-1372-496c-b75e-4f5205d48dcd\") " Oct 30 17:14:18.583349 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:18.583160 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/025e9d83-1372-496c-b75e-4f5205d48dcd-istio-data" pod "025e9d83-1372-496c-b75e-4f5205d48dcd" (UID: "025e9d83-1372-496c-b75e-4f5205d48dcd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:14:18.583609 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:18.583571 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/025e9d83-1372-496c-b75e-4f5205d48dcd-istiod-ca-cert" pod "025e9d83-1372-496c-b75e-4f5205d48dcd" (UID: "025e9d83-1372-496c-b75e-4f5205d48dcd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:14:18.583738 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:18.583718 5052 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/025e9d83-1372-496c-b75e-4f5205d48dcd-workload-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:14:18.583810 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:18.583742 5052 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/025e9d83-1372-496c-b75e-4f5205d48dcd-workload-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:14:18.583810 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:18.583756 5052 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/025e9d83-1372-496c-b75e-4f5205d48dcd-credential-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:14:18.583810 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:18.583770 5052 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/025e9d83-1372-496c-b75e-4f5205d48dcd-istio-data\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:14:18.583810 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:18.583783 5052 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/025e9d83-1372-496c-b75e-4f5205d48dcd-istiod-ca-cert\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:14:18.585035 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:18.585009 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/025e9d83-1372-496c-b75e-4f5205d48dcd-istio-podinfo" pod "025e9d83-1372-496c-b75e-4f5205d48dcd" (UID: "025e9d83-1372-496c-b75e-4f5205d48dcd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:14:18.585956 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:18.585926 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/025e9d83-1372-496c-b75e-4f5205d48dcd-istio-token" pod "025e9d83-1372-496c-b75e-4f5205d48dcd" (UID: "025e9d83-1372-496c-b75e-4f5205d48dcd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:14:18.586371 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:18.586251 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/025e9d83-1372-496c-b75e-4f5205d48dcd-kube-api-access-72vht" pod "025e9d83-1372-496c-b75e-4f5205d48dcd" (UID: "025e9d83-1372-496c-b75e-4f5205d48dcd"). InnerVolumeSpecName "kube-api-access-72vht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:14:18.586371 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:18.586359 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/025e9d83-1372-496c-b75e-4f5205d48dcd-istio-envoy" pod "025e9d83-1372-496c-b75e-4f5205d48dcd" (UID: "025e9d83-1372-496c-b75e-4f5205d48dcd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:14:18.609777 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:18.599375 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-6jzqj" Oct 30 17:14:18.684902 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:18.684867 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fb4a801-fd82-4036-ba89-79fdf32d3be3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fb4a801-fd82-4036-ba89-79fdf32d3be3-credential-socket\") pod \"1fb4a801-fd82-4036-ba89-79fdf32d3be3\" (UID: \"1fb4a801-fd82-4036-ba89-79fdf32d3be3\") " Oct 30 17:14:18.685097 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:18.684944 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fb4a801-fd82-4036-ba89-79fdf32d3be3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fb4a801-fd82-4036-ba89-79fdf32d3be3-istio-data\") pod \"1fb4a801-fd82-4036-ba89-79fdf32d3be3\" (UID: \"1fb4a801-fd82-4036-ba89-79fdf32d3be3\") " Oct 30 17:14:18.685097 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:18.684985 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fb4a801-fd82-4036-ba89-79fdf32d3be3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fb4a801-fd82-4036-ba89-79fdf32d3be3-istiod-ca-cert\") pod \"1fb4a801-fd82-4036-ba89-79fdf32d3be3\" (UID: \"1fb4a801-fd82-4036-ba89-79fdf32d3be3\") " Oct 30 17:14:18.685097 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:18.685002 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fb4a801-fd82-4036-ba89-79fdf32d3be3-istio-token\" (UniqueName: \"kubernetes.io/projected/1fb4a801-fd82-4036-ba89-79fdf32d3be3-istio-token\") pod \"1fb4a801-fd82-4036-ba89-79fdf32d3be3\" (UID: \"1fb4a801-fd82-4036-ba89-79fdf32d3be3\") " Oct 30 17:14:18.685097 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:18.685085 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fb4a801-fd82-4036-ba89-79fdf32d3be3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fb4a801-fd82-4036-ba89-79fdf32d3be3-workload-certs\") pod \"1fb4a801-fd82-4036-ba89-79fdf32d3be3\" (UID: \"1fb4a801-fd82-4036-ba89-79fdf32d3be3\") " Oct 30 17:14:18.685321 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:18.685147 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fb4a801-fd82-4036-ba89-79fdf32d3be3-credential-socket" pod "1fb4a801-fd82-4036-ba89-79fdf32d3be3" (UID: "1fb4a801-fd82-4036-ba89-79fdf32d3be3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:14:18.685321 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:18.685163 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fb4a801-fd82-4036-ba89-79fdf32d3be3-kube-api-access-lwrx9\" (UniqueName: \"kubernetes.io/projected/1fb4a801-fd82-4036-ba89-79fdf32d3be3-kube-api-access-lwrx9\") pod \"1fb4a801-fd82-4036-ba89-79fdf32d3be3\" (UID: \"1fb4a801-fd82-4036-ba89-79fdf32d3be3\") " Oct 30 17:14:18.685321 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:18.685224 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fb4a801-fd82-4036-ba89-79fdf32d3be3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fb4a801-fd82-4036-ba89-79fdf32d3be3-workload-socket\") pod \"1fb4a801-fd82-4036-ba89-79fdf32d3be3\" (UID: \"1fb4a801-fd82-4036-ba89-79fdf32d3be3\") " Oct 30 17:14:18.685321 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:18.685281 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1fb4a801-fd82-4036-ba89-79fdf32d3be3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fb4a801-fd82-4036-ba89-79fdf32d3be3-istio-podinfo\") pod \"1fb4a801-fd82-4036-ba89-79fdf32d3be3\" (UID: \"1fb4a801-fd82-4036-ba89-79fdf32d3be3\") " Oct 30 17:14:18.685524 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:18.685377 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fb4a801-fd82-4036-ba89-79fdf32d3be3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fb4a801-fd82-4036-ba89-79fdf32d3be3-istio-envoy\") pod \"1fb4a801-fd82-4036-ba89-79fdf32d3be3\" (UID: \"1fb4a801-fd82-4036-ba89-79fdf32d3be3\") " Oct 30 17:14:18.685753 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:18.685280 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fb4a801-fd82-4036-ba89-79fdf32d3be3-istio-data" pod "1fb4a801-fd82-4036-ba89-79fdf32d3be3" (UID: "1fb4a801-fd82-4036-ba89-79fdf32d3be3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:14:18.685875 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:18.685555 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fb4a801-fd82-4036-ba89-79fdf32d3be3-workload-socket" pod "1fb4a801-fd82-4036-ba89-79fdf32d3be3" (UID: "1fb4a801-fd82-4036-ba89-79fdf32d3be3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:14:18.685875 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:18.685618 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fb4a801-fd82-4036-ba89-79fdf32d3be3-istiod-ca-cert" pod "1fb4a801-fd82-4036-ba89-79fdf32d3be3" (UID: "1fb4a801-fd82-4036-ba89-79fdf32d3be3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:14:18.686064 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:18.686040 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fb4a801-fd82-4036-ba89-79fdf32d3be3-workload-certs" pod "1fb4a801-fd82-4036-ba89-79fdf32d3be3" (UID: "1fb4a801-fd82-4036-ba89-79fdf32d3be3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:14:18.686177 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:18.686148 5052 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/025e9d83-1372-496c-b75e-4f5205d48dcd-istio-envoy\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:14:18.686239 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:18.686181 5052 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fb4a801-fd82-4036-ba89-79fdf32d3be3-istio-data\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:14:18.686239 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:18.686197 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72vht\" (UniqueName: \"kubernetes.io/projected/025e9d83-1372-496c-b75e-4f5205d48dcd-kube-api-access-72vht\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:14:18.686239 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:18.686213 5052 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fb4a801-fd82-4036-ba89-79fdf32d3be3-istiod-ca-cert\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:14:18.686239 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:18.686227 5052 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fb4a801-fd82-4036-ba89-79fdf32d3be3-workload-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:14:18.686442 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:18.686239 5052 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/025e9d83-1372-496c-b75e-4f5205d48dcd-istio-token\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:14:18.686442 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:18.686251 5052 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fb4a801-fd82-4036-ba89-79fdf32d3be3-workload-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:14:18.686442 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:18.686265 5052 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/025e9d83-1372-496c-b75e-4f5205d48dcd-istio-podinfo\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:14:18.686442 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:18.686278 5052 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fb4a801-fd82-4036-ba89-79fdf32d3be3-credential-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:14:18.687924 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:18.687897 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fb4a801-fd82-4036-ba89-79fdf32d3be3-istio-token" pod "1fb4a801-fd82-4036-ba89-79fdf32d3be3" (UID: "1fb4a801-fd82-4036-ba89-79fdf32d3be3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:14:18.688030 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:18.687939 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fb4a801-fd82-4036-ba89-79fdf32d3be3-kube-api-access-lwrx9" pod "1fb4a801-fd82-4036-ba89-79fdf32d3be3" (UID: "1fb4a801-fd82-4036-ba89-79fdf32d3be3"). InnerVolumeSpecName "kube-api-access-lwrx9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:14:18.688492 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:18.688470 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fb4a801-fd82-4036-ba89-79fdf32d3be3-istio-envoy" pod "1fb4a801-fd82-4036-ba89-79fdf32d3be3" (UID: "1fb4a801-fd82-4036-ba89-79fdf32d3be3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:14:18.689720 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:18.689695 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1fb4a801-fd82-4036-ba89-79fdf32d3be3-istio-podinfo" pod "1fb4a801-fd82-4036-ba89-79fdf32d3be3" (UID: "1fb4a801-fd82-4036-ba89-79fdf32d3be3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:14:18.787558 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:18.787480 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwrx9\" (UniqueName: \"kubernetes.io/projected/1fb4a801-fd82-4036-ba89-79fdf32d3be3-kube-api-access-lwrx9\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:14:18.787558 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:18.787511 5052 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fb4a801-fd82-4036-ba89-79fdf32d3be3-istio-podinfo\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:14:18.787558 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:18.787522 5052 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fb4a801-fd82-4036-ba89-79fdf32d3be3-istio-envoy\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:14:18.787558 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:18.787531 5052 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fb4a801-fd82-4036-ba89-79fdf32d3be3-istio-token\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:14:18.842362 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:14:18.842325 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 17:14:19.023297 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:19.023262 5052 generic.go:356] "Generic (PLEG): container finished" podID="025e9d83-1372-496c-b75e-4f5205d48dcd" containerID="538de255129a6d08fdad51382b6432e2f8d70e4102e68fef4706550d0780ae26" exitCode=0 Oct 30 17:14:19.023534 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:19.023324 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-kgzxp" Oct 30 17:14:19.023534 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:19.023358 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-kgzxp" event={"ID":"025e9d83-1372-496c-b75e-4f5205d48dcd","Type":"ContainerDied","Data":"538de255129a6d08fdad51382b6432e2f8d70e4102e68fef4706550d0780ae26"} Oct 30 17:14:19.023534 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:19.023426 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-kgzxp" event={"ID":"025e9d83-1372-496c-b75e-4f5205d48dcd","Type":"ContainerDied","Data":"b3ba54dc2b2ab591ad8d57526daac4c0496b1174fafcd1dc10fe6f3699d01c22"} Oct 30 17:14:19.023534 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:19.023461 5052 scope.go:122] "RemoveContainer" containerID="538de255129a6d08fdad51382b6432e2f8d70e4102e68fef4706550d0780ae26" Oct 30 17:14:19.025319 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:19.025293 5052 generic.go:356] "Generic (PLEG): container finished" podID="1fb4a801-fd82-4036-ba89-79fdf32d3be3" containerID="b9e3fbca33f82af327ba9d2612b8bfb6b0763849081e591fa80f05e22433665a" exitCode=0 Oct 30 17:14:19.025448 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:19.025356 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-6jzqj" Oct 30 17:14:19.025448 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:19.025398 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-6jzqj" event={"ID":"1fb4a801-fd82-4036-ba89-79fdf32d3be3","Type":"ContainerDied","Data":"b9e3fbca33f82af327ba9d2612b8bfb6b0763849081e591fa80f05e22433665a"} Oct 30 17:14:19.025618 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:19.025483 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-6jzqj" event={"ID":"1fb4a801-fd82-4036-ba89-79fdf32d3be3","Type":"ContainerDied","Data":"73fb543e4645885a1ddf6207dcef913129be63e23f30da9fe57d905c7f899baf"} Oct 30 17:14:19.036556 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:19.036525 5052 scope.go:122] "RemoveContainer" containerID="538de255129a6d08fdad51382b6432e2f8d70e4102e68fef4706550d0780ae26" Oct 30 17:14:19.036860 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:14:19.036836 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"538de255129a6d08fdad51382b6432e2f8d70e4102e68fef4706550d0780ae26\": container with ID starting with 538de255129a6d08fdad51382b6432e2f8d70e4102e68fef4706550d0780ae26 not found: ID does not exist" containerID="538de255129a6d08fdad51382b6432e2f8d70e4102e68fef4706550d0780ae26" Oct 30 17:14:19.036965 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:19.036863 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"538de255129a6d08fdad51382b6432e2f8d70e4102e68fef4706550d0780ae26"} err="failed to get container status \"538de255129a6d08fdad51382b6432e2f8d70e4102e68fef4706550d0780ae26\": rpc error: code = NotFound desc = could not find container \"538de255129a6d08fdad51382b6432e2f8d70e4102e68fef4706550d0780ae26\": container with ID starting with 538de255129a6d08fdad51382b6432e2f8d70e4102e68fef4706550d0780ae26 not found: ID does not exist" Oct 30 17:14:19.036965 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:19.036879 5052 scope.go:122] "RemoveContainer" containerID="b9e3fbca33f82af327ba9d2612b8bfb6b0763849081e591fa80f05e22433665a" Oct 30 17:14:19.047215 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:19.047187 5052 scope.go:122] "RemoveContainer" containerID="8ecd743ed8132bc3c3146a7ea6e84cb5048848ec8e3982af6e37ab90b57a97a0" Oct 30 17:14:19.056546 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:19.056524 5052 scope.go:122] "RemoveContainer" containerID="b9e3fbca33f82af327ba9d2612b8bfb6b0763849081e591fa80f05e22433665a" Oct 30 17:14:19.056975 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:14:19.056947 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9e3fbca33f82af327ba9d2612b8bfb6b0763849081e591fa80f05e22433665a\": container with ID starting with b9e3fbca33f82af327ba9d2612b8bfb6b0763849081e591fa80f05e22433665a not found: ID does not exist" containerID="b9e3fbca33f82af327ba9d2612b8bfb6b0763849081e591fa80f05e22433665a" Oct 30 17:14:19.057095 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:19.056982 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9e3fbca33f82af327ba9d2612b8bfb6b0763849081e591fa80f05e22433665a"} err="failed to get container status \"b9e3fbca33f82af327ba9d2612b8bfb6b0763849081e591fa80f05e22433665a\": rpc error: code = NotFound desc = could not find container \"b9e3fbca33f82af327ba9d2612b8bfb6b0763849081e591fa80f05e22433665a\": container with ID starting with b9e3fbca33f82af327ba9d2612b8bfb6b0763849081e591fa80f05e22433665a not found: ID does not exist" Oct 30 17:14:19.057095 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:19.057006 5052 scope.go:122] "RemoveContainer" containerID="8ecd743ed8132bc3c3146a7ea6e84cb5048848ec8e3982af6e37ab90b57a97a0" Oct 30 17:14:19.057395 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:14:19.057361 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ecd743ed8132bc3c3146a7ea6e84cb5048848ec8e3982af6e37ab90b57a97a0\": container with ID starting with 8ecd743ed8132bc3c3146a7ea6e84cb5048848ec8e3982af6e37ab90b57a97a0 not found: ID does not exist" containerID="8ecd743ed8132bc3c3146a7ea6e84cb5048848ec8e3982af6e37ab90b57a97a0" Oct 30 17:14:19.057487 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:19.057404 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ecd743ed8132bc3c3146a7ea6e84cb5048848ec8e3982af6e37ab90b57a97a0"} err="failed to get container status \"8ecd743ed8132bc3c3146a7ea6e84cb5048848ec8e3982af6e37ab90b57a97a0\": rpc error: code = NotFound desc = could not find container \"8ecd743ed8132bc3c3146a7ea6e84cb5048848ec8e3982af6e37ab90b57a97a0\": container with ID starting with 8ecd743ed8132bc3c3146a7ea6e84cb5048848ec8e3982af6e37ab90b57a97a0 not found: ID does not exist" Oct 30 17:14:19.080928 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:19.080890 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-6jzqj"] Oct 30 17:14:19.093740 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:19.093708 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-6jzqj"] Oct 30 17:14:19.118904 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:19.118870 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-kgzxp"] Oct 30 17:14:19.125383 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:19.125359 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-kgzxp"] Oct 30 17:14:20.034387 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:20.034348 5052 generic.go:356] "Generic (PLEG): container finished" podID="0261ebf2-6dfd-4df5-b5de-4b7207ca761b" containerID="679ff824e61208b26c583fee45e391f0e33d81c23c74fcb9f7cc6f1a87d566ab" exitCode=0 Oct 30 17:14:20.034911 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:20.034422 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ddc10d-openshift-default-56bfdd795c-rjzgr" event={"ID":"0261ebf2-6dfd-4df5-b5de-4b7207ca761b","Type":"ContainerDied","Data":"679ff824e61208b26c583fee45e391f0e33d81c23c74fcb9f7cc6f1a87d566ab"} Oct 30 17:14:20.035172 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:20.035150 5052 scope.go:122] "RemoveContainer" containerID="679ff824e61208b26c583fee45e391f0e33d81c23c74fcb9f7cc6f1a87d566ab" Oct 30 17:14:20.037320 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:20.037292 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:14:20.037423 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:20.037360 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:14:20.037423 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:20.037394 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:14:20.600745 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:20.600710 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="025e9d83-1372-496c-b75e-4f5205d48dcd" path="/var/lib/kubelet/pods/025e9d83-1372-496c-b75e-4f5205d48dcd/volumes" Oct 30 17:14:20.601270 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:20.601255 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fb4a801-fd82-4036-ba89-79fdf32d3be3" path="/var/lib/kubelet/pods/1fb4a801-fd82-4036-ba89-79fdf32d3be3/volumes" Oct 30 17:14:21.046890 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:21.046851 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ddc10d-openshift-default-56bfdd795c-rjzgr" event={"ID":"0261ebf2-6dfd-4df5-b5de-4b7207ca761b","Type":"ContainerStarted","Data":"c0565b2a4bdbe56e618b4c6badff19e973ba08732d43eec64ab1afa4a4d38ce9"} Oct 30 17:14:21.497755 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:21.497718 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-ddc10d-openshift-default-56bfdd795c-rjzgr" Oct 30 17:14:21.497964 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:21.497894 5052 patch_prober.go:29] interesting pod/e2e-ipp-b-ddc10d-openshift-default-56bfdd795c-rjzgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body= Oct 30 17:14:21.497964 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:21.497945 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ddc10d-openshift-default-56bfdd795c-rjzgr" podUID="0261ebf2-6dfd-4df5-b5de-4b7207ca761b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" Oct 30 17:14:22.498271 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:22.498236 5052 patch_prober.go:29] interesting pod/e2e-ipp-b-ddc10d-openshift-default-56bfdd795c-rjzgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body= Oct 30 17:14:22.498708 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:22.498288 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ddc10d-openshift-default-56bfdd795c-rjzgr" podUID="0261ebf2-6dfd-4df5-b5de-4b7207ca761b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" Oct 30 17:14:23.498326 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:23.498293 5052 patch_prober.go:29] interesting pod/e2e-ipp-b-ddc10d-openshift-default-56bfdd795c-rjzgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body= Oct 30 17:14:23.498750 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:23.498343 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ddc10d-openshift-default-56bfdd795c-rjzgr" podUID="0261ebf2-6dfd-4df5-b5de-4b7207ca761b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" Oct 30 17:14:24.497929 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:24.497888 5052 patch_prober.go:29] interesting pod/e2e-ipp-b-ddc10d-openshift-default-56bfdd795c-rjzgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body= Oct 30 17:14:24.498124 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:24.497977 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ddc10d-openshift-default-56bfdd795c-rjzgr" podUID="0261ebf2-6dfd-4df5-b5de-4b7207ca761b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" Oct 30 17:14:25.498471 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:25.498437 5052 patch_prober.go:29] interesting pod/e2e-ipp-b-ddc10d-openshift-default-56bfdd795c-rjzgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body= Oct 30 17:14:25.498885 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:25.498504 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ddc10d-openshift-default-56bfdd795c-rjzgr" podUID="0261ebf2-6dfd-4df5-b5de-4b7207ca761b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" Oct 30 17:14:26.498384 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:26.498348 5052 patch_prober.go:29] interesting pod/e2e-ipp-b-ddc10d-openshift-default-56bfdd795c-rjzgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body= Oct 30 17:14:26.498621 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:26.498407 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ddc10d-openshift-default-56bfdd795c-rjzgr" podUID="0261ebf2-6dfd-4df5-b5de-4b7207ca761b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" Oct 30 17:14:27.504789 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:27.504751 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-ddc10d-openshift-default-56bfdd795c-rjzgr" Oct 30 17:14:27.505321 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:27.505266 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-ddc10d-openshift-default-56bfdd795c-rjzgr" Oct 30 17:14:27.506024 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:27.506004 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-ddc10d-openshift-default-56bfdd795c-rjzgr" Oct 30 17:14:27.589911 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:27.589870 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-967fac-7cbc897887-drl9k"] Oct 30 17:14:27.590258 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:27.590184 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-967fac-7cbc897887-drl9k" podUID="051f70f5-3c9d-44e7-9d47-d5b21feeee6e" containerName="maas-api" containerID="cri-o://870984cc4d77133579753418ba7ac24b263679c34e5c59b9255a908e704fbb3b" gracePeriod=30 Oct 30 17:14:27.950442 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:27.950066 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-967fac-7cbc897887-drl9k" Oct 30 17:14:28.217627 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:28.201848 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/051f70f5-3c9d-44e7-9d47-d5b21feeee6e-kube-api-access-2jcwt\" (UniqueName: \"kubernetes.io/projected/051f70f5-3c9d-44e7-9d47-d5b21feeee6e-kube-api-access-2jcwt\") pod \"051f70f5-3c9d-44e7-9d47-d5b21feeee6e\" (UID: \"051f70f5-3c9d-44e7-9d47-d5b21feeee6e\") " Oct 30 17:14:28.217627 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:28.201908 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/051f70f5-3c9d-44e7-9d47-d5b21feeee6e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/051f70f5-3c9d-44e7-9d47-d5b21feeee6e-openshift-service-ca\") pod \"051f70f5-3c9d-44e7-9d47-d5b21feeee6e\" (UID: \"051f70f5-3c9d-44e7-9d47-d5b21feeee6e\") " Oct 30 17:14:28.217627 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:28.201939 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/051f70f5-3c9d-44e7-9d47-d5b21feeee6e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/051f70f5-3c9d-44e7-9d47-d5b21feeee6e-maas-api-tls\") pod \"051f70f5-3c9d-44e7-9d47-d5b21feeee6e\" (UID: \"051f70f5-3c9d-44e7-9d47-d5b21feeee6e\") " Oct 30 17:14:28.217627 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:28.201995 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/051f70f5-3c9d-44e7-9d47-d5b21feeee6e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/051f70f5-3c9d-44e7-9d47-d5b21feeee6e-metrics-certs\") pod \"051f70f5-3c9d-44e7-9d47-d5b21feeee6e\" (UID: \"051f70f5-3c9d-44e7-9d47-d5b21feeee6e\") " Oct 30 17:14:28.224743 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:28.224498 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/051f70f5-3c9d-44e7-9d47-d5b21feeee6e-openshift-service-ca" pod "051f70f5-3c9d-44e7-9d47-d5b21feeee6e" (UID: "051f70f5-3c9d-44e7-9d47-d5b21feeee6e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:14:28.321068 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:28.304782 5052 generic.go:356] "Generic (PLEG): container finished" podID="051f70f5-3c9d-44e7-9d47-d5b21feeee6e" containerID="870984cc4d77133579753418ba7ac24b263679c34e5c59b9255a908e704fbb3b" exitCode=0 Oct 30 17:14:28.321068 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:28.305808 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-967fac-7cbc897887-drl9k" Oct 30 17:14:28.321068 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:28.306343 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-967fac-7cbc897887-drl9k" event={"ID":"051f70f5-3c9d-44e7-9d47-d5b21feeee6e","Type":"ContainerDied","Data":"870984cc4d77133579753418ba7ac24b263679c34e5c59b9255a908e704fbb3b"} Oct 30 17:14:28.321068 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:28.306450 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-967fac-7cbc897887-drl9k" event={"ID":"051f70f5-3c9d-44e7-9d47-d5b21feeee6e","Type":"ContainerDied","Data":"6cfd84e4a9facfa944db9a50732fd292c3ca0c8fb2416c348b3b42b272e4c036"} Oct 30 17:14:28.321068 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:28.306664 5052 scope.go:122] "RemoveContainer" containerID="870984cc4d77133579753418ba7ac24b263679c34e5c59b9255a908e704fbb3b" Oct 30 17:14:28.321068 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:28.317807 5052 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/051f70f5-3c9d-44e7-9d47-d5b21feeee6e-openshift-service-ca\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:14:28.321068 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:28.319613 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/051f70f5-3c9d-44e7-9d47-d5b21feeee6e-kube-api-access-2jcwt" pod "051f70f5-3c9d-44e7-9d47-d5b21feeee6e" (UID: "051f70f5-3c9d-44e7-9d47-d5b21feeee6e"). InnerVolumeSpecName "kube-api-access-2jcwt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:14:28.335892 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:28.331811 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/051f70f5-3c9d-44e7-9d47-d5b21feeee6e-maas-api-tls" pod "051f70f5-3c9d-44e7-9d47-d5b21feeee6e" (UID: "051f70f5-3c9d-44e7-9d47-d5b21feeee6e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:14:28.335892 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:28.332243 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/051f70f5-3c9d-44e7-9d47-d5b21feeee6e-metrics-certs" pod "051f70f5-3c9d-44e7-9d47-d5b21feeee6e" (UID: "051f70f5-3c9d-44e7-9d47-d5b21feeee6e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:14:28.344098 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:28.344077 5052 scope.go:122] "RemoveContainer" containerID="870984cc4d77133579753418ba7ac24b263679c34e5c59b9255a908e704fbb3b" Oct 30 17:14:28.344439 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:14:28.344416 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"870984cc4d77133579753418ba7ac24b263679c34e5c59b9255a908e704fbb3b\": container with ID starting with 870984cc4d77133579753418ba7ac24b263679c34e5c59b9255a908e704fbb3b not found: ID does not exist" containerID="870984cc4d77133579753418ba7ac24b263679c34e5c59b9255a908e704fbb3b" Oct 30 17:14:28.344551 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:28.344448 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"870984cc4d77133579753418ba7ac24b263679c34e5c59b9255a908e704fbb3b"} err="failed to get container status \"870984cc4d77133579753418ba7ac24b263679c34e5c59b9255a908e704fbb3b\": rpc error: code = NotFound desc = could not find container \"870984cc4d77133579753418ba7ac24b263679c34e5c59b9255a908e704fbb3b\": container with ID starting with 870984cc4d77133579753418ba7ac24b263679c34e5c59b9255a908e704fbb3b not found: ID does not exist" Oct 30 17:14:28.419904 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:28.419864 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jcwt\" (UniqueName: \"kubernetes.io/projected/051f70f5-3c9d-44e7-9d47-d5b21feeee6e-kube-api-access-2jcwt\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:14:28.420159 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:28.420130 5052 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/051f70f5-3c9d-44e7-9d47-d5b21feeee6e-maas-api-tls\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:14:28.420159 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:28.420162 5052 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/051f70f5-3c9d-44e7-9d47-d5b21feeee6e-metrics-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:14:28.661773 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:28.661744 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-967fac-7cbc897887-drl9k"] Oct 30 17:14:28.666850 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:28.666827 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-967fac-7cbc897887-drl9k"] Oct 30 17:14:30.600474 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:30.600439 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="051f70f5-3c9d-44e7-9d47-d5b21feeee6e" path="/var/lib/kubelet/pods/051f70f5-3c9d-44e7-9d47-d5b21feeee6e/volumes" Oct 30 17:14:35.738914 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:35.738874 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-fvfrl"] Oct 30 17:14:35.739443 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:35.739122 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-fvfrl" podUID="bc4d11ae-edd2-4143-95fd-b57bf4486e76" containerName="istio-proxy" containerID="cri-o://64186719511d27956a6d28fbb00d1f613a2086dc379cddd7f08590bdee8afc9c" gracePeriod=30 Oct 30 17:14:41.025484 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:41.025420 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-fvfrl" Oct 30 17:14:41.148646 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:41.148572 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bc4d11ae-edd2-4143-95fd-b57bf4486e76-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc4d11ae-edd2-4143-95fd-b57bf4486e76-istio-podinfo\") pod \"bc4d11ae-edd2-4143-95fd-b57bf4486e76\" (UID: \"bc4d11ae-edd2-4143-95fd-b57bf4486e76\") " Oct 30 17:14:41.148854 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:41.148688 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc4d11ae-edd2-4143-95fd-b57bf4486e76-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc4d11ae-edd2-4143-95fd-b57bf4486e76-istiod-ca-cert\") pod \"bc4d11ae-edd2-4143-95fd-b57bf4486e76\" (UID: \"bc4d11ae-edd2-4143-95fd-b57bf4486e76\") " Oct 30 17:14:41.148854 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:41.148715 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc4d11ae-edd2-4143-95fd-b57bf4486e76-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc4d11ae-edd2-4143-95fd-b57bf4486e76-istio-envoy\") pod \"bc4d11ae-edd2-4143-95fd-b57bf4486e76\" (UID: \"bc4d11ae-edd2-4143-95fd-b57bf4486e76\") " Oct 30 17:14:41.148854 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:41.148751 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc4d11ae-edd2-4143-95fd-b57bf4486e76-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc4d11ae-edd2-4143-95fd-b57bf4486e76-credential-socket\") pod \"bc4d11ae-edd2-4143-95fd-b57bf4486e76\" (UID: \"bc4d11ae-edd2-4143-95fd-b57bf4486e76\") " Oct 30 17:14:41.148854 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:41.148781 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc4d11ae-edd2-4143-95fd-b57bf4486e76-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc4d11ae-edd2-4143-95fd-b57bf4486e76-workload-certs\") pod \"bc4d11ae-edd2-4143-95fd-b57bf4486e76\" (UID: \"bc4d11ae-edd2-4143-95fd-b57bf4486e76\") " Oct 30 17:14:41.148854 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:41.148830 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc4d11ae-edd2-4143-95fd-b57bf4486e76-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc4d11ae-edd2-4143-95fd-b57bf4486e76-workload-socket\") pod \"bc4d11ae-edd2-4143-95fd-b57bf4486e76\" (UID: \"bc4d11ae-edd2-4143-95fd-b57bf4486e76\") " Oct 30 17:14:41.149233 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:41.148920 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc4d11ae-edd2-4143-95fd-b57bf4486e76-kube-api-access-jbhpg\" (UniqueName: \"kubernetes.io/projected/bc4d11ae-edd2-4143-95fd-b57bf4486e76-kube-api-access-jbhpg\") pod \"bc4d11ae-edd2-4143-95fd-b57bf4486e76\" (UID: \"bc4d11ae-edd2-4143-95fd-b57bf4486e76\") " Oct 30 17:14:41.149233 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:41.148948 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc4d11ae-edd2-4143-95fd-b57bf4486e76-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc4d11ae-edd2-4143-95fd-b57bf4486e76-istio-data\") pod \"bc4d11ae-edd2-4143-95fd-b57bf4486e76\" (UID: \"bc4d11ae-edd2-4143-95fd-b57bf4486e76\") " Oct 30 17:14:41.149233 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:41.149070 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc4d11ae-edd2-4143-95fd-b57bf4486e76-credential-socket" pod "bc4d11ae-edd2-4143-95fd-b57bf4486e76" (UID: "bc4d11ae-edd2-4143-95fd-b57bf4486e76"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:14:41.149233 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:41.149145 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc4d11ae-edd2-4143-95fd-b57bf4486e76-workload-certs" pod "bc4d11ae-edd2-4143-95fd-b57bf4486e76" (UID: "bc4d11ae-edd2-4143-95fd-b57bf4486e76"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:14:41.149233 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:41.149152 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc4d11ae-edd2-4143-95fd-b57bf4486e76-istiod-ca-cert" pod "bc4d11ae-edd2-4143-95fd-b57bf4486e76" (UID: "bc4d11ae-edd2-4143-95fd-b57bf4486e76"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:14:41.149233 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:41.149205 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc4d11ae-edd2-4143-95fd-b57bf4486e76-istio-token\" (UniqueName: \"kubernetes.io/projected/bc4d11ae-edd2-4143-95fd-b57bf4486e76-istio-token\") pod \"bc4d11ae-edd2-4143-95fd-b57bf4486e76\" (UID: \"bc4d11ae-edd2-4143-95fd-b57bf4486e76\") " Oct 30 17:14:41.149665 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:41.149389 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc4d11ae-edd2-4143-95fd-b57bf4486e76-istio-data" pod "bc4d11ae-edd2-4143-95fd-b57bf4486e76" (UID: "bc4d11ae-edd2-4143-95fd-b57bf4486e76"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:14:41.150023 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:41.149961 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc4d11ae-edd2-4143-95fd-b57bf4486e76-workload-socket" pod "bc4d11ae-edd2-4143-95fd-b57bf4486e76" (UID: "bc4d11ae-edd2-4143-95fd-b57bf4486e76"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:14:41.150188 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:41.150165 5052 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc4d11ae-edd2-4143-95fd-b57bf4486e76-istio-data\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:14:41.150266 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:41.150193 5052 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc4d11ae-edd2-4143-95fd-b57bf4486e76-istiod-ca-cert\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:14:41.150266 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:41.150208 5052 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc4d11ae-edd2-4143-95fd-b57bf4486e76-credential-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:14:41.150266 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:41.150222 5052 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc4d11ae-edd2-4143-95fd-b57bf4486e76-workload-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:14:41.150266 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:41.150237 5052 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc4d11ae-edd2-4143-95fd-b57bf4486e76-workload-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:14:41.151100 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:41.151073 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bc4d11ae-edd2-4143-95fd-b57bf4486e76-istio-podinfo" pod "bc4d11ae-edd2-4143-95fd-b57bf4486e76" (UID: "bc4d11ae-edd2-4143-95fd-b57bf4486e76"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:14:41.151177 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:41.151165 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc4d11ae-edd2-4143-95fd-b57bf4486e76-istio-envoy" pod "bc4d11ae-edd2-4143-95fd-b57bf4486e76" (UID: "bc4d11ae-edd2-4143-95fd-b57bf4486e76"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:14:41.151542 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:41.151515 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc4d11ae-edd2-4143-95fd-b57bf4486e76-kube-api-access-jbhpg" pod "bc4d11ae-edd2-4143-95fd-b57bf4486e76" (UID: "bc4d11ae-edd2-4143-95fd-b57bf4486e76"). InnerVolumeSpecName "kube-api-access-jbhpg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:14:41.151858 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:41.151843 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc4d11ae-edd2-4143-95fd-b57bf4486e76-istio-token" pod "bc4d11ae-edd2-4143-95fd-b57bf4486e76" (UID: "bc4d11ae-edd2-4143-95fd-b57bf4486e76"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:14:41.251813 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:41.251782 5052 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc4d11ae-edd2-4143-95fd-b57bf4486e76-istio-token\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:14:41.251813 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:41.251813 5052 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc4d11ae-edd2-4143-95fd-b57bf4486e76-istio-podinfo\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:14:41.252008 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:41.251827 5052 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc4d11ae-edd2-4143-95fd-b57bf4486e76-istio-envoy\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:14:41.252008 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:41.251840 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbhpg\" (UniqueName: \"kubernetes.io/projected/bc4d11ae-edd2-4143-95fd-b57bf4486e76-kube-api-access-jbhpg\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:14:41.405257 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:41.405227 5052 generic.go:356] "Generic (PLEG): container finished" podID="bc4d11ae-edd2-4143-95fd-b57bf4486e76" containerID="64186719511d27956a6d28fbb00d1f613a2086dc379cddd7f08590bdee8afc9c" exitCode=0 Oct 30 17:14:41.405423 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:41.405269 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-fvfrl" Oct 30 17:14:41.405423 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:41.405319 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-fvfrl" event={"ID":"bc4d11ae-edd2-4143-95fd-b57bf4486e76","Type":"ContainerDied","Data":"64186719511d27956a6d28fbb00d1f613a2086dc379cddd7f08590bdee8afc9c"} Oct 30 17:14:41.405423 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:41.405368 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-fvfrl" event={"ID":"bc4d11ae-edd2-4143-95fd-b57bf4486e76","Type":"ContainerDied","Data":"62fcf89e01fbbf9036289232ab19a8ccd9c13583951d8e27583c0bd75c1ba97d"} Oct 30 17:14:41.405423 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:41.405389 5052 scope.go:122] "RemoveContainer" containerID="64186719511d27956a6d28fbb00d1f613a2086dc379cddd7f08590bdee8afc9c" Oct 30 17:14:41.416765 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:41.416749 5052 scope.go:122] "RemoveContainer" containerID="64186719511d27956a6d28fbb00d1f613a2086dc379cddd7f08590bdee8afc9c" Oct 30 17:14:41.417007 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:14:41.416988 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64186719511d27956a6d28fbb00d1f613a2086dc379cddd7f08590bdee8afc9c\": container with ID starting with 64186719511d27956a6d28fbb00d1f613a2086dc379cddd7f08590bdee8afc9c not found: ID does not exist" containerID="64186719511d27956a6d28fbb00d1f613a2086dc379cddd7f08590bdee8afc9c" Oct 30 17:14:41.417066 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:41.417015 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64186719511d27956a6d28fbb00d1f613a2086dc379cddd7f08590bdee8afc9c"} err="failed to get container status \"64186719511d27956a6d28fbb00d1f613a2086dc379cddd7f08590bdee8afc9c\": rpc error: code = NotFound desc = could not find container \"64186719511d27956a6d28fbb00d1f613a2086dc379cddd7f08590bdee8afc9c\": container with ID starting with 64186719511d27956a6d28fbb00d1f613a2086dc379cddd7f08590bdee8afc9c not found: ID does not exist" Oct 30 17:14:41.463920 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:41.463900 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-fvfrl"] Oct 30 17:14:41.476926 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:41.476903 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-fvfrl"] Oct 30 17:14:41.540495 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:41.540462 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ddc10d-55c9674d94-hwbjl"] Oct 30 17:14:41.573631 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:41.573575 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ddc10d-55f45f45f-4dt5v"] Oct 30 17:14:42.599495 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:42.599463 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc4d11ae-edd2-4143-95fd-b57bf4486e76" path="/var/lib/kubelet/pods/bc4d11ae-edd2-4143-95fd-b57bf4486e76/volumes" Oct 30 17:14:45.090786 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:45.089642 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cdwcj"] Oct 30 17:14:46.541184 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:46.541136 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-ddc10d-55c9674d94-hwbjl" podUID="6227d9b1-b976-4afd-ab68-c7e051272d11" containerName="payload-processing" containerID="cri-o://84bba1d9b88b1de3c7da69e2306744cebced1f9497c2edea37aed83e39164172" gracePeriod=25 Oct 30 17:14:46.573855 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:46.573819 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ddc10d-55f45f45f-4dt5v" podUID="ec94c66b-d7c7-43cb-8df7-e7cf32251abb" containerName="payload-pre-processing" containerID="cri-o://79bda2629a22bcf1ce22d67a5f8309983baa31e0758f90fbb2361c669bc042f2" gracePeriod=25 Oct 30 17:14:46.964276 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:46.964245 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ddc10d-55c9674d94-hwbjl" Oct 30 17:14:46.968100 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:46.968076 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ddc10d-55f45f45f-4dt5v" Oct 30 17:14:46.985525 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:46.985502 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec94c66b-d7c7-43cb-8df7-e7cf32251abb-kube-api-access-pgfbq\" (UniqueName: \"kubernetes.io/projected/ec94c66b-d7c7-43cb-8df7-e7cf32251abb-kube-api-access-pgfbq\") pod \"ec94c66b-d7c7-43cb-8df7-e7cf32251abb\" (UID: \"ec94c66b-d7c7-43cb-8df7-e7cf32251abb\") " Oct 30 17:14:46.985651 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:46.985635 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec94c66b-d7c7-43cb-8df7-e7cf32251abb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec94c66b-d7c7-43cb-8df7-e7cf32251abb-plugins-config-volume\") pod \"ec94c66b-d7c7-43cb-8df7-e7cf32251abb\" (UID: \"ec94c66b-d7c7-43cb-8df7-e7cf32251abb\") " Oct 30 17:14:46.985731 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:46.985720 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6227d9b1-b976-4afd-ab68-c7e051272d11-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6227d9b1-b976-4afd-ab68-c7e051272d11-plugins-config-volume\") pod \"6227d9b1-b976-4afd-ab68-c7e051272d11\" (UID: \"6227d9b1-b976-4afd-ab68-c7e051272d11\") " Oct 30 17:14:46.985778 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:46.985756 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6227d9b1-b976-4afd-ab68-c7e051272d11-kube-api-access-p25qf\" (UniqueName: \"kubernetes.io/projected/6227d9b1-b976-4afd-ab68-c7e051272d11-kube-api-access-p25qf\") pod \"6227d9b1-b976-4afd-ab68-c7e051272d11\" (UID: \"6227d9b1-b976-4afd-ab68-c7e051272d11\") " Oct 30 17:14:46.986013 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:46.985991 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec94c66b-d7c7-43cb-8df7-e7cf32251abb-plugins-config-volume" pod "ec94c66b-d7c7-43cb-8df7-e7cf32251abb" (UID: "ec94c66b-d7c7-43cb-8df7-e7cf32251abb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:14:46.986161 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:46.986127 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6227d9b1-b976-4afd-ab68-c7e051272d11-plugins-config-volume" pod "6227d9b1-b976-4afd-ab68-c7e051272d11" (UID: "6227d9b1-b976-4afd-ab68-c7e051272d11"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:14:46.986295 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:46.986208 5052 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec94c66b-d7c7-43cb-8df7-e7cf32251abb-plugins-config-volume\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:14:46.987760 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:46.987737 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec94c66b-d7c7-43cb-8df7-e7cf32251abb-kube-api-access-pgfbq" pod "ec94c66b-d7c7-43cb-8df7-e7cf32251abb" (UID: "ec94c66b-d7c7-43cb-8df7-e7cf32251abb"). InnerVolumeSpecName "kube-api-access-pgfbq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:14:46.987855 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:46.987812 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6227d9b1-b976-4afd-ab68-c7e051272d11-kube-api-access-p25qf" pod "6227d9b1-b976-4afd-ab68-c7e051272d11" (UID: "6227d9b1-b976-4afd-ab68-c7e051272d11"). InnerVolumeSpecName "kube-api-access-p25qf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:14:47.087274 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:47.087194 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgfbq\" (UniqueName: \"kubernetes.io/projected/ec94c66b-d7c7-43cb-8df7-e7cf32251abb-kube-api-access-pgfbq\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:14:47.087274 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:47.087226 5052 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6227d9b1-b976-4afd-ab68-c7e051272d11-plugins-config-volume\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:14:47.087274 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:47.087238 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p25qf\" (UniqueName: \"kubernetes.io/projected/6227d9b1-b976-4afd-ab68-c7e051272d11-kube-api-access-p25qf\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:14:47.449404 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:47.449366 5052 generic.go:356] "Generic (PLEG): container finished" podID="6227d9b1-b976-4afd-ab68-c7e051272d11" containerID="84bba1d9b88b1de3c7da69e2306744cebced1f9497c2edea37aed83e39164172" exitCode=0 Oct 30 17:14:47.449561 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:47.449408 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ddc10d-55c9674d94-hwbjl" Oct 30 17:14:47.449561 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:47.449449 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ddc10d-55c9674d94-hwbjl" event={"ID":"6227d9b1-b976-4afd-ab68-c7e051272d11","Type":"ContainerDied","Data":"84bba1d9b88b1de3c7da69e2306744cebced1f9497c2edea37aed83e39164172"} Oct 30 17:14:47.449561 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:47.449505 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ddc10d-55c9674d94-hwbjl" event={"ID":"6227d9b1-b976-4afd-ab68-c7e051272d11","Type":"ContainerDied","Data":"ce575eaf5027410a2dc4aac3fa842e8ee6058f1259ef7ae0c48431bf7cad2bcb"} Oct 30 17:14:47.449561 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:47.449530 5052 scope.go:122] "RemoveContainer" containerID="84bba1d9b88b1de3c7da69e2306744cebced1f9497c2edea37aed83e39164172" Oct 30 17:14:47.451295 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:47.451270 5052 generic.go:356] "Generic (PLEG): container finished" podID="ec94c66b-d7c7-43cb-8df7-e7cf32251abb" containerID="79bda2629a22bcf1ce22d67a5f8309983baa31e0758f90fbb2361c669bc042f2" exitCode=0 Oct 30 17:14:47.451414 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:47.451316 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ddc10d-55f45f45f-4dt5v" Oct 30 17:14:47.451414 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:47.451355 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ddc10d-55f45f45f-4dt5v" event={"ID":"ec94c66b-d7c7-43cb-8df7-e7cf32251abb","Type":"ContainerDied","Data":"79bda2629a22bcf1ce22d67a5f8309983baa31e0758f90fbb2361c669bc042f2"} Oct 30 17:14:47.451414 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:47.451397 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ddc10d-55f45f45f-4dt5v" event={"ID":"ec94c66b-d7c7-43cb-8df7-e7cf32251abb","Type":"ContainerDied","Data":"e2bccb87c105481f636610000b99281e0d37e661ed7873d430829560b5c17c43"} Oct 30 17:14:47.460293 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:47.460273 5052 scope.go:122] "RemoveContainer" containerID="84bba1d9b88b1de3c7da69e2306744cebced1f9497c2edea37aed83e39164172" Oct 30 17:14:47.460715 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:14:47.460555 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84bba1d9b88b1de3c7da69e2306744cebced1f9497c2edea37aed83e39164172\": container with ID starting with 84bba1d9b88b1de3c7da69e2306744cebced1f9497c2edea37aed83e39164172 not found: ID does not exist" containerID="84bba1d9b88b1de3c7da69e2306744cebced1f9497c2edea37aed83e39164172" Oct 30 17:14:47.460715 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:47.460597 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84bba1d9b88b1de3c7da69e2306744cebced1f9497c2edea37aed83e39164172"} err="failed to get container status \"84bba1d9b88b1de3c7da69e2306744cebced1f9497c2edea37aed83e39164172\": rpc error: code = NotFound desc = could not find container \"84bba1d9b88b1de3c7da69e2306744cebced1f9497c2edea37aed83e39164172\": container with ID starting with 84bba1d9b88b1de3c7da69e2306744cebced1f9497c2edea37aed83e39164172 not found: ID does not exist" Oct 30 17:14:47.460715 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:47.460622 5052 scope.go:122] "RemoveContainer" containerID="79bda2629a22bcf1ce22d67a5f8309983baa31e0758f90fbb2361c669bc042f2" Oct 30 17:14:47.470754 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:47.470733 5052 scope.go:122] "RemoveContainer" containerID="79bda2629a22bcf1ce22d67a5f8309983baa31e0758f90fbb2361c669bc042f2" Oct 30 17:14:47.471017 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:14:47.470997 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79bda2629a22bcf1ce22d67a5f8309983baa31e0758f90fbb2361c669bc042f2\": container with ID starting with 79bda2629a22bcf1ce22d67a5f8309983baa31e0758f90fbb2361c669bc042f2 not found: ID does not exist" containerID="79bda2629a22bcf1ce22d67a5f8309983baa31e0758f90fbb2361c669bc042f2" Oct 30 17:14:47.471086 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:47.471026 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79bda2629a22bcf1ce22d67a5f8309983baa31e0758f90fbb2361c669bc042f2"} err="failed to get container status \"79bda2629a22bcf1ce22d67a5f8309983baa31e0758f90fbb2361c669bc042f2\": rpc error: code = NotFound desc = could not find container \"79bda2629a22bcf1ce22d67a5f8309983baa31e0758f90fbb2361c669bc042f2\": container with ID starting with 79bda2629a22bcf1ce22d67a5f8309983baa31e0758f90fbb2361c669bc042f2 not found: ID does not exist" Oct 30 17:14:47.507108 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:47.507086 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ddc10d-55c9674d94-hwbjl"] Oct 30 17:14:47.510278 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:47.510254 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ddc10d-55c9674d94-hwbjl"] Oct 30 17:14:47.545616 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:47.545550 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ddc10d-55f45f45f-4dt5v"] Oct 30 17:14:47.549337 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:47.549308 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ddc10d-55f45f45f-4dt5v"] Oct 30 17:14:48.613109 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:48.613069 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6227d9b1-b976-4afd-ab68-c7e051272d11" path="/var/lib/kubelet/pods/6227d9b1-b976-4afd-ab68-c7e051272d11/volumes" Oct 30 17:14:48.613722 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:48.613702 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec94c66b-d7c7-43cb-8df7-e7cf32251abb" path="/var/lib/kubelet/pods/ec94c66b-d7c7-43cb-8df7-e7cf32251abb/volumes" Oct 30 17:14:48.841434 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:14:48.841394 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 17:14:55.602597 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:55.602552 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4def7e-6d5bf4d5c8-62jp6"] Oct 30 17:14:55.603190 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:55.602788 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-4def7e-6d5bf4d5c8-62jp6" podUID="358f6b76-be1e-474d-bf65-28c808a052ce" containerName="payload-processing" containerID="cri-o://8af1e80341c6fe659a966431b25a75bc29e36fed602850289582fc8cc9d7ef48" gracePeriod=30 Oct 30 17:14:55.911683 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:55.911653 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-4def7e-6d5bf4d5c8-62jp6" Oct 30 17:14:56.047610 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:56.047138 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/358f6b76-be1e-474d-bf65-28c808a052ce-kube-api-access-snj7t\" (UniqueName: \"kubernetes.io/projected/358f6b76-be1e-474d-bf65-28c808a052ce-kube-api-access-snj7t\") pod \"358f6b76-be1e-474d-bf65-28c808a052ce\" (UID: \"358f6b76-be1e-474d-bf65-28c808a052ce\") " Oct 30 17:14:56.047610 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:56.047386 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/358f6b76-be1e-474d-bf65-28c808a052ce-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/358f6b76-be1e-474d-bf65-28c808a052ce-plugins-config-volume\") pod \"358f6b76-be1e-474d-bf65-28c808a052ce\" (UID: \"358f6b76-be1e-474d-bf65-28c808a052ce\") " Oct 30 17:14:56.106184 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:56.085643 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/358f6b76-be1e-474d-bf65-28c808a052ce-plugins-config-volume" pod "358f6b76-be1e-474d-bf65-28c808a052ce" (UID: "358f6b76-be1e-474d-bf65-28c808a052ce"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:14:56.141359 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:56.141260 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/358f6b76-be1e-474d-bf65-28c808a052ce-kube-api-access-snj7t" pod "358f6b76-be1e-474d-bf65-28c808a052ce" (UID: "358f6b76-be1e-474d-bf65-28c808a052ce"). InnerVolumeSpecName "kube-api-access-snj7t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:14:56.149479 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:56.149441 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snj7t\" (UniqueName: \"kubernetes.io/projected/358f6b76-be1e-474d-bf65-28c808a052ce-kube-api-access-snj7t\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:14:56.149479 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:56.149481 5052 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/358f6b76-be1e-474d-bf65-28c808a052ce-plugins-config-volume\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:14:56.514304 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:56.514266 5052 generic.go:356] "Generic (PLEG): container finished" podID="358f6b76-be1e-474d-bf65-28c808a052ce" containerID="8af1e80341c6fe659a966431b25a75bc29e36fed602850289582fc8cc9d7ef48" exitCode=0 Oct 30 17:14:56.514526 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:56.514317 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-4def7e-6d5bf4d5c8-62jp6" Oct 30 17:14:56.514526 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:56.514351 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-4def7e-6d5bf4d5c8-62jp6" event={"ID":"358f6b76-be1e-474d-bf65-28c808a052ce","Type":"ContainerDied","Data":"8af1e80341c6fe659a966431b25a75bc29e36fed602850289582fc8cc9d7ef48"} Oct 30 17:14:56.514526 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:56.514416 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-4def7e-6d5bf4d5c8-62jp6" event={"ID":"358f6b76-be1e-474d-bf65-28c808a052ce","Type":"ContainerDied","Data":"f3fac603f8c1f585ddb9920dc3a0270c862a9539e28ece915db1aa9836eac682"} Oct 30 17:14:56.514526 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:56.514440 5052 scope.go:122] "RemoveContainer" containerID="8af1e80341c6fe659a966431b25a75bc29e36fed602850289582fc8cc9d7ef48" Oct 30 17:14:56.524948 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:56.524924 5052 scope.go:122] "RemoveContainer" containerID="8af1e80341c6fe659a966431b25a75bc29e36fed602850289582fc8cc9d7ef48" Oct 30 17:14:56.525236 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:14:56.525214 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8af1e80341c6fe659a966431b25a75bc29e36fed602850289582fc8cc9d7ef48\": container with ID starting with 8af1e80341c6fe659a966431b25a75bc29e36fed602850289582fc8cc9d7ef48 not found: ID does not exist" containerID="8af1e80341c6fe659a966431b25a75bc29e36fed602850289582fc8cc9d7ef48" Oct 30 17:14:56.525311 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:56.525247 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8af1e80341c6fe659a966431b25a75bc29e36fed602850289582fc8cc9d7ef48"} err="failed to get container status \"8af1e80341c6fe659a966431b25a75bc29e36fed602850289582fc8cc9d7ef48\": rpc error: code = NotFound desc = could not find container \"8af1e80341c6fe659a966431b25a75bc29e36fed602850289582fc8cc9d7ef48\": container with ID starting with 8af1e80341c6fe659a966431b25a75bc29e36fed602850289582fc8cc9d7ef48 not found: ID does not exist" Oct 30 17:14:56.571019 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:56.570990 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4def7e-6d5bf4d5c8-62jp6"] Oct 30 17:14:56.583355 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:56.583328 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4def7e-6d5bf4d5c8-62jp6"] Oct 30 17:14:56.599135 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:14:56.599108 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="358f6b76-be1e-474d-bf65-28c808a052ce" path="/var/lib/kubelet/pods/358f6b76-be1e-474d-bf65-28c808a052ce/volumes" Oct 30 17:15:00.196304 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:00.196265 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7c4b84-29889675-2mrhn"] Oct 30 17:15:00.197287 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:00.197244 5052 cpu_manager.go:414] "containerMap: removing container" podUID="051f70f5-3c9d-44e7-9d47-d5b21feeee6e" containerName="maas-api" Oct 30 17:15:00.197287 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:00.197270 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="051f70f5-3c9d-44e7-9d47-d5b21feeee6e" containerName="maas-api" Oct 30 17:15:00.197563 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:00.197292 5052 cpu_manager.go:414] "containerMap: removing container" podUID="1fb4a801-fd82-4036-ba89-79fdf32d3be3" containerName="istio-proxy" Oct 30 17:15:00.197563 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:00.197300 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fb4a801-fd82-4036-ba89-79fdf32d3be3" containerName="istio-proxy" Oct 30 17:15:00.197563 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:00.197311 5052 cpu_manager.go:414] "containerMap: removing container" podUID="1fb4a801-fd82-4036-ba89-79fdf32d3be3" containerName="istio-proxy" Oct 30 17:15:00.197563 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:00.197318 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fb4a801-fd82-4036-ba89-79fdf32d3be3" containerName="istio-proxy" Oct 30 17:15:00.197563 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:00.197340 5052 cpu_manager.go:414] "containerMap: removing container" podUID="ce11f38f-0ae0-4881-aa0f-346a0dd513c4" containerName="istio-proxy" Oct 30 17:15:00.197563 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:00.197347 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce11f38f-0ae0-4881-aa0f-346a0dd513c4" containerName="istio-proxy" Oct 30 17:15:00.197563 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:00.197373 5052 cpu_manager.go:414] "containerMap: removing container" podUID="358f6b76-be1e-474d-bf65-28c808a052ce" containerName="payload-processing" Oct 30 17:15:00.197563 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:00.197380 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="358f6b76-be1e-474d-bf65-28c808a052ce" containerName="payload-processing" Oct 30 17:15:00.197563 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:00.197394 5052 cpu_manager.go:414] "containerMap: removing container" podUID="ce11f38f-0ae0-4881-aa0f-346a0dd513c4" containerName="istio-proxy" Oct 30 17:15:00.197563 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:00.197402 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce11f38f-0ae0-4881-aa0f-346a0dd513c4" containerName="istio-proxy" Oct 30 17:15:00.197563 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:00.197414 5052 cpu_manager.go:414] "containerMap: removing container" podUID="8f6b9354-378f-4451-9ff7-be7f7fec12f2" containerName="main" Oct 30 17:15:00.197563 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:00.197421 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f6b9354-378f-4451-9ff7-be7f7fec12f2" containerName="main" Oct 30 17:15:00.197563 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:00.197431 5052 cpu_manager.go:414] "containerMap: removing container" podUID="6227d9b1-b976-4afd-ab68-c7e051272d11" containerName="payload-processing" Oct 30 17:15:00.197563 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:00.197438 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6227d9b1-b976-4afd-ab68-c7e051272d11" containerName="payload-processing" Oct 30 17:15:00.197563 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:00.197456 5052 cpu_manager.go:414] "containerMap: removing container" podUID="bd662f3a-659f-477c-b8d5-94c64288c0c1" containerName="main" Oct 30 17:15:00.197563 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:00.197463 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd662f3a-659f-477c-b8d5-94c64288c0c1" containerName="main" Oct 30 17:15:00.197563 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:00.197481 5052 cpu_manager.go:414] "containerMap: removing container" podUID="bc4d11ae-edd2-4143-95fd-b57bf4486e76" containerName="istio-proxy" Oct 30 17:15:00.197563 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:00.197489 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc4d11ae-edd2-4143-95fd-b57bf4486e76" containerName="istio-proxy" Oct 30 17:15:00.197563 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:00.197507 5052 cpu_manager.go:414] "containerMap: removing container" podUID="8451a193-e5be-47eb-9e5b-75c5214d3abc" containerName="istio-proxy" Oct 30 17:15:00.197563 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:00.197515 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8451a193-e5be-47eb-9e5b-75c5214d3abc" containerName="istio-proxy" Oct 30 17:15:00.197563 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:00.197554 5052 cpu_manager.go:414] "containerMap: removing container" podUID="523c6f22-3610-4d8f-a048-acb661bb23b5" containerName="istio-proxy" Oct 30 17:15:00.197563 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:00.197564 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="523c6f22-3610-4d8f-a048-acb661bb23b5" containerName="istio-proxy" Oct 30 17:15:00.198847 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:00.197611 5052 cpu_manager.go:414] "containerMap: removing container" podUID="025e9d83-1372-496c-b75e-4f5205d48dcd" containerName="istio-proxy" Oct 30 17:15:00.198847 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:00.197620 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="025e9d83-1372-496c-b75e-4f5205d48dcd" containerName="istio-proxy" Oct 30 17:15:00.198847 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:00.197634 5052 cpu_manager.go:414] "containerMap: removing container" podUID="ec94c66b-d7c7-43cb-8df7-e7cf32251abb" containerName="payload-pre-processing" Oct 30 17:15:00.198847 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:00.197642 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec94c66b-d7c7-43cb-8df7-e7cf32251abb" containerName="payload-pre-processing" Oct 30 17:15:00.198847 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:00.197800 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="051f70f5-3c9d-44e7-9d47-d5b21feeee6e" containerName="maas-api" Oct 30 17:15:00.198847 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:00.197818 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fb4a801-fd82-4036-ba89-79fdf32d3be3" containerName="istio-proxy" Oct 30 17:15:00.198847 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:00.197828 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fb4a801-fd82-4036-ba89-79fdf32d3be3" containerName="istio-proxy" Oct 30 17:15:00.198847 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:00.197842 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce11f38f-0ae0-4881-aa0f-346a0dd513c4" containerName="istio-proxy" Oct 30 17:15:00.198847 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:00.197854 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="358f6b76-be1e-474d-bf65-28c808a052ce" containerName="payload-processing" Oct 30 17:15:00.198847 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:00.197864 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f6b9354-378f-4451-9ff7-be7f7fec12f2" containerName="main" Oct 30 17:15:00.198847 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:00.197874 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="6227d9b1-b976-4afd-ab68-c7e051272d11" containerName="payload-processing" Oct 30 17:15:00.198847 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:00.197885 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd662f3a-659f-477c-b8d5-94c64288c0c1" containerName="main" Oct 30 17:15:00.198847 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:00.197894 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc4d11ae-edd2-4143-95fd-b57bf4486e76" containerName="istio-proxy" Oct 30 17:15:00.198847 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:00.197903 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="8451a193-e5be-47eb-9e5b-75c5214d3abc" containerName="istio-proxy" Oct 30 17:15:00.198847 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:00.197915 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="523c6f22-3610-4d8f-a048-acb661bb23b5" containerName="istio-proxy" Oct 30 17:15:00.198847 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:00.197928 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="025e9d83-1372-496c-b75e-4f5205d48dcd" containerName="istio-proxy" Oct 30 17:15:00.198847 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:00.197937 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec94c66b-d7c7-43cb-8df7-e7cf32251abb" containerName="payload-pre-processing" Oct 30 17:15:00.203368 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:00.203345 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7c4b84-29889675-2mrhn" Oct 30 17:15:00.211313 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:00.211268 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889675-d29kx"] Oct 30 17:15:00.212527 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:00.212503 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce11f38f-0ae0-4881-aa0f-346a0dd513c4" containerName="istio-proxy" Oct 30 17:15:00.221730 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:00.221705 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-263f00-29889675-2hrdn"] Oct 30 17:15:00.221855 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:00.221830 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889675-d29kx" Oct 30 17:15:00.227515 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:00.226614 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-263f00-29889675-2hrdn" Oct 30 17:15:00.227767 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:00.227723 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7c4b84-29889675-2mrhn"] Oct 30 17:15:00.245833 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:00.245803 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889675-d29kx"] Oct 30 17:15:00.252451 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:00.252422 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bfecdd-29889675-6rzrt"] Oct 30 17:15:00.267510 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:00.267477 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-263f00-29889675-2hrdn"] Oct 30 17:15:00.267694 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:00.267534 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bfecdd-29889675-6rzrt"] Oct 30 17:15:00.267774 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:00.267708 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bfecdd-29889675-6rzrt" Oct 30 17:15:00.274912 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:00.274888 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 17:15:00.311892 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:00.311866 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c5fe268-6035-4334-916e-b13bca739541-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-263f00-29889675-2hrdn\" (UID: \"1c5fe268-6035-4334-916e-b13bca739541\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-263f00-29889675-2hrdn" Oct 30 17:15:00.312020 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:00.311909 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkwgv\" (UniqueName: \"kubernetes.io/projected/1c5fe268-6035-4334-916e-b13bca739541-kube-api-access-tkwgv\") pod \"maas-api-key-cleanup-e2e-worker-w4-263f00-29889675-2hrdn\" (UID: \"1c5fe268-6035-4334-916e-b13bca739541\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-263f00-29889675-2hrdn" Oct 30 17:15:00.312020 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:00.311981 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdzv2\" (UniqueName: \"kubernetes.io/projected/4bcebabf-2dba-4645-b30d-6280c85ca947-kube-api-access-sdzv2\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-7c4b84-29889675-2mrhn\" (UID: \"4bcebabf-2dba-4645-b30d-6280c85ca947\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7c4b84-29889675-2mrhn" Oct 30 17:15:00.312113 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:00.312038 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbb4905a-78c4-4697-84c0-bd1a665ff034-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889675-d29kx\" (UID: \"cbb4905a-78c4-4697-84c0-bd1a665ff034\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889675-d29kx" Oct 30 17:15:00.312175 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:00.312146 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkvn6\" (UniqueName: \"kubernetes.io/projected/cbb4905a-78c4-4697-84c0-bd1a665ff034-kube-api-access-lkvn6\") pod \"maas-api-key-cleanup-29889675-d29kx\" (UID: \"cbb4905a-78c4-4697-84c0-bd1a665ff034\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889675-d29kx" Oct 30 17:15:00.312213 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:00.312197 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bcebabf-2dba-4645-b30d-6280c85ca947-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-7c4b84-29889675-2mrhn\" (UID: \"4bcebabf-2dba-4645-b30d-6280c85ca947\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7c4b84-29889675-2mrhn" Oct 30 17:15:00.413051 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:00.413021 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c5fe268-6035-4334-916e-b13bca739541-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-263f00-29889675-2hrdn\" (UID: \"1c5fe268-6035-4334-916e-b13bca739541\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-263f00-29889675-2hrdn" Oct 30 17:15:00.413216 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:00.413063 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkwgv\" (UniqueName: \"kubernetes.io/projected/1c5fe268-6035-4334-916e-b13bca739541-kube-api-access-tkwgv\") pod \"maas-api-key-cleanup-e2e-worker-w4-263f00-29889675-2hrdn\" (UID: \"1c5fe268-6035-4334-916e-b13bca739541\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-263f00-29889675-2hrdn" Oct 30 17:15:00.413216 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:00.413090 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdzv2\" (UniqueName: \"kubernetes.io/projected/4bcebabf-2dba-4645-b30d-6280c85ca947-kube-api-access-sdzv2\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-7c4b84-29889675-2mrhn\" (UID: \"4bcebabf-2dba-4645-b30d-6280c85ca947\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7c4b84-29889675-2mrhn" Oct 30 17:15:00.413216 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:00.413116 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbb4905a-78c4-4697-84c0-bd1a665ff034-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889675-d29kx\" (UID: \"cbb4905a-78c4-4697-84c0-bd1a665ff034\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889675-d29kx" Oct 30 17:15:00.413216 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:00.413157 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8dk9\" (UniqueName: \"kubernetes.io/projected/7c8c2e2b-b5e5-4e6c-ace8-db859e693dde-kube-api-access-m8dk9\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-bfecdd-29889675-6rzrt\" (UID: \"7c8c2e2b-b5e5-4e6c-ace8-db859e693dde\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bfecdd-29889675-6rzrt" Oct 30 17:15:00.413386 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:00.413239 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkvn6\" (UniqueName: \"kubernetes.io/projected/cbb4905a-78c4-4697-84c0-bd1a665ff034-kube-api-access-lkvn6\") pod \"maas-api-key-cleanup-29889675-d29kx\" (UID: \"cbb4905a-78c4-4697-84c0-bd1a665ff034\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889675-d29kx" Oct 30 17:15:00.413386 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:00.413257 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bcebabf-2dba-4645-b30d-6280c85ca947-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-7c4b84-29889675-2mrhn\" (UID: \"4bcebabf-2dba-4645-b30d-6280c85ca947\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7c4b84-29889675-2mrhn" Oct 30 17:15:00.413386 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:00.413370 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c8c2e2b-b5e5-4e6c-ace8-db859e693dde-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-bfecdd-29889675-6rzrt\" (UID: \"7c8c2e2b-b5e5-4e6c-ace8-db859e693dde\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bfecdd-29889675-6rzrt" Oct 30 17:15:00.413798 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:00.413778 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c5fe268-6035-4334-916e-b13bca739541-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-263f00-29889675-2hrdn\" (UID: \"1c5fe268-6035-4334-916e-b13bca739541\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-263f00-29889675-2hrdn" Oct 30 17:15:00.413967 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:00.413943 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbb4905a-78c4-4697-84c0-bd1a665ff034-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889675-d29kx\" (UID: \"cbb4905a-78c4-4697-84c0-bd1a665ff034\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889675-d29kx" Oct 30 17:15:00.414026 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:00.414010 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bcebabf-2dba-4645-b30d-6280c85ca947-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-7c4b84-29889675-2mrhn\" (UID: \"4bcebabf-2dba-4645-b30d-6280c85ca947\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7c4b84-29889675-2mrhn" Oct 30 17:15:00.438050 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:00.438025 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkvn6\" (UniqueName: \"kubernetes.io/projected/cbb4905a-78c4-4697-84c0-bd1a665ff034-kube-api-access-lkvn6\") pod \"maas-api-key-cleanup-29889675-d29kx\" (UID: \"cbb4905a-78c4-4697-84c0-bd1a665ff034\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889675-d29kx" Oct 30 17:15:00.438182 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:00.438032 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdzv2\" (UniqueName: \"kubernetes.io/projected/4bcebabf-2dba-4645-b30d-6280c85ca947-kube-api-access-sdzv2\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-7c4b84-29889675-2mrhn\" (UID: \"4bcebabf-2dba-4645-b30d-6280c85ca947\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7c4b84-29889675-2mrhn" Oct 30 17:15:00.438182 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:00.438060 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkwgv\" (UniqueName: \"kubernetes.io/projected/1c5fe268-6035-4334-916e-b13bca739541-kube-api-access-tkwgv\") pod \"maas-api-key-cleanup-e2e-worker-w4-263f00-29889675-2hrdn\" (UID: \"1c5fe268-6035-4334-916e-b13bca739541\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-263f00-29889675-2hrdn" Oct 30 17:15:00.515285 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:00.515209 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c8c2e2b-b5e5-4e6c-ace8-db859e693dde-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-bfecdd-29889675-6rzrt\" (UID: \"7c8c2e2b-b5e5-4e6c-ace8-db859e693dde\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bfecdd-29889675-6rzrt" Oct 30 17:15:00.515422 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:00.515384 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8dk9\" (UniqueName: \"kubernetes.io/projected/7c8c2e2b-b5e5-4e6c-ace8-db859e693dde-kube-api-access-m8dk9\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-bfecdd-29889675-6rzrt\" (UID: \"7c8c2e2b-b5e5-4e6c-ace8-db859e693dde\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bfecdd-29889675-6rzrt" Oct 30 17:15:00.515906 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:00.515880 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c8c2e2b-b5e5-4e6c-ace8-db859e693dde-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-bfecdd-29889675-6rzrt\" (UID: \"7c8c2e2b-b5e5-4e6c-ace8-db859e693dde\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bfecdd-29889675-6rzrt" Oct 30 17:15:00.539789 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:00.539756 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8dk9\" (UniqueName: \"kubernetes.io/projected/7c8c2e2b-b5e5-4e6c-ace8-db859e693dde-kube-api-access-m8dk9\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-bfecdd-29889675-6rzrt\" (UID: \"7c8c2e2b-b5e5-4e6c-ace8-db859e693dde\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bfecdd-29889675-6rzrt" Oct 30 17:15:00.576606 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:00.575801 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7c4b84-29889675-2mrhn" Oct 30 17:15:00.588015 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:00.587975 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889675-d29kx" Oct 30 17:15:00.594287 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:00.594257 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-263f00-29889675-2hrdn" Oct 30 17:15:00.605617 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:00.601887 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bfecdd-29889675-6rzrt" Oct 30 17:15:00.938574 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:00.938546 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7c4b84-29889675-2mrhn"] Oct 30 17:15:00.941551 ip-10-0-131-136 kubenswrapper[5052]: W1030 17:15:00.941502 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bcebabf_2dba_4645_b30d_6280c85ca947.slice/crio-01ee1299d1e89bc1ea6f732300b258ca9458cfe538903feffa9c17878816fbde WatchSource:0}: Error finding container 01ee1299d1e89bc1ea6f732300b258ca9458cfe538903feffa9c17878816fbde: Status 404 returned error can't find the container with id 01ee1299d1e89bc1ea6f732300b258ca9458cfe538903feffa9c17878816fbde Oct 30 17:15:00.945913 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:00.945892 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889675-d29kx"] Oct 30 17:15:00.949082 ip-10-0-131-136 kubenswrapper[5052]: W1030 17:15:00.949058 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbb4905a_78c4_4697_84c0_bd1a665ff034.slice/crio-4029a3cdf41203e651007df9c052ab56b9cd1ff06814a2a990fb57cc793e9ec9 WatchSource:0}: Error finding container 4029a3cdf41203e651007df9c052ab56b9cd1ff06814a2a990fb57cc793e9ec9: Status 404 returned error can't find the container with id 4029a3cdf41203e651007df9c052ab56b9cd1ff06814a2a990fb57cc793e9ec9 Oct 30 17:15:00.957143 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:00.957119 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bfecdd-29889675-6rzrt"] Oct 30 17:15:00.959738 ip-10-0-131-136 kubenswrapper[5052]: W1030 17:15:00.959719 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c8c2e2b_b5e5_4e6c_ace8_db859e693dde.slice/crio-a681e7e4b014f0eb468008c758a280f94fe8830d7502a841e3f4fa98f7b16d0d WatchSource:0}: Error finding container a681e7e4b014f0eb468008c758a280f94fe8830d7502a841e3f4fa98f7b16d0d: Status 404 returned error can't find the container with id a681e7e4b014f0eb468008c758a280f94fe8830d7502a841e3f4fa98f7b16d0d Oct 30 17:15:00.961227 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:00.961208 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-263f00-29889675-2hrdn"] Oct 30 17:15:00.961937 ip-10-0-131-136 kubenswrapper[5052]: W1030 17:15:00.961918 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c5fe268_6035_4334_916e_b13bca739541.slice/crio-1e86cdc12d90f38b97e644938c732e9bd8e093309545d2d9d2b962965df6f06b WatchSource:0}: Error finding container 1e86cdc12d90f38b97e644938c732e9bd8e093309545d2d9d2b962965df6f06b: Status 404 returned error can't find the container with id 1e86cdc12d90f38b97e644938c732e9bd8e093309545d2d9d2b962965df6f06b Oct 30 17:15:01.586087 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:01.577340 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889675-d29kx" event={"ID":"cbb4905a-78c4-4697-84c0-bd1a665ff034","Type":"ContainerStarted","Data":"4029a3cdf41203e651007df9c052ab56b9cd1ff06814a2a990fb57cc793e9ec9"} Oct 30 17:15:01.586087 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:01.584628 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-263f00-29889675-2hrdn" event={"ID":"1c5fe268-6035-4334-916e-b13bca739541","Type":"ContainerStarted","Data":"08c50cb7306f9efbe7f8789269ee7cbf05eb74c7a7ab4a0b59d0507618e663ef"} Oct 30 17:15:01.586087 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:01.584714 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-263f00-29889675-2hrdn" event={"ID":"1c5fe268-6035-4334-916e-b13bca739541","Type":"ContainerStarted","Data":"1e86cdc12d90f38b97e644938c732e9bd8e093309545d2d9d2b962965df6f06b"} Oct 30 17:15:01.586087 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:01.584740 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bfecdd-29889675-6rzrt" event={"ID":"7c8c2e2b-b5e5-4e6c-ace8-db859e693dde","Type":"ContainerStarted","Data":"a681e7e4b014f0eb468008c758a280f94fe8830d7502a841e3f4fa98f7b16d0d"} Oct 30 17:15:01.588881 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:01.588799 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7c4b84-29889675-2mrhn" event={"ID":"4bcebabf-2dba-4645-b30d-6280c85ca947","Type":"ContainerStarted","Data":"3783a1296c582e1f6532302d73b4c56ba16b310d784c1411cc6ebb9e44034457"} Oct 30 17:15:01.589046 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:01.588908 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7c4b84-29889675-2mrhn" event={"ID":"4bcebabf-2dba-4645-b30d-6280c85ca947","Type":"ContainerStarted","Data":"01ee1299d1e89bc1ea6f732300b258ca9458cfe538903feffa9c17878816fbde"} Oct 30 17:15:01.631242 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:01.631180 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-263f00-29889675-2hrdn" podStartSLOduration=1.631164793 podStartE2EDuration="1.631164793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:15:01.629344042 +0000 UTC m=+3257.156679261" watchObservedRunningTime="2026-10-30 17:15:01.631164793 +0000 UTC m=+3257.158500012" Oct 30 17:15:01.672933 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:01.672881 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7c4b84-29889675-2mrhn" podStartSLOduration=1.6728686750000001 podStartE2EDuration="1.672868675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:15:01.670739844 +0000 UTC m=+3257.198075064" watchObservedRunningTime="2026-10-30 17:15:01.672868675 +0000 UTC m=+3257.200203894" Oct 30 17:15:02.712607 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:02.712401 5052 generic.go:356] "Generic (PLEG): container finished" podID="1c5fe268-6035-4334-916e-b13bca739541" containerID="08c50cb7306f9efbe7f8789269ee7cbf05eb74c7a7ab4a0b59d0507618e663ef" exitCode=60 Oct 30 17:15:02.712607 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:02.712507 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-263f00-29889675-2hrdn" event={"ID":"1c5fe268-6035-4334-916e-b13bca739541","Type":"ContainerDied","Data":"08c50cb7306f9efbe7f8789269ee7cbf05eb74c7a7ab4a0b59d0507618e663ef"} Oct 30 17:15:02.713191 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:02.713077 5052 scope.go:122] "RemoveContainer" containerID="08c50cb7306f9efbe7f8789269ee7cbf05eb74c7a7ab4a0b59d0507618e663ef" Oct 30 17:15:02.761658 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:02.761626 5052 generic.go:356] "Generic (PLEG): container finished" podID="7c8c2e2b-b5e5-4e6c-ace8-db859e693dde" containerID="b50f840d8a9c5b76e4057b802741e55fb31b31b2274d5c1a093002bbcdcdefea" exitCode=60 Oct 30 17:15:02.762423 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:02.762048 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bfecdd-29889675-6rzrt" event={"ID":"7c8c2e2b-b5e5-4e6c-ace8-db859e693dde","Type":"ContainerDied","Data":"b50f840d8a9c5b76e4057b802741e55fb31b31b2274d5c1a093002bbcdcdefea"} Oct 30 17:15:02.762613 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:02.762216 5052 scope.go:122] "RemoveContainer" containerID="b50f840d8a9c5b76e4057b802741e55fb31b31b2274d5c1a093002bbcdcdefea" Oct 30 17:15:02.767846 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:02.767818 5052 generic.go:356] "Generic (PLEG): container finished" podID="4bcebabf-2dba-4645-b30d-6280c85ca947" containerID="3783a1296c582e1f6532302d73b4c56ba16b310d784c1411cc6ebb9e44034457" exitCode=60 Oct 30 17:15:02.767984 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:02.767920 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7c4b84-29889675-2mrhn" event={"ID":"4bcebabf-2dba-4645-b30d-6280c85ca947","Type":"ContainerDied","Data":"3783a1296c582e1f6532302d73b4c56ba16b310d784c1411cc6ebb9e44034457"} Oct 30 17:15:02.768302 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:02.768288 5052 scope.go:122] "RemoveContainer" containerID="3783a1296c582e1f6532302d73b4c56ba16b310d784c1411cc6ebb9e44034457" Oct 30 17:15:02.771638 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:02.771611 5052 generic.go:356] "Generic (PLEG): container finished" podID="cbb4905a-78c4-4697-84c0-bd1a665ff034" containerID="2ab10670d3f4ba205335ead304e2c1bb6553f25260f50f3601e31290d9caa337" exitCode=60 Oct 30 17:15:02.771746 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:02.771674 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889675-d29kx" event={"ID":"cbb4905a-78c4-4697-84c0-bd1a665ff034","Type":"ContainerDied","Data":"2ab10670d3f4ba205335ead304e2c1bb6553f25260f50f3601e31290d9caa337"} Oct 30 17:15:02.772355 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:02.772332 5052 scope.go:122] "RemoveContainer" containerID="2ab10670d3f4ba205335ead304e2c1bb6553f25260f50f3601e31290d9caa337" Oct 30 17:15:03.781468 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:03.781418 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-263f00-29889675-2hrdn" event={"ID":"1c5fe268-6035-4334-916e-b13bca739541","Type":"ContainerStarted","Data":"99e46d04ff0046e0d7df60b6e3810826794c9e6365bca386f436e15480a73818"} Oct 30 17:15:03.783633 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:03.783603 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bfecdd-29889675-6rzrt" event={"ID":"7c8c2e2b-b5e5-4e6c-ace8-db859e693dde","Type":"ContainerStarted","Data":"34e7982ee1236758e886805457bc2cbd3e5e35fd441ce4ec113498a15fc5a958"} Oct 30 17:15:03.785519 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:03.785495 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7c4b84-29889675-2mrhn" event={"ID":"4bcebabf-2dba-4645-b30d-6280c85ca947","Type":"ContainerStarted","Data":"1482a7fc04a0d93366e2eac59d08795e5aa3c69b802f8adcadd89d63468d826a"} Oct 30 17:15:03.787361 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:03.787339 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889675-d29kx" event={"ID":"cbb4905a-78c4-4697-84c0-bd1a665ff034","Type":"ContainerStarted","Data":"db5868ff1ebe552152d3a736e979f27c4b1f706aa6201c869c312a35af226ca4"} Oct 30 17:15:03.911938 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:03.911889 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889675-d29kx" podStartSLOduration=3.911875132 podStartE2EDuration="3.911875132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:15:03.909188802 +0000 UTC m=+3259.436524025" watchObservedRunningTime="2026-10-30 17:15:03.911875132 +0000 UTC m=+3259.439210394" Oct 30 17:15:03.955382 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:03.955336 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bfecdd-29889675-6rzrt" podStartSLOduration=3.955322847 podStartE2EDuration="3.955322847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:15:03.953148701 +0000 UTC m=+3259.480483915" watchObservedRunningTime="2026-10-30 17:15:03.955322847 +0000 UTC m=+3259.482658106" Oct 30 17:15:04.797157 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:04.797123 5052 generic.go:356] "Generic (PLEG): container finished" podID="4bcebabf-2dba-4645-b30d-6280c85ca947" containerID="1482a7fc04a0d93366e2eac59d08795e5aa3c69b802f8adcadd89d63468d826a" exitCode=60 Oct 30 17:15:04.797623 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:04.797213 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7c4b84-29889675-2mrhn" event={"ID":"4bcebabf-2dba-4645-b30d-6280c85ca947","Type":"ContainerDied","Data":"1482a7fc04a0d93366e2eac59d08795e5aa3c69b802f8adcadd89d63468d826a"} Oct 30 17:15:04.797623 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:04.797288 5052 scope.go:122] "RemoveContainer" containerID="3783a1296c582e1f6532302d73b4c56ba16b310d784c1411cc6ebb9e44034457" Oct 30 17:15:04.797623 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:04.797504 5052 scope.go:122] "RemoveContainer" containerID="1482a7fc04a0d93366e2eac59d08795e5aa3c69b802f8adcadd89d63468d826a" Oct 30 17:15:04.797885 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:15:04.797863 5052 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-7c4b84-29889675-2mrhn_odh-ai-gateway-infra(4bcebabf-2dba-4645-b30d-6280c85ca947)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7c4b84-29889675-2mrhn" podUID="4bcebabf-2dba-4645-b30d-6280c85ca947" Oct 30 17:15:04.799897 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:04.799872 5052 generic.go:356] "Generic (PLEG): container finished" podID="cbb4905a-78c4-4697-84c0-bd1a665ff034" containerID="db5868ff1ebe552152d3a736e979f27c4b1f706aa6201c869c312a35af226ca4" exitCode=60 Oct 30 17:15:04.800004 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:04.799956 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889675-d29kx" event={"ID":"cbb4905a-78c4-4697-84c0-bd1a665ff034","Type":"ContainerDied","Data":"db5868ff1ebe552152d3a736e979f27c4b1f706aa6201c869c312a35af226ca4"} Oct 30 17:15:04.800193 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:04.800179 5052 scope.go:122] "RemoveContainer" containerID="db5868ff1ebe552152d3a736e979f27c4b1f706aa6201c869c312a35af226ca4" Oct 30 17:15:04.800518 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:15:04.800494 5052 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889675-d29kx_odh-ai-gateway-infra(cbb4905a-78c4-4697-84c0-bd1a665ff034)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889675-d29kx" podUID="cbb4905a-78c4-4697-84c0-bd1a665ff034" Oct 30 17:15:04.801893 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:04.801870 5052 generic.go:356] "Generic (PLEG): container finished" podID="1c5fe268-6035-4334-916e-b13bca739541" containerID="99e46d04ff0046e0d7df60b6e3810826794c9e6365bca386f436e15480a73818" exitCode=60 Oct 30 17:15:04.802012 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:04.801950 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-263f00-29889675-2hrdn" event={"ID":"1c5fe268-6035-4334-916e-b13bca739541","Type":"ContainerDied","Data":"99e46d04ff0046e0d7df60b6e3810826794c9e6365bca386f436e15480a73818"} Oct 30 17:15:04.802253 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:04.802235 5052 scope.go:122] "RemoveContainer" containerID="99e46d04ff0046e0d7df60b6e3810826794c9e6365bca386f436e15480a73818" Oct 30 17:15:04.802554 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:15:04.802527 5052 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-263f00-29889675-2hrdn_odh-ai-gateway-infra(1c5fe268-6035-4334-916e-b13bca739541)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-263f00-29889675-2hrdn" podUID="1c5fe268-6035-4334-916e-b13bca739541" Oct 30 17:15:04.803973 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:04.803948 5052 generic.go:356] "Generic (PLEG): container finished" podID="7c8c2e2b-b5e5-4e6c-ace8-db859e693dde" containerID="34e7982ee1236758e886805457bc2cbd3e5e35fd441ce4ec113498a15fc5a958" exitCode=60 Oct 30 17:15:04.804063 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:04.804018 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bfecdd-29889675-6rzrt" event={"ID":"7c8c2e2b-b5e5-4e6c-ace8-db859e693dde","Type":"ContainerDied","Data":"34e7982ee1236758e886805457bc2cbd3e5e35fd441ce4ec113498a15fc5a958"} Oct 30 17:15:04.804185 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:04.804172 5052 scope.go:122] "RemoveContainer" containerID="34e7982ee1236758e886805457bc2cbd3e5e35fd441ce4ec113498a15fc5a958" Oct 30 17:15:04.804433 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:15:04.804416 5052 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-bfecdd-29889675-6rzrt_odh-ai-gateway-infra(7c8c2e2b-b5e5-4e6c-ace8-db859e693dde)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bfecdd-29889675-6rzrt" podUID="7c8c2e2b-b5e5-4e6c-ace8-db859e693dde" Oct 30 17:15:04.812033 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:04.812018 5052 scope.go:122] "RemoveContainer" containerID="2ab10670d3f4ba205335ead304e2c1bb6553f25260f50f3601e31290d9caa337" Oct 30 17:15:04.826013 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:04.825992 5052 scope.go:122] "RemoveContainer" containerID="08c50cb7306f9efbe7f8789269ee7cbf05eb74c7a7ab4a0b59d0507618e663ef" Oct 30 17:15:04.842268 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:04.842246 5052 scope.go:122] "RemoveContainer" containerID="b50f840d8a9c5b76e4057b802741e55fb31b31b2274d5c1a093002bbcdcdefea" Oct 30 17:15:05.130094 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:05.130010 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-4ac125-openshift-default-6569d9988c-4zscj"] Oct 30 17:15:05.130296 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:05.130267 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-4ac125-openshift-default-6569d9988c-4zscj" podUID="4be3cd3f-d7d6-40ec-ad72-8012cfbdb946" containerName="istio-proxy" containerID="cri-o://f1c6f98407745add1f293feea3bee17bc861c368fe6984268cec2cd76aac0adf" gracePeriod=30 Oct 30 17:15:05.913651 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:05.903996 5052 scope.go:122] "RemoveContainer" containerID="99e46d04ff0046e0d7df60b6e3810826794c9e6365bca386f436e15480a73818" Oct 30 17:15:05.913651 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:15:05.904395 5052 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-263f00-29889675-2hrdn_odh-ai-gateway-infra(1c5fe268-6035-4334-916e-b13bca739541)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-263f00-29889675-2hrdn" podUID="1c5fe268-6035-4334-916e-b13bca739541" Oct 30 17:15:05.913651 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:05.907453 5052 scope.go:122] "RemoveContainer" containerID="34e7982ee1236758e886805457bc2cbd3e5e35fd441ce4ec113498a15fc5a958" Oct 30 17:15:05.913651 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:15:05.907830 5052 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-bfecdd-29889675-6rzrt_odh-ai-gateway-infra(7c8c2e2b-b5e5-4e6c-ace8-db859e693dde)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bfecdd-29889675-6rzrt" podUID="7c8c2e2b-b5e5-4e6c-ace8-db859e693dde" Oct 30 17:15:05.913651 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:05.910047 5052 scope.go:122] "RemoveContainer" containerID="1482a7fc04a0d93366e2eac59d08795e5aa3c69b802f8adcadd89d63468d826a" Oct 30 17:15:05.913651 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:15:05.910339 5052 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-7c4b84-29889675-2mrhn_odh-ai-gateway-infra(4bcebabf-2dba-4645-b30d-6280c85ca947)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7c4b84-29889675-2mrhn" podUID="4bcebabf-2dba-4645-b30d-6280c85ca947" Oct 30 17:15:05.913651 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:05.912651 5052 scope.go:122] "RemoveContainer" containerID="db5868ff1ebe552152d3a736e979f27c4b1f706aa6201c869c312a35af226ca4" Oct 30 17:15:05.913651 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:15:05.912913 5052 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889675-d29kx_odh-ai-gateway-infra(cbb4905a-78c4-4697-84c0-bd1a665ff034)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889675-d29kx" podUID="cbb4905a-78c4-4697-84c0-bd1a665ff034" Oct 30 17:15:06.136963 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:06.136928 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-dc74d5-slj4g"] Oct 30 17:15:06.142681 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:06.142660 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-dc74d5-slj4g" Oct 30 17:15:06.160915 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:06.160882 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-dc74d5-slj4g"] Oct 30 17:15:06.164775 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:06.164719 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:15:06.279388 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:06.279353 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3cf66546-2340-4526-b4e2-9aebde2a705b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-dc74d5-slj4g\" (UID: \"3cf66546-2340-4526-b4e2-9aebde2a705b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-dc74d5-slj4g" Oct 30 17:15:06.279571 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:06.279481 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7c5fx\" (UniqueName: \"kubernetes.io/projected/3cf66546-2340-4526-b4e2-9aebde2a705b-kube-api-access-7c5fx\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-dc74d5-slj4g\" (UID: \"3cf66546-2340-4526-b4e2-9aebde2a705b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-dc74d5-slj4g" Oct 30 17:15:06.279696 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:06.279646 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3cf66546-2340-4526-b4e2-9aebde2a705b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-dc74d5-slj4g\" (UID: \"3cf66546-2340-4526-b4e2-9aebde2a705b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-dc74d5-slj4g" Oct 30 17:15:06.392066 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:06.390674 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3cf66546-2340-4526-b4e2-9aebde2a705b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-dc74d5-slj4g\" (UID: \"3cf66546-2340-4526-b4e2-9aebde2a705b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-dc74d5-slj4g" Oct 30 17:15:06.392066 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:06.390889 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c5fx\" (UniqueName: \"kubernetes.io/projected/3cf66546-2340-4526-b4e2-9aebde2a705b-kube-api-access-7c5fx\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-dc74d5-slj4g\" (UID: \"3cf66546-2340-4526-b4e2-9aebde2a705b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-dc74d5-slj4g" Oct 30 17:15:06.392066 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:06.390985 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3cf66546-2340-4526-b4e2-9aebde2a705b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-dc74d5-slj4g\" (UID: \"3cf66546-2340-4526-b4e2-9aebde2a705b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-dc74d5-slj4g" Oct 30 17:15:06.392066 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:06.391958 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3cf66546-2340-4526-b4e2-9aebde2a705b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-dc74d5-slj4g\" (UID: \"3cf66546-2340-4526-b4e2-9aebde2a705b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-dc74d5-slj4g" Oct 30 17:15:06.394411 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:06.394379 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3cf66546-2340-4526-b4e2-9aebde2a705b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-dc74d5-slj4g\" (UID: \"3cf66546-2340-4526-b4e2-9aebde2a705b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-dc74d5-slj4g" Oct 30 17:15:06.418107 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:06.418046 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7c5fx\" (UniqueName: \"kubernetes.io/projected/3cf66546-2340-4526-b4e2-9aebde2a705b-kube-api-access-7c5fx\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-dc74d5-slj4g\" (UID: \"3cf66546-2340-4526-b4e2-9aebde2a705b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-dc74d5-slj4g" Oct 30 17:15:06.456938 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:06.456913 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-dc74d5-slj4g" Oct 30 17:15:06.771216 ip-10-0-131-136 kubenswrapper[5052]: W1030 17:15:06.771183 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cf66546_2340_4526_b4e2_9aebde2a705b.slice/crio-ec87c4a74dfe6adecb86939b51a38916f6f1516876dde29dc4a1d984b5d45c1b WatchSource:0}: Error finding container ec87c4a74dfe6adecb86939b51a38916f6f1516876dde29dc4a1d984b5d45c1b: Status 404 returned error can't find the container with id ec87c4a74dfe6adecb86939b51a38916f6f1516876dde29dc4a1d984b5d45c1b Oct 30 17:15:06.771318 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:06.771263 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-dc74d5-slj4g"] Oct 30 17:15:06.921720 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:06.921686 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-dc74d5-slj4g" event={"ID":"3cf66546-2340-4526-b4e2-9aebde2a705b","Type":"ContainerStarted","Data":"ec87c4a74dfe6adecb86939b51a38916f6f1516876dde29dc4a1d984b5d45c1b"} Oct 30 17:15:07.930220 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:07.930179 5052 generic.go:356] "Generic (PLEG): container finished" podID="3cf66546-2340-4526-b4e2-9aebde2a705b" containerID="02ff36057c33cfa54424bd9203d4db063cde5cb59d199c726a5691d1b31b430a" exitCode=0 Oct 30 17:15:07.930627 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:07.930268 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-dc74d5-slj4g" event={"ID":"3cf66546-2340-4526-b4e2-9aebde2a705b","Type":"ContainerDied","Data":"02ff36057c33cfa54424bd9203d4db063cde5cb59d199c726a5691d1b31b430a"} Oct 30 17:15:09.097631 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:09.097597 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-dc74d5-slj4g" Oct 30 17:15:09.157178 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:09.157150 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3cf66546-2340-4526-b4e2-9aebde2a705b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3cf66546-2340-4526-b4e2-9aebde2a705b-maas-api-openshift-ca\") pod \"3cf66546-2340-4526-b4e2-9aebde2a705b\" (UID: \"3cf66546-2340-4526-b4e2-9aebde2a705b\") " Oct 30 17:15:09.157424 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:09.157408 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3cf66546-2340-4526-b4e2-9aebde2a705b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3cf66546-2340-4526-b4e2-9aebde2a705b-maas-api-xks-ca\") pod \"3cf66546-2340-4526-b4e2-9aebde2a705b\" (UID: \"3cf66546-2340-4526-b4e2-9aebde2a705b\") " Oct 30 17:15:09.157508 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:09.157462 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cf66546-2340-4526-b4e2-9aebde2a705b-kube-api-access-7c5fx\" (UniqueName: \"kubernetes.io/projected/3cf66546-2340-4526-b4e2-9aebde2a705b-kube-api-access-7c5fx\") pod \"3cf66546-2340-4526-b4e2-9aebde2a705b\" (UID: \"3cf66546-2340-4526-b4e2-9aebde2a705b\") " Oct 30 17:15:09.157573 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:09.157520 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cf66546-2340-4526-b4e2-9aebde2a705b-maas-api-openshift-ca" pod "3cf66546-2340-4526-b4e2-9aebde2a705b" (UID: "3cf66546-2340-4526-b4e2-9aebde2a705b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:15:09.158121 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:09.158092 5052 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3cf66546-2340-4526-b4e2-9aebde2a705b-maas-api-openshift-ca\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:15:09.159705 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:09.159681 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3cf66546-2340-4526-b4e2-9aebde2a705b-maas-api-xks-ca" pod "3cf66546-2340-4526-b4e2-9aebde2a705b" (UID: "3cf66546-2340-4526-b4e2-9aebde2a705b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:15:09.159800 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:09.159773 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cf66546-2340-4526-b4e2-9aebde2a705b-kube-api-access-7c5fx" pod "3cf66546-2340-4526-b4e2-9aebde2a705b" (UID: "3cf66546-2340-4526-b4e2-9aebde2a705b"). InnerVolumeSpecName "kube-api-access-7c5fx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:09.259395 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:09.259315 5052 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3cf66546-2340-4526-b4e2-9aebde2a705b-maas-api-xks-ca\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:15:09.259395 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:09.259348 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7c5fx\" (UniqueName: \"kubernetes.io/projected/3cf66546-2340-4526-b4e2-9aebde2a705b-kube-api-access-7c5fx\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:15:09.945221 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:09.945178 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-dc74d5-slj4g" event={"ID":"3cf66546-2340-4526-b4e2-9aebde2a705b","Type":"ContainerDied","Data":"ec87c4a74dfe6adecb86939b51a38916f6f1516876dde29dc4a1d984b5d45c1b"} Oct 30 17:15:09.945221 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:09.945220 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-dc74d5-slj4g" Oct 30 17:15:09.945442 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:09.945272 5052 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec87c4a74dfe6adecb86939b51a38916f6f1516876dde29dc4a1d984b5d45c1b" Oct 30 17:15:10.420712 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:10.420688 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-4ac125-openshift-default-6569d9988c-4zscj" Oct 30 17:15:10.479779 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:10.479752 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4be3cd3f-d7d6-40ec-ad72-8012cfbdb946-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4be3cd3f-d7d6-40ec-ad72-8012cfbdb946-workload-socket\") pod \"4be3cd3f-d7d6-40ec-ad72-8012cfbdb946\" (UID: \"4be3cd3f-d7d6-40ec-ad72-8012cfbdb946\") " Oct 30 17:15:10.479899 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:10.479883 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4be3cd3f-d7d6-40ec-ad72-8012cfbdb946-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4be3cd3f-d7d6-40ec-ad72-8012cfbdb946-credential-socket\") pod \"4be3cd3f-d7d6-40ec-ad72-8012cfbdb946\" (UID: \"4be3cd3f-d7d6-40ec-ad72-8012cfbdb946\") " Oct 30 17:15:10.479988 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:10.479974 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4be3cd3f-d7d6-40ec-ad72-8012cfbdb946-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4be3cd3f-d7d6-40ec-ad72-8012cfbdb946-workload-certs\") pod \"4be3cd3f-d7d6-40ec-ad72-8012cfbdb946\" (UID: \"4be3cd3f-d7d6-40ec-ad72-8012cfbdb946\") " Oct 30 17:15:10.480076 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:10.480026 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4be3cd3f-d7d6-40ec-ad72-8012cfbdb946-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4be3cd3f-d7d6-40ec-ad72-8012cfbdb946-istiod-ca-cert\") pod \"4be3cd3f-d7d6-40ec-ad72-8012cfbdb946\" (UID: \"4be3cd3f-d7d6-40ec-ad72-8012cfbdb946\") " Oct 30 17:15:10.480076 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:10.480034 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4be3cd3f-d7d6-40ec-ad72-8012cfbdb946-workload-socket" pod "4be3cd3f-d7d6-40ec-ad72-8012cfbdb946" (UID: "4be3cd3f-d7d6-40ec-ad72-8012cfbdb946"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:15:10.480150 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:10.480132 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4be3cd3f-d7d6-40ec-ad72-8012cfbdb946-istio-token\" (UniqueName: \"kubernetes.io/projected/4be3cd3f-d7d6-40ec-ad72-8012cfbdb946-istio-token\") pod \"4be3cd3f-d7d6-40ec-ad72-8012cfbdb946\" (UID: \"4be3cd3f-d7d6-40ec-ad72-8012cfbdb946\") " Oct 30 17:15:10.480203 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:10.480186 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4be3cd3f-d7d6-40ec-ad72-8012cfbdb946-kube-api-access-jzdww\" (UniqueName: \"kubernetes.io/projected/4be3cd3f-d7d6-40ec-ad72-8012cfbdb946-kube-api-access-jzdww\") pod \"4be3cd3f-d7d6-40ec-ad72-8012cfbdb946\" (UID: \"4be3cd3f-d7d6-40ec-ad72-8012cfbdb946\") " Oct 30 17:15:10.480260 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:10.480200 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4be3cd3f-d7d6-40ec-ad72-8012cfbdb946-credential-socket" pod "4be3cd3f-d7d6-40ec-ad72-8012cfbdb946" (UID: "4be3cd3f-d7d6-40ec-ad72-8012cfbdb946"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:15:10.480260 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:10.480231 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4be3cd3f-d7d6-40ec-ad72-8012cfbdb946-workload-certs" pod "4be3cd3f-d7d6-40ec-ad72-8012cfbdb946" (UID: "4be3cd3f-d7d6-40ec-ad72-8012cfbdb946"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:15:10.480375 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:10.480257 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4be3cd3f-d7d6-40ec-ad72-8012cfbdb946-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4be3cd3f-d7d6-40ec-ad72-8012cfbdb946-istio-envoy\") pod \"4be3cd3f-d7d6-40ec-ad72-8012cfbdb946\" (UID: \"4be3cd3f-d7d6-40ec-ad72-8012cfbdb946\") " Oct 30 17:15:10.480375 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:10.480305 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4be3cd3f-d7d6-40ec-ad72-8012cfbdb946-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4be3cd3f-d7d6-40ec-ad72-8012cfbdb946-istio-podinfo\") pod \"4be3cd3f-d7d6-40ec-ad72-8012cfbdb946\" (UID: \"4be3cd3f-d7d6-40ec-ad72-8012cfbdb946\") " Oct 30 17:15:10.480375 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:10.480339 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4be3cd3f-d7d6-40ec-ad72-8012cfbdb946-istiod-ca-cert" pod "4be3cd3f-d7d6-40ec-ad72-8012cfbdb946" (UID: "4be3cd3f-d7d6-40ec-ad72-8012cfbdb946"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:15:10.480644 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:10.480380 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4be3cd3f-d7d6-40ec-ad72-8012cfbdb946-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4be3cd3f-d7d6-40ec-ad72-8012cfbdb946-istio-data\") pod \"4be3cd3f-d7d6-40ec-ad72-8012cfbdb946\" (UID: \"4be3cd3f-d7d6-40ec-ad72-8012cfbdb946\") " Oct 30 17:15:10.480839 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:10.480801 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4be3cd3f-d7d6-40ec-ad72-8012cfbdb946-istio-data" pod "4be3cd3f-d7d6-40ec-ad72-8012cfbdb946" (UID: "4be3cd3f-d7d6-40ec-ad72-8012cfbdb946"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:15:10.481354 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:10.481319 5052 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4be3cd3f-d7d6-40ec-ad72-8012cfbdb946-workload-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:15:10.481505 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:10.481374 5052 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4be3cd3f-d7d6-40ec-ad72-8012cfbdb946-istiod-ca-cert\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:15:10.481505 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:10.481388 5052 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4be3cd3f-d7d6-40ec-ad72-8012cfbdb946-istio-data\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:15:10.481923 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:10.481399 5052 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4be3cd3f-d7d6-40ec-ad72-8012cfbdb946-workload-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:15:10.482036 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:10.481939 5052 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4be3cd3f-d7d6-40ec-ad72-8012cfbdb946-credential-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:15:10.484631 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:10.484112 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4be3cd3f-d7d6-40ec-ad72-8012cfbdb946-istio-token" pod "4be3cd3f-d7d6-40ec-ad72-8012cfbdb946" (UID: "4be3cd3f-d7d6-40ec-ad72-8012cfbdb946"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:10.484631 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:10.484321 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4be3cd3f-d7d6-40ec-ad72-8012cfbdb946-kube-api-access-jzdww" pod "4be3cd3f-d7d6-40ec-ad72-8012cfbdb946" (UID: "4be3cd3f-d7d6-40ec-ad72-8012cfbdb946"). InnerVolumeSpecName "kube-api-access-jzdww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:10.484631 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:10.484340 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4be3cd3f-d7d6-40ec-ad72-8012cfbdb946-istio-podinfo" pod "4be3cd3f-d7d6-40ec-ad72-8012cfbdb946" (UID: "4be3cd3f-d7d6-40ec-ad72-8012cfbdb946"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:15:10.486679 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:10.486653 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4be3cd3f-d7d6-40ec-ad72-8012cfbdb946-istio-envoy" pod "4be3cd3f-d7d6-40ec-ad72-8012cfbdb946" (UID: "4be3cd3f-d7d6-40ec-ad72-8012cfbdb946"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:15:10.583791 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:10.583712 5052 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4be3cd3f-d7d6-40ec-ad72-8012cfbdb946-istio-token\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:15:10.583791 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:10.583740 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzdww\" (UniqueName: \"kubernetes.io/projected/4be3cd3f-d7d6-40ec-ad72-8012cfbdb946-kube-api-access-jzdww\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:15:10.583791 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:10.583750 5052 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4be3cd3f-d7d6-40ec-ad72-8012cfbdb946-istio-envoy\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:15:10.583791 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:10.583759 5052 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4be3cd3f-d7d6-40ec-ad72-8012cfbdb946-istio-podinfo\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:15:10.953714 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:10.953681 5052 generic.go:356] "Generic (PLEG): container finished" podID="4be3cd3f-d7d6-40ec-ad72-8012cfbdb946" containerID="f1c6f98407745add1f293feea3bee17bc861c368fe6984268cec2cd76aac0adf" exitCode=0 Oct 30 17:15:10.953899 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:10.953742 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-4ac125-openshift-default-6569d9988c-4zscj" Oct 30 17:15:10.953899 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:10.953762 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4ac125-openshift-default-6569d9988c-4zscj" event={"ID":"4be3cd3f-d7d6-40ec-ad72-8012cfbdb946","Type":"ContainerDied","Data":"f1c6f98407745add1f293feea3bee17bc861c368fe6984268cec2cd76aac0adf"} Oct 30 17:15:10.953899 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:10.953824 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4ac125-openshift-default-6569d9988c-4zscj" event={"ID":"4be3cd3f-d7d6-40ec-ad72-8012cfbdb946","Type":"ContainerDied","Data":"12a9f5d398806e9ec2ad61c84597181b7b4d04605d069ff2484c90dfb0643cdb"} Oct 30 17:15:10.953899 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:10.953855 5052 scope.go:122] "RemoveContainer" containerID="f1c6f98407745add1f293feea3bee17bc861c368fe6984268cec2cd76aac0adf" Oct 30 17:15:10.964277 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:10.964259 5052 scope.go:122] "RemoveContainer" containerID="81768e865fda8052bd1932d84fde94c0d357a5da7e2b3420fecccefb935956fc" Oct 30 17:15:10.973446 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:10.973427 5052 scope.go:122] "RemoveContainer" containerID="f1c6f98407745add1f293feea3bee17bc861c368fe6984268cec2cd76aac0adf" Oct 30 17:15:10.973723 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:15:10.973702 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1c6f98407745add1f293feea3bee17bc861c368fe6984268cec2cd76aac0adf\": container with ID starting with f1c6f98407745add1f293feea3bee17bc861c368fe6984268cec2cd76aac0adf not found: ID does not exist" containerID="f1c6f98407745add1f293feea3bee17bc861c368fe6984268cec2cd76aac0adf" Oct 30 17:15:10.973790 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:10.973731 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1c6f98407745add1f293feea3bee17bc861c368fe6984268cec2cd76aac0adf"} err="failed to get container status \"f1c6f98407745add1f293feea3bee17bc861c368fe6984268cec2cd76aac0adf\": rpc error: code = NotFound desc = could not find container \"f1c6f98407745add1f293feea3bee17bc861c368fe6984268cec2cd76aac0adf\": container with ID starting with f1c6f98407745add1f293feea3bee17bc861c368fe6984268cec2cd76aac0adf not found: ID does not exist" Oct 30 17:15:10.973790 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:10.973752 5052 scope.go:122] "RemoveContainer" containerID="81768e865fda8052bd1932d84fde94c0d357a5da7e2b3420fecccefb935956fc" Oct 30 17:15:10.973984 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:15:10.973966 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81768e865fda8052bd1932d84fde94c0d357a5da7e2b3420fecccefb935956fc\": container with ID starting with 81768e865fda8052bd1932d84fde94c0d357a5da7e2b3420fecccefb935956fc not found: ID does not exist" containerID="81768e865fda8052bd1932d84fde94c0d357a5da7e2b3420fecccefb935956fc" Oct 30 17:15:10.974037 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:10.973989 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81768e865fda8052bd1932d84fde94c0d357a5da7e2b3420fecccefb935956fc"} err="failed to get container status \"81768e865fda8052bd1932d84fde94c0d357a5da7e2b3420fecccefb935956fc\": rpc error: code = NotFound desc = could not find container \"81768e865fda8052bd1932d84fde94c0d357a5da7e2b3420fecccefb935956fc\": container with ID starting with 81768e865fda8052bd1932d84fde94c0d357a5da7e2b3420fecccefb935956fc not found: ID does not exist" Oct 30 17:15:11.026145 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:11.026107 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-4ac125-openshift-default-6569d9988c-4zscj"] Oct 30 17:15:11.040776 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:11.040739 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-4ac125-openshift-default-6569d9988c-4zscj"] Oct 30 17:15:11.837746 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:11.837673 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-661509-openshift-default-b5d9994f-xlcqt"] Oct 30 17:15:11.838125 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:11.837860 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-661509-openshift-default-b5d9994f-xlcqt" podUID="24bfdf71-2df9-478f-a6af-eca34fc212ca" containerName="istio-proxy" containerID="cri-o://3795d7d00d009b87c7b9d83bb82a861bfd1cce311005cc3e5dde058a74e4c64a" gracePeriod=30 Oct 30 17:15:12.598777 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:12.598743 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4be3cd3f-d7d6-40ec-ad72-8012cfbdb946" path="/var/lib/kubelet/pods/4be3cd3f-d7d6-40ec-ad72-8012cfbdb946/volumes" Oct 30 17:15:13.573669 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:13.573633 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3ed10a-6dcb868f5-2xwbx"] Oct 30 17:15:14.595463 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:14.595429 5052 scope.go:122] "RemoveContainer" containerID="99e46d04ff0046e0d7df60b6e3810826794c9e6365bca386f436e15480a73818" Oct 30 17:15:14.983083 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:14.983051 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-263f00-29889675-2hrdn" event={"ID":"1c5fe268-6035-4334-916e-b13bca739541","Type":"ContainerStarted","Data":"c17b37fc38ffa615c3858b223c8b7e9d870d9a9910d6d98b4baedb7b91f19a62"} Oct 30 17:15:15.596820 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:15.596791 5052 scope.go:122] "RemoveContainer" containerID="34e7982ee1236758e886805457bc2cbd3e5e35fd441ce4ec113498a15fc5a958" Oct 30 17:15:15.597513 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:15.596905 5052 scope.go:122] "RemoveContainer" containerID="db5868ff1ebe552152d3a736e979f27c4b1f706aa6201c869c312a35af226ca4" Oct 30 17:15:15.661489 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:15.661451 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-cb57p"] Oct 30 17:15:15.661782 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:15.661737 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-cb57p" podUID="e4036d90-5e91-477e-b81f-91c2dceee6c9" containerName="istio-proxy" containerID="cri-o://670199c105c11a949f324aa937df5522c7ee9bff0b95204e0ca9c22d9ce3c414" gracePeriod=30 Oct 30 17:15:16.015305 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:16.015238 5052 generic.go:356] "Generic (PLEG): container finished" podID="1c5fe268-6035-4334-916e-b13bca739541" containerID="c17b37fc38ffa615c3858b223c8b7e9d870d9a9910d6d98b4baedb7b91f19a62" exitCode=60 Oct 30 17:15:16.015512 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:16.015398 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-263f00-29889675-2hrdn" event={"ID":"1c5fe268-6035-4334-916e-b13bca739541","Type":"ContainerDied","Data":"c17b37fc38ffa615c3858b223c8b7e9d870d9a9910d6d98b4baedb7b91f19a62"} Oct 30 17:15:16.015512 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:16.015459 5052 scope.go:122] "RemoveContainer" containerID="99e46d04ff0046e0d7df60b6e3810826794c9e6365bca386f436e15480a73818" Oct 30 17:15:16.015994 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:16.015974 5052 scope.go:122] "RemoveContainer" containerID="c17b37fc38ffa615c3858b223c8b7e9d870d9a9910d6d98b4baedb7b91f19a62" Oct 30 17:15:16.016345 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:15:16.016319 5052 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-263f00-29889675-2hrdn_odh-ai-gateway-infra(1c5fe268-6035-4334-916e-b13bca739541)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-263f00-29889675-2hrdn" podUID="1c5fe268-6035-4334-916e-b13bca739541" Oct 30 17:15:16.043438 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:16.043405 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-263f00-29889675-2hrdn"] Oct 30 17:15:16.144116 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:16.144084 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ddc10d-99f47dfb8-w7ktd"] Oct 30 17:15:16.144967 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:16.144948 5052 cpu_manager.go:414] "containerMap: removing container" podUID="3cf66546-2340-4526-b4e2-9aebde2a705b" containerName="revoke-keys" Oct 30 17:15:16.144967 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:16.144967 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3cf66546-2340-4526-b4e2-9aebde2a705b" containerName="revoke-keys" Oct 30 17:15:16.145097 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:16.145020 5052 cpu_manager.go:414] "containerMap: removing container" podUID="4be3cd3f-d7d6-40ec-ad72-8012cfbdb946" containerName="istio-proxy" Oct 30 17:15:16.145097 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:16.145026 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4be3cd3f-d7d6-40ec-ad72-8012cfbdb946" containerName="istio-proxy" Oct 30 17:15:16.145097 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:16.145032 5052 cpu_manager.go:414] "containerMap: removing container" podUID="4be3cd3f-d7d6-40ec-ad72-8012cfbdb946" containerName="istio-proxy" Oct 30 17:15:16.145097 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:16.145037 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4be3cd3f-d7d6-40ec-ad72-8012cfbdb946" containerName="istio-proxy" Oct 30 17:15:16.145221 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:16.145133 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="4be3cd3f-d7d6-40ec-ad72-8012cfbdb946" containerName="istio-proxy" Oct 30 17:15:16.145221 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:16.145142 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="4be3cd3f-d7d6-40ec-ad72-8012cfbdb946" containerName="istio-proxy" Oct 30 17:15:16.145221 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:16.145151 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="3cf66546-2340-4526-b4e2-9aebde2a705b" containerName="revoke-keys" Oct 30 17:15:16.148825 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:16.148805 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ddc10d-99f47dfb8-w7ktd" Oct 30 17:15:16.165255 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:16.165197 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-ddc10d\"" Oct 30 17:15:16.165488 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:16.165467 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-ddc10d-dockercfg-7xtkr\"" Oct 30 17:15:16.169095 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:16.169073 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ddc10d-99f47dfb8-w7ktd"] Oct 30 17:15:16.208554 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:16.208517 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b774d2ea-e597-4122-b520-7ccdec91972f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ddc10d-99f47dfb8-w7ktd\" (UID: \"b774d2ea-e597-4122-b520-7ccdec91972f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ddc10d-99f47dfb8-w7ktd" Oct 30 17:15:16.208738 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:16.208699 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkq26\" (UniqueName: \"kubernetes.io/projected/b774d2ea-e597-4122-b520-7ccdec91972f-kube-api-access-hkq26\") pod \"payload-processing-e2e-ipp-b-ddc10d-99f47dfb8-w7ktd\" (UID: \"b774d2ea-e597-4122-b520-7ccdec91972f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ddc10d-99f47dfb8-w7ktd" Oct 30 17:15:16.241386 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:16.241362 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ddc10d-858465d86c-khzjm"] Oct 30 17:15:16.245800 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:16.245783 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ddc10d-858465d86c-khzjm" Oct 30 17:15:16.265885 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:16.265860 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ddc10d-858465d86c-khzjm"] Oct 30 17:15:16.309864 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:16.309831 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e6b7be8-b362-4477-86b5-cf8ff0cd266f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ddc10d-858465d86c-khzjm\" (UID: \"1e6b7be8-b362-4477-86b5-cf8ff0cd266f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ddc10d-858465d86c-khzjm" Oct 30 17:15:16.310038 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:16.309891 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrr8m\" (UniqueName: \"kubernetes.io/projected/1e6b7be8-b362-4477-86b5-cf8ff0cd266f-kube-api-access-lrr8m\") pod \"payload-pre-processing-e2e-ipp-b-ddc10d-858465d86c-khzjm\" (UID: \"1e6b7be8-b362-4477-86b5-cf8ff0cd266f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ddc10d-858465d86c-khzjm" Oct 30 17:15:16.310089 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:16.310043 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b774d2ea-e597-4122-b520-7ccdec91972f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ddc10d-99f47dfb8-w7ktd\" (UID: \"b774d2ea-e597-4122-b520-7ccdec91972f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ddc10d-99f47dfb8-w7ktd" Oct 30 17:15:16.310171 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:16.310154 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkq26\" (UniqueName: \"kubernetes.io/projected/b774d2ea-e597-4122-b520-7ccdec91972f-kube-api-access-hkq26\") pod \"payload-processing-e2e-ipp-b-ddc10d-99f47dfb8-w7ktd\" (UID: \"b774d2ea-e597-4122-b520-7ccdec91972f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ddc10d-99f47dfb8-w7ktd" Oct 30 17:15:16.310840 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:16.310813 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b774d2ea-e597-4122-b520-7ccdec91972f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ddc10d-99f47dfb8-w7ktd\" (UID: \"b774d2ea-e597-4122-b520-7ccdec91972f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ddc10d-99f47dfb8-w7ktd" Oct 30 17:15:16.334973 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:16.334945 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkq26\" (UniqueName: \"kubernetes.io/projected/b774d2ea-e597-4122-b520-7ccdec91972f-kube-api-access-hkq26\") pod \"payload-processing-e2e-ipp-b-ddc10d-99f47dfb8-w7ktd\" (UID: \"b774d2ea-e597-4122-b520-7ccdec91972f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ddc10d-99f47dfb8-w7ktd" Oct 30 17:15:16.411675 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:16.411637 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e6b7be8-b362-4477-86b5-cf8ff0cd266f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ddc10d-858465d86c-khzjm\" (UID: \"1e6b7be8-b362-4477-86b5-cf8ff0cd266f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ddc10d-858465d86c-khzjm" Oct 30 17:15:16.411974 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:16.411948 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrr8m\" (UniqueName: \"kubernetes.io/projected/1e6b7be8-b362-4477-86b5-cf8ff0cd266f-kube-api-access-lrr8m\") pod \"payload-pre-processing-e2e-ipp-b-ddc10d-858465d86c-khzjm\" (UID: \"1e6b7be8-b362-4477-86b5-cf8ff0cd266f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ddc10d-858465d86c-khzjm" Oct 30 17:15:16.413499 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:16.413472 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e6b7be8-b362-4477-86b5-cf8ff0cd266f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ddc10d-858465d86c-khzjm\" (UID: \"1e6b7be8-b362-4477-86b5-cf8ff0cd266f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ddc10d-858465d86c-khzjm" Oct 30 17:15:16.437261 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:16.437233 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrr8m\" (UniqueName: \"kubernetes.io/projected/1e6b7be8-b362-4477-86b5-cf8ff0cd266f-kube-api-access-lrr8m\") pod \"payload-pre-processing-e2e-ipp-b-ddc10d-858465d86c-khzjm\" (UID: \"1e6b7be8-b362-4477-86b5-cf8ff0cd266f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ddc10d-858465d86c-khzjm" Oct 30 17:15:16.463188 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:16.463163 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ddc10d-99f47dfb8-w7ktd" Oct 30 17:15:16.563110 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:16.562710 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ddc10d-858465d86c-khzjm" Oct 30 17:15:16.590576 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:16.590548 5052 scope.go:122] "RemoveContainer" containerID="1482a7fc04a0d93366e2eac59d08795e5aa3c69b802f8adcadd89d63468d826a" Oct 30 17:15:16.791251 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:16.791224 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ddc10d-99f47dfb8-w7ktd"] Oct 30 17:15:16.829753 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:16.829716 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-g6lll"] Oct 30 17:15:16.830007 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:16.829976 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-g6lll" podUID="0057b70d-3924-4e84-87f5-4cfc7e7babf5" containerName="istio-proxy" containerID="cri-o://48313a13198f7585ade8b6eff18fc89e7ac1427dc5757db7a6e919cbeeedb20c" gracePeriod=30 Oct 30 17:15:16.916337 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:16.916277 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ddc10d-858465d86c-khzjm"] Oct 30 17:15:16.990608 ip-10-0-131-136 kubenswrapper[5052]: W1030 17:15:16.983442 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e6b7be8_b362_4477_86b5_cf8ff0cd266f.slice/crio-9f637c21cb81bee5df3e1e7931b70ee5aee2e5bececdc9dfba62c9404e119f19 WatchSource:0}: Error finding container 9f637c21cb81bee5df3e1e7931b70ee5aee2e5bececdc9dfba62c9404e119f19: Status 404 returned error can't find the container with id 9f637c21cb81bee5df3e1e7931b70ee5aee2e5bececdc9dfba62c9404e119f19 Oct 30 17:15:17.048037 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:17.047995 5052 generic.go:356] "Generic (PLEG): container finished" podID="cbb4905a-78c4-4697-84c0-bd1a665ff034" containerID="d8eeb86755c8f7288dc10544d73a1f2a08ee3132915978fd504f150d483869ba" exitCode=60 Oct 30 17:15:17.048342 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:17.048312 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889675-d29kx" event={"ID":"cbb4905a-78c4-4697-84c0-bd1a665ff034","Type":"ContainerDied","Data":"d8eeb86755c8f7288dc10544d73a1f2a08ee3132915978fd504f150d483869ba"} Oct 30 17:15:17.048444 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:17.048398 5052 scope.go:122] "RemoveContainer" containerID="db5868ff1ebe552152d3a736e979f27c4b1f706aa6201c869c312a35af226ca4" Oct 30 17:15:17.052168 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:17.050782 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ddc10d-858465d86c-khzjm" event={"ID":"1e6b7be8-b362-4477-86b5-cf8ff0cd266f","Type":"ContainerStarted","Data":"9f637c21cb81bee5df3e1e7931b70ee5aee2e5bececdc9dfba62c9404e119f19"} Oct 30 17:15:17.054538 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:17.054445 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ddc10d-99f47dfb8-w7ktd" event={"ID":"b774d2ea-e597-4122-b520-7ccdec91972f","Type":"ContainerStarted","Data":"1af04d518bb57339c2f6cbeed71295f337b014f2af8b74da2d09dc2c3b2de1bc"} Oct 30 17:15:17.061433 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:17.058942 5052 scope.go:122] "RemoveContainer" containerID="d8eeb86755c8f7288dc10544d73a1f2a08ee3132915978fd504f150d483869ba" Oct 30 17:15:17.061433 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:15:17.059294 5052 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889675-d29kx_odh-ai-gateway-infra(cbb4905a-78c4-4697-84c0-bd1a665ff034)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889675-d29kx" podUID="cbb4905a-78c4-4697-84c0-bd1a665ff034" Oct 30 17:15:17.064714 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:17.064649 5052 generic.go:356] "Generic (PLEG): container finished" podID="7c8c2e2b-b5e5-4e6c-ace8-db859e693dde" containerID="506f53e2f087e6e7df1b8b0c9ae9ac96bbe7baa99b86708878305e6e3cecf322" exitCode=60 Oct 30 17:15:17.064714 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:17.064686 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bfecdd-29889675-6rzrt" event={"ID":"7c8c2e2b-b5e5-4e6c-ace8-db859e693dde","Type":"ContainerDied","Data":"506f53e2f087e6e7df1b8b0c9ae9ac96bbe7baa99b86708878305e6e3cecf322"} Oct 30 17:15:17.066512 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:17.065627 5052 scope.go:122] "RemoveContainer" containerID="506f53e2f087e6e7df1b8b0c9ae9ac96bbe7baa99b86708878305e6e3cecf322" Oct 30 17:15:17.066512 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:15:17.066457 5052 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-bfecdd-29889675-6rzrt_odh-ai-gateway-infra(7c8c2e2b-b5e5-4e6c-ace8-db859e693dde)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bfecdd-29889675-6rzrt" podUID="7c8c2e2b-b5e5-4e6c-ace8-db859e693dde" Oct 30 17:15:17.085703 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:17.085673 5052 scope.go:122] "RemoveContainer" containerID="34e7982ee1236758e886805457bc2cbd3e5e35fd441ce4ec113498a15fc5a958" Oct 30 17:15:17.395130 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:17.395014 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-263f00-29889675-2hrdn" Oct 30 17:15:17.516022 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:17.515949 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-661509-openshift-default-b5d9994f-xlcqt" Oct 30 17:15:17.581861 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:17.581801 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24bfdf71-2df9-478f-a6af-eca34fc212ca-kube-api-access-s6rrm\" (UniqueName: \"kubernetes.io/projected/24bfdf71-2df9-478f-a6af-eca34fc212ca-kube-api-access-s6rrm\") pod \"24bfdf71-2df9-478f-a6af-eca34fc212ca\" (UID: \"24bfdf71-2df9-478f-a6af-eca34fc212ca\") " Oct 30 17:15:17.582184 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:17.582163 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24bfdf71-2df9-478f-a6af-eca34fc212ca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24bfdf71-2df9-478f-a6af-eca34fc212ca-istio-envoy\") pod \"24bfdf71-2df9-478f-a6af-eca34fc212ca\" (UID: \"24bfdf71-2df9-478f-a6af-eca34fc212ca\") " Oct 30 17:15:17.582316 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:17.582303 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24bfdf71-2df9-478f-a6af-eca34fc212ca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24bfdf71-2df9-478f-a6af-eca34fc212ca-istiod-ca-cert\") pod \"24bfdf71-2df9-478f-a6af-eca34fc212ca\" (UID: \"24bfdf71-2df9-478f-a6af-eca34fc212ca\") " Oct 30 17:15:17.582516 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:17.582501 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/24bfdf71-2df9-478f-a6af-eca34fc212ca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24bfdf71-2df9-478f-a6af-eca34fc212ca-istio-podinfo\") pod \"24bfdf71-2df9-478f-a6af-eca34fc212ca\" (UID: \"24bfdf71-2df9-478f-a6af-eca34fc212ca\") " Oct 30 17:15:17.582735 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:17.582721 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c5fe268-6035-4334-916e-b13bca739541-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c5fe268-6035-4334-916e-b13bca739541-maas-api-service-ca\") pod \"1c5fe268-6035-4334-916e-b13bca739541\" (UID: \"1c5fe268-6035-4334-916e-b13bca739541\") " Oct 30 17:15:17.583228 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:17.583199 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24bfdf71-2df9-478f-a6af-eca34fc212ca-istiod-ca-cert" pod "24bfdf71-2df9-478f-a6af-eca34fc212ca" (UID: "24bfdf71-2df9-478f-a6af-eca34fc212ca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:15:17.583334 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:17.583262 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24bfdf71-2df9-478f-a6af-eca34fc212ca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24bfdf71-2df9-478f-a6af-eca34fc212ca-workload-socket\") pod \"24bfdf71-2df9-478f-a6af-eca34fc212ca\" (UID: \"24bfdf71-2df9-478f-a6af-eca34fc212ca\") " Oct 30 17:15:17.583334 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:17.583325 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c5fe268-6035-4334-916e-b13bca739541-kube-api-access-tkwgv\" (UniqueName: \"kubernetes.io/projected/1c5fe268-6035-4334-916e-b13bca739541-kube-api-access-tkwgv\") pod \"1c5fe268-6035-4334-916e-b13bca739541\" (UID: \"1c5fe268-6035-4334-916e-b13bca739541\") " Oct 30 17:15:17.583450 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:17.583364 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24bfdf71-2df9-478f-a6af-eca34fc212ca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24bfdf71-2df9-478f-a6af-eca34fc212ca-istio-data\") pod \"24bfdf71-2df9-478f-a6af-eca34fc212ca\" (UID: \"24bfdf71-2df9-478f-a6af-eca34fc212ca\") " Oct 30 17:15:17.583450 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:17.583412 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24bfdf71-2df9-478f-a6af-eca34fc212ca-istio-token\" (UniqueName: \"kubernetes.io/projected/24bfdf71-2df9-478f-a6af-eca34fc212ca-istio-token\") pod \"24bfdf71-2df9-478f-a6af-eca34fc212ca\" (UID: \"24bfdf71-2df9-478f-a6af-eca34fc212ca\") " Oct 30 17:15:17.583560 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:17.583483 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24bfdf71-2df9-478f-a6af-eca34fc212ca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24bfdf71-2df9-478f-a6af-eca34fc212ca-credential-socket\") pod \"24bfdf71-2df9-478f-a6af-eca34fc212ca\" (UID: \"24bfdf71-2df9-478f-a6af-eca34fc212ca\") " Oct 30 17:15:17.583560 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:17.583516 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24bfdf71-2df9-478f-a6af-eca34fc212ca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24bfdf71-2df9-478f-a6af-eca34fc212ca-workload-certs\") pod \"24bfdf71-2df9-478f-a6af-eca34fc212ca\" (UID: \"24bfdf71-2df9-478f-a6af-eca34fc212ca\") " Oct 30 17:15:17.583774 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:17.583736 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c5fe268-6035-4334-916e-b13bca739541-maas-api-service-ca" pod "1c5fe268-6035-4334-916e-b13bca739541" (UID: "1c5fe268-6035-4334-916e-b13bca739541"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:15:17.584210 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:17.584188 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24bfdf71-2df9-478f-a6af-eca34fc212ca-istio-data" pod "24bfdf71-2df9-478f-a6af-eca34fc212ca" (UID: "24bfdf71-2df9-478f-a6af-eca34fc212ca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:15:17.584317 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:17.584208 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24bfdf71-2df9-478f-a6af-eca34fc212ca-workload-certs" pod "24bfdf71-2df9-478f-a6af-eca34fc212ca" (UID: "24bfdf71-2df9-478f-a6af-eca34fc212ca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:15:17.584437 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:17.584410 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24bfdf71-2df9-478f-a6af-eca34fc212ca-credential-socket" pod "24bfdf71-2df9-478f-a6af-eca34fc212ca" (UID: "24bfdf71-2df9-478f-a6af-eca34fc212ca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:15:17.584740 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:17.584717 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24bfdf71-2df9-478f-a6af-eca34fc212ca-workload-socket" pod "24bfdf71-2df9-478f-a6af-eca34fc212ca" (UID: "24bfdf71-2df9-478f-a6af-eca34fc212ca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:15:17.585249 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:17.585229 5052 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24bfdf71-2df9-478f-a6af-eca34fc212ca-istio-data\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:15:17.585389 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:17.585375 5052 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24bfdf71-2df9-478f-a6af-eca34fc212ca-credential-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:15:17.585492 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:17.585479 5052 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24bfdf71-2df9-478f-a6af-eca34fc212ca-workload-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:15:17.585611 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:17.585598 5052 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24bfdf71-2df9-478f-a6af-eca34fc212ca-istiod-ca-cert\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:15:17.586771 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:17.586753 5052 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c5fe268-6035-4334-916e-b13bca739541-maas-api-service-ca\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:15:17.586881 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:17.586775 5052 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24bfdf71-2df9-478f-a6af-eca34fc212ca-workload-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:15:17.587091 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:17.587073 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/24bfdf71-2df9-478f-a6af-eca34fc212ca-istio-podinfo" pod "24bfdf71-2df9-478f-a6af-eca34fc212ca" (UID: "24bfdf71-2df9-478f-a6af-eca34fc212ca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:15:17.588513 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:17.588490 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24bfdf71-2df9-478f-a6af-eca34fc212ca-istio-envoy" pod "24bfdf71-2df9-478f-a6af-eca34fc212ca" (UID: "24bfdf71-2df9-478f-a6af-eca34fc212ca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:15:17.588808 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:17.588779 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24bfdf71-2df9-478f-a6af-eca34fc212ca-istio-token" pod "24bfdf71-2df9-478f-a6af-eca34fc212ca" (UID: "24bfdf71-2df9-478f-a6af-eca34fc212ca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:17.588920 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:17.588834 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24bfdf71-2df9-478f-a6af-eca34fc212ca-kube-api-access-s6rrm" pod "24bfdf71-2df9-478f-a6af-eca34fc212ca" (UID: "24bfdf71-2df9-478f-a6af-eca34fc212ca"). InnerVolumeSpecName "kube-api-access-s6rrm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:17.590025 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:17.590006 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c5fe268-6035-4334-916e-b13bca739541-kube-api-access-tkwgv" pod "1c5fe268-6035-4334-916e-b13bca739541" (UID: "1c5fe268-6035-4334-916e-b13bca739541"). InnerVolumeSpecName "kube-api-access-tkwgv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:17.688371 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:17.688336 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6rrm\" (UniqueName: \"kubernetes.io/projected/24bfdf71-2df9-478f-a6af-eca34fc212ca-kube-api-access-s6rrm\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:15:17.688371 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:17.688373 5052 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24bfdf71-2df9-478f-a6af-eca34fc212ca-istio-envoy\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:15:17.688594 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:17.688387 5052 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24bfdf71-2df9-478f-a6af-eca34fc212ca-istio-podinfo\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:15:17.688594 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:17.688403 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkwgv\" (UniqueName: \"kubernetes.io/projected/1c5fe268-6035-4334-916e-b13bca739541-kube-api-access-tkwgv\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:15:17.688594 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:17.688419 5052 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24bfdf71-2df9-478f-a6af-eca34fc212ca-istio-token\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:15:18.077344 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:18.077273 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ddc10d-858465d86c-khzjm" event={"ID":"1e6b7be8-b362-4477-86b5-cf8ff0cd266f","Type":"ContainerStarted","Data":"566bd7adb5bd0d2bde443cda2a8b67c8a663a40494132372a366a9b92b9e9b28"} Oct 30 17:15:18.077991 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:18.077961 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ddc10d-858465d86c-khzjm" Oct 30 17:15:18.079237 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:18.079195 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ddc10d-99f47dfb8-w7ktd" event={"ID":"b774d2ea-e597-4122-b520-7ccdec91972f","Type":"ContainerStarted","Data":"22c599594fea7cb342ef291f39b94b1b3bbd26fffaa19962f4cada57ad111fdf"} Oct 30 17:15:18.079378 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:18.079266 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-ddc10d-99f47dfb8-w7ktd" Oct 30 17:15:18.080726 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:18.080706 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-263f00-29889675-2hrdn" Oct 30 17:15:18.080821 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:18.080710 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-263f00-29889675-2hrdn" event={"ID":"1c5fe268-6035-4334-916e-b13bca739541","Type":"ContainerDied","Data":"1e86cdc12d90f38b97e644938c732e9bd8e093309545d2d9d2b962965df6f06b"} Oct 30 17:15:18.080874 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:18.080831 5052 scope.go:122] "RemoveContainer" containerID="c17b37fc38ffa615c3858b223c8b7e9d870d9a9910d6d98b4baedb7b91f19a62" Oct 30 17:15:18.084693 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:18.084667 5052 generic.go:356] "Generic (PLEG): container finished" podID="4bcebabf-2dba-4645-b30d-6280c85ca947" containerID="6b917ea699e959238041ca6cdce98fe7855a79dd0ae4edd539d39a7ad0a3593b" exitCode=60 Oct 30 17:15:18.084917 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:18.084745 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7c4b84-29889675-2mrhn" event={"ID":"4bcebabf-2dba-4645-b30d-6280c85ca947","Type":"ContainerDied","Data":"6b917ea699e959238041ca6cdce98fe7855a79dd0ae4edd539d39a7ad0a3593b"} Oct 30 17:15:18.085441 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:18.085136 5052 scope.go:122] "RemoveContainer" containerID="6b917ea699e959238041ca6cdce98fe7855a79dd0ae4edd539d39a7ad0a3593b" Oct 30 17:15:18.085568 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:15:18.085441 5052 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-7c4b84-29889675-2mrhn_odh-ai-gateway-infra(4bcebabf-2dba-4645-b30d-6280c85ca947)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7c4b84-29889675-2mrhn" podUID="4bcebabf-2dba-4645-b30d-6280c85ca947" Oct 30 17:15:18.089067 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:18.089043 5052 generic.go:356] "Generic (PLEG): container finished" podID="24bfdf71-2df9-478f-a6af-eca34fc212ca" containerID="3795d7d00d009b87c7b9d83bb82a861bfd1cce311005cc3e5dde058a74e4c64a" exitCode=0 Oct 30 17:15:18.089163 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:18.089077 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-661509-openshift-default-b5d9994f-xlcqt" Oct 30 17:15:18.089163 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:18.089101 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-661509-openshift-default-b5d9994f-xlcqt" event={"ID":"24bfdf71-2df9-478f-a6af-eca34fc212ca","Type":"ContainerDied","Data":"3795d7d00d009b87c7b9d83bb82a861bfd1cce311005cc3e5dde058a74e4c64a"} Oct 30 17:15:18.089163 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:18.089131 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-661509-openshift-default-b5d9994f-xlcqt" event={"ID":"24bfdf71-2df9-478f-a6af-eca34fc212ca","Type":"ContainerDied","Data":"1b3b8638e44dc25bbbdf1205f0a29d87edc5427971ae4764fae74b4f4b1c3843"} Oct 30 17:15:18.093724 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:18.093668 5052 scope.go:122] "RemoveContainer" containerID="1482a7fc04a0d93366e2eac59d08795e5aa3c69b802f8adcadd89d63468d826a" Oct 30 17:15:18.105506 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:18.105480 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889675-d29kx"] Oct 30 17:15:18.109016 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:18.108995 5052 scope.go:122] "RemoveContainer" containerID="3795d7d00d009b87c7b9d83bb82a861bfd1cce311005cc3e5dde058a74e4c64a" Oct 30 17:15:18.120728 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:18.120706 5052 scope.go:122] "RemoveContainer" containerID="3795d7d00d009b87c7b9d83bb82a861bfd1cce311005cc3e5dde058a74e4c64a" Oct 30 17:15:18.121313 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:15:18.121036 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3795d7d00d009b87c7b9d83bb82a861bfd1cce311005cc3e5dde058a74e4c64a\": container with ID starting with 3795d7d00d009b87c7b9d83bb82a861bfd1cce311005cc3e5dde058a74e4c64a not found: ID does not exist" containerID="3795d7d00d009b87c7b9d83bb82a861bfd1cce311005cc3e5dde058a74e4c64a" Oct 30 17:15:18.121313 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:18.121069 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3795d7d00d009b87c7b9d83bb82a861bfd1cce311005cc3e5dde058a74e4c64a"} err="failed to get container status \"3795d7d00d009b87c7b9d83bb82a861bfd1cce311005cc3e5dde058a74e4c64a\": rpc error: code = NotFound desc = could not find container \"3795d7d00d009b87c7b9d83bb82a861bfd1cce311005cc3e5dde058a74e4c64a\": container with ID starting with 3795d7d00d009b87c7b9d83bb82a861bfd1cce311005cc3e5dde058a74e4c64a not found: ID does not exist" Oct 30 17:15:18.127151 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:18.127109 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ddc10d-858465d86c-khzjm" podStartSLOduration=2.127096602 podStartE2EDuration="2.127096602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:15:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:15:18.121264133 +0000 UTC m=+3273.648599355" watchObservedRunningTime="2026-10-30 17:15:18.127096602 +0000 UTC m=+3273.654431815" Oct 30 17:15:18.220108 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:18.220043 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-ddc10d-99f47dfb8-w7ktd" podStartSLOduration=2.220028477 podStartE2EDuration="2.220028477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:15:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:15:18.215861243 +0000 UTC m=+3273.743196489" watchObservedRunningTime="2026-10-30 17:15:18.220028477 +0000 UTC m=+3273.747363696" Oct 30 17:15:18.226988 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:18.226859 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bfecdd-29889675-6rzrt"] Oct 30 17:15:18.264623 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:18.261633 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-263f00-29889675-2hrdn"] Oct 30 17:15:18.267206 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:18.267176 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-263f00-29889675-2hrdn"] Oct 30 17:15:18.315202 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:18.314987 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889675-d29kx" Oct 30 17:15:18.315202 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:18.315086 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-661509-openshift-default-b5d9994f-xlcqt"] Oct 30 17:15:18.317341 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:18.317311 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-661509-openshift-default-b5d9994f-xlcqt"] Oct 30 17:15:18.411269 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:18.411199 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbb4905a-78c4-4697-84c0-bd1a665ff034-maas-api-service-ca" pod "cbb4905a-78c4-4697-84c0-bd1a665ff034" (UID: "cbb4905a-78c4-4697-84c0-bd1a665ff034"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:15:18.411531 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:18.411508 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbb4905a-78c4-4697-84c0-bd1a665ff034-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbb4905a-78c4-4697-84c0-bd1a665ff034-maas-api-service-ca\") pod \"cbb4905a-78c4-4697-84c0-bd1a665ff034\" (UID: \"cbb4905a-78c4-4697-84c0-bd1a665ff034\") " Oct 30 17:15:18.412259 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:18.411814 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbb4905a-78c4-4697-84c0-bd1a665ff034-kube-api-access-lkvn6\" (UniqueName: \"kubernetes.io/projected/cbb4905a-78c4-4697-84c0-bd1a665ff034-kube-api-access-lkvn6\") pod \"cbb4905a-78c4-4697-84c0-bd1a665ff034\" (UID: \"cbb4905a-78c4-4697-84c0-bd1a665ff034\") " Oct 30 17:15:18.413487 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:18.413465 5052 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbb4905a-78c4-4697-84c0-bd1a665ff034-maas-api-service-ca\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:15:18.420624 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:18.414791 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbb4905a-78c4-4697-84c0-bd1a665ff034-kube-api-access-lkvn6" pod "cbb4905a-78c4-4697-84c0-bd1a665ff034" (UID: "cbb4905a-78c4-4697-84c0-bd1a665ff034"). InnerVolumeSpecName "kube-api-access-lkvn6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:18.476202 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:18.476171 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bfecdd-29889675-6rzrt" Oct 30 17:15:18.514911 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:18.514882 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c8c2e2b-b5e5-4e6c-ace8-db859e693dde-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c8c2e2b-b5e5-4e6c-ace8-db859e693dde-maas-api-service-ca\") pod \"7c8c2e2b-b5e5-4e6c-ace8-db859e693dde\" (UID: \"7c8c2e2b-b5e5-4e6c-ace8-db859e693dde\") " Oct 30 17:15:18.515073 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:18.514938 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c8c2e2b-b5e5-4e6c-ace8-db859e693dde-kube-api-access-m8dk9\" (UniqueName: \"kubernetes.io/projected/7c8c2e2b-b5e5-4e6c-ace8-db859e693dde-kube-api-access-m8dk9\") pod \"7c8c2e2b-b5e5-4e6c-ace8-db859e693dde\" (UID: \"7c8c2e2b-b5e5-4e6c-ace8-db859e693dde\") " Oct 30 17:15:18.515220 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:18.515198 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c8c2e2b-b5e5-4e6c-ace8-db859e693dde-maas-api-service-ca" pod "7c8c2e2b-b5e5-4e6c-ace8-db859e693dde" (UID: "7c8c2e2b-b5e5-4e6c-ace8-db859e693dde"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:15:18.515389 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:18.515369 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkvn6\" (UniqueName: \"kubernetes.io/projected/cbb4905a-78c4-4697-84c0-bd1a665ff034-kube-api-access-lkvn6\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:15:18.515455 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:18.515393 5052 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c8c2e2b-b5e5-4e6c-ace8-db859e693dde-maas-api-service-ca\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:15:18.517162 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:18.517135 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c8c2e2b-b5e5-4e6c-ace8-db859e693dde-kube-api-access-m8dk9" pod "7c8c2e2b-b5e5-4e6c-ace8-db859e693dde" (UID: "7c8c2e2b-b5e5-4e6c-ace8-db859e693dde"). InnerVolumeSpecName "kube-api-access-m8dk9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:18.574832 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:18.574792 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3ed10a-6dcb868f5-2xwbx" podUID="b4d2755a-efbb-41f5-983f-309ee1cff113" containerName="payload-pre-processing" containerID="cri-o://510c0776948b928bb418a1d124f9c173d33a14c4345219c0fc13a1b771033a89" gracePeriod=25 Oct 30 17:15:18.599394 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:18.599330 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c5fe268-6035-4334-916e-b13bca739541" path="/var/lib/kubelet/pods/1c5fe268-6035-4334-916e-b13bca739541/volumes" Oct 30 17:15:18.599961 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:18.599944 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24bfdf71-2df9-478f-a6af-eca34fc212ca" path="/var/lib/kubelet/pods/24bfdf71-2df9-478f-a6af-eca34fc212ca/volumes" Oct 30 17:15:18.617023 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:18.616999 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8dk9\" (UniqueName: \"kubernetes.io/projected/7c8c2e2b-b5e5-4e6c-ace8-db859e693dde-kube-api-access-m8dk9\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:15:18.856387 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:15:18.856294 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 17:15:19.100664 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:19.100551 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889675-d29kx" Oct 30 17:15:19.100664 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:19.100528 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889675-d29kx" event={"ID":"cbb4905a-78c4-4697-84c0-bd1a665ff034","Type":"ContainerDied","Data":"4029a3cdf41203e651007df9c052ab56b9cd1ff06814a2a990fb57cc793e9ec9"} Oct 30 17:15:19.101219 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:19.100760 5052 scope.go:122] "RemoveContainer" containerID="d8eeb86755c8f7288dc10544d73a1f2a08ee3132915978fd504f150d483869ba" Oct 30 17:15:19.104619 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:19.104559 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bfecdd-29889675-6rzrt" event={"ID":"7c8c2e2b-b5e5-4e6c-ace8-db859e693dde","Type":"ContainerDied","Data":"a681e7e4b014f0eb468008c758a280f94fe8830d7502a841e3f4fa98f7b16d0d"} Oct 30 17:15:19.105170 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:19.105151 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bfecdd-29889675-6rzrt" Oct 30 17:15:19.113870 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:19.113849 5052 scope.go:122] "RemoveContainer" containerID="506f53e2f087e6e7df1b8b0c9ae9ac96bbe7baa99b86708878305e6e3cecf322" Oct 30 17:15:19.159946 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:19.159916 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889675-d29kx"] Oct 30 17:15:19.163450 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:19.163420 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889675-d29kx"] Oct 30 17:15:19.177723 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:19.177693 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7c4b84-29889675-2mrhn"] Oct 30 17:15:19.205816 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:19.205744 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bfecdd-29889675-6rzrt"] Oct 30 17:15:19.223935 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:19.223894 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bfecdd-29889675-6rzrt"] Oct 30 17:15:19.373085 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:19.373014 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7c4b84-29889675-2mrhn" Oct 30 17:15:19.434422 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:19.434374 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bcebabf-2dba-4645-b30d-6280c85ca947-kube-api-access-sdzv2\" (UniqueName: \"kubernetes.io/projected/4bcebabf-2dba-4645-b30d-6280c85ca947-kube-api-access-sdzv2\") pod \"4bcebabf-2dba-4645-b30d-6280c85ca947\" (UID: \"4bcebabf-2dba-4645-b30d-6280c85ca947\") " Oct 30 17:15:19.434612 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:19.434453 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bcebabf-2dba-4645-b30d-6280c85ca947-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bcebabf-2dba-4645-b30d-6280c85ca947-maas-api-service-ca\") pod \"4bcebabf-2dba-4645-b30d-6280c85ca947\" (UID: \"4bcebabf-2dba-4645-b30d-6280c85ca947\") " Oct 30 17:15:19.435223 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:19.435192 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bcebabf-2dba-4645-b30d-6280c85ca947-maas-api-service-ca" pod "4bcebabf-2dba-4645-b30d-6280c85ca947" (UID: "4bcebabf-2dba-4645-b30d-6280c85ca947"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:15:19.436925 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:19.436891 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bcebabf-2dba-4645-b30d-6280c85ca947-kube-api-access-sdzv2" pod "4bcebabf-2dba-4645-b30d-6280c85ca947" (UID: "4bcebabf-2dba-4645-b30d-6280c85ca947"). InnerVolumeSpecName "kube-api-access-sdzv2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:19.536024 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:19.535985 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdzv2\" (UniqueName: \"kubernetes.io/projected/4bcebabf-2dba-4645-b30d-6280c85ca947-kube-api-access-sdzv2\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:15:19.536024 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:19.536017 5052 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bcebabf-2dba-4645-b30d-6280c85ca947-maas-api-service-ca\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:15:20.113239 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:20.113203 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7c4b84-29889675-2mrhn" Oct 30 17:15:20.113239 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:20.113226 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7c4b84-29889675-2mrhn" event={"ID":"4bcebabf-2dba-4645-b30d-6280c85ca947","Type":"ContainerDied","Data":"01ee1299d1e89bc1ea6f732300b258ca9458cfe538903feffa9c17878816fbde"} Oct 30 17:15:20.113830 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:20.113303 5052 scope.go:122] "RemoveContainer" containerID="6b917ea699e959238041ca6cdce98fe7855a79dd0ae4edd539d39a7ad0a3593b" Oct 30 17:15:20.184686 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:20.184656 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7c4b84-29889675-2mrhn"] Oct 30 17:15:20.198009 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:20.197981 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7c4b84-29889675-2mrhn"] Oct 30 17:15:20.599560 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:20.599522 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bcebabf-2dba-4645-b30d-6280c85ca947" path="/var/lib/kubelet/pods/4bcebabf-2dba-4645-b30d-6280c85ca947/volumes" Oct 30 17:15:20.600304 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:20.600262 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c8c2e2b-b5e5-4e6c-ace8-db859e693dde" path="/var/lib/kubelet/pods/7c8c2e2b-b5e5-4e6c-ace8-db859e693dde/volumes" Oct 30 17:15:20.600922 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:20.600903 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbb4905a-78c4-4697-84c0-bd1a665ff034" path="/var/lib/kubelet/pods/cbb4905a-78c4-4697-84c0-bd1a665ff034/volumes" Oct 30 17:15:20.947633 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:20.947566 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cdwcj"] Oct 30 17:15:20.986850 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:20.986819 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-cb57p" Oct 30 17:15:21.053269 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:21.052839 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4036d90-5e91-477e-b81f-91c2dceee6c9-kube-api-access-mkpv6\" (UniqueName: \"kubernetes.io/projected/e4036d90-5e91-477e-b81f-91c2dceee6c9-kube-api-access-mkpv6\") pod \"e4036d90-5e91-477e-b81f-91c2dceee6c9\" (UID: \"e4036d90-5e91-477e-b81f-91c2dceee6c9\") " Oct 30 17:15:21.053269 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:21.052892 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4036d90-5e91-477e-b81f-91c2dceee6c9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4036d90-5e91-477e-b81f-91c2dceee6c9-istio-data\") pod \"e4036d90-5e91-477e-b81f-91c2dceee6c9\" (UID: \"e4036d90-5e91-477e-b81f-91c2dceee6c9\") " Oct 30 17:15:21.053269 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:21.052949 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4036d90-5e91-477e-b81f-91c2dceee6c9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4036d90-5e91-477e-b81f-91c2dceee6c9-istiod-ca-cert\") pod \"e4036d90-5e91-477e-b81f-91c2dceee6c9\" (UID: \"e4036d90-5e91-477e-b81f-91c2dceee6c9\") " Oct 30 17:15:21.053269 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:21.052973 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4036d90-5e91-477e-b81f-91c2dceee6c9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4036d90-5e91-477e-b81f-91c2dceee6c9-credential-socket\") pod \"e4036d90-5e91-477e-b81f-91c2dceee6c9\" (UID: \"e4036d90-5e91-477e-b81f-91c2dceee6c9\") " Oct 30 17:15:21.053269 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:21.053067 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4036d90-5e91-477e-b81f-91c2dceee6c9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4036d90-5e91-477e-b81f-91c2dceee6c9-workload-certs\") pod \"e4036d90-5e91-477e-b81f-91c2dceee6c9\" (UID: \"e4036d90-5e91-477e-b81f-91c2dceee6c9\") " Oct 30 17:15:21.053269 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:21.053111 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4036d90-5e91-477e-b81f-91c2dceee6c9-istio-token\" (UniqueName: \"kubernetes.io/projected/e4036d90-5e91-477e-b81f-91c2dceee6c9-istio-token\") pod \"e4036d90-5e91-477e-b81f-91c2dceee6c9\" (UID: \"e4036d90-5e91-477e-b81f-91c2dceee6c9\") " Oct 30 17:15:21.053269 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:21.053173 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4036d90-5e91-477e-b81f-91c2dceee6c9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4036d90-5e91-477e-b81f-91c2dceee6c9-istio-envoy\") pod \"e4036d90-5e91-477e-b81f-91c2dceee6c9\" (UID: \"e4036d90-5e91-477e-b81f-91c2dceee6c9\") " Oct 30 17:15:21.053269 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:21.053217 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4036d90-5e91-477e-b81f-91c2dceee6c9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4036d90-5e91-477e-b81f-91c2dceee6c9-workload-socket\") pod \"e4036d90-5e91-477e-b81f-91c2dceee6c9\" (UID: \"e4036d90-5e91-477e-b81f-91c2dceee6c9\") " Oct 30 17:15:21.053269 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:21.053247 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4036d90-5e91-477e-b81f-91c2dceee6c9-istio-data" pod "e4036d90-5e91-477e-b81f-91c2dceee6c9" (UID: "e4036d90-5e91-477e-b81f-91c2dceee6c9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:15:21.053269 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:21.053265 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e4036d90-5e91-477e-b81f-91c2dceee6c9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4036d90-5e91-477e-b81f-91c2dceee6c9-istio-podinfo\") pod \"e4036d90-5e91-477e-b81f-91c2dceee6c9\" (UID: \"e4036d90-5e91-477e-b81f-91c2dceee6c9\") " Oct 30 17:15:21.054035 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:21.054009 5052 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4036d90-5e91-477e-b81f-91c2dceee6c9-istio-data\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:15:21.054623 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:21.054283 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4036d90-5e91-477e-b81f-91c2dceee6c9-credential-socket" pod "e4036d90-5e91-477e-b81f-91c2dceee6c9" (UID: "e4036d90-5e91-477e-b81f-91c2dceee6c9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:15:21.054623 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:21.054329 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4036d90-5e91-477e-b81f-91c2dceee6c9-istiod-ca-cert" pod "e4036d90-5e91-477e-b81f-91c2dceee6c9" (UID: "e4036d90-5e91-477e-b81f-91c2dceee6c9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:15:21.054892 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:21.054852 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4036d90-5e91-477e-b81f-91c2dceee6c9-workload-socket" pod "e4036d90-5e91-477e-b81f-91c2dceee6c9" (UID: "e4036d90-5e91-477e-b81f-91c2dceee6c9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:15:21.063166 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:21.057986 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4036d90-5e91-477e-b81f-91c2dceee6c9-workload-certs" pod "e4036d90-5e91-477e-b81f-91c2dceee6c9" (UID: "e4036d90-5e91-477e-b81f-91c2dceee6c9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:15:21.063166 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:21.059457 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4036d90-5e91-477e-b81f-91c2dceee6c9-kube-api-access-mkpv6" pod "e4036d90-5e91-477e-b81f-91c2dceee6c9" (UID: "e4036d90-5e91-477e-b81f-91c2dceee6c9"). InnerVolumeSpecName "kube-api-access-mkpv6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:21.064656 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:21.064628 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e4036d90-5e91-477e-b81f-91c2dceee6c9-istio-podinfo" pod "e4036d90-5e91-477e-b81f-91c2dceee6c9" (UID: "e4036d90-5e91-477e-b81f-91c2dceee6c9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:15:21.065969 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:21.065939 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4036d90-5e91-477e-b81f-91c2dceee6c9-istio-envoy" pod "e4036d90-5e91-477e-b81f-91c2dceee6c9" (UID: "e4036d90-5e91-477e-b81f-91c2dceee6c9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:15:21.070350 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:21.070309 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4036d90-5e91-477e-b81f-91c2dceee6c9-istio-token" pod "e4036d90-5e91-477e-b81f-91c2dceee6c9" (UID: "e4036d90-5e91-477e-b81f-91c2dceee6c9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:21.123423 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:21.123339 5052 generic.go:356] "Generic (PLEG): container finished" podID="e4036d90-5e91-477e-b81f-91c2dceee6c9" containerID="670199c105c11a949f324aa937df5522c7ee9bff0b95204e0ca9c22d9ce3c414" exitCode=0 Oct 30 17:15:21.123423 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:21.123386 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-cb57p" Oct 30 17:15:21.123949 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:21.123411 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-cb57p" event={"ID":"e4036d90-5e91-477e-b81f-91c2dceee6c9","Type":"ContainerDied","Data":"670199c105c11a949f324aa937df5522c7ee9bff0b95204e0ca9c22d9ce3c414"} Oct 30 17:15:21.123949 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:21.123468 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-cb57p" event={"ID":"e4036d90-5e91-477e-b81f-91c2dceee6c9","Type":"ContainerDied","Data":"521908381a2a94c97ccc673e0cbe34907905b8ae68879361f1c4dfc95cdbd8b7"} Oct 30 17:15:21.123949 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:21.123497 5052 scope.go:122] "RemoveContainer" containerID="670199c105c11a949f324aa937df5522c7ee9bff0b95204e0ca9c22d9ce3c414" Oct 30 17:15:21.154617 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:21.143703 5052 scope.go:122] "RemoveContainer" containerID="670199c105c11a949f324aa937df5522c7ee9bff0b95204e0ca9c22d9ce3c414" Oct 30 17:15:21.154617 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:15:21.144104 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"670199c105c11a949f324aa937df5522c7ee9bff0b95204e0ca9c22d9ce3c414\": container with ID starting with 670199c105c11a949f324aa937df5522c7ee9bff0b95204e0ca9c22d9ce3c414 not found: ID does not exist" containerID="670199c105c11a949f324aa937df5522c7ee9bff0b95204e0ca9c22d9ce3c414" Oct 30 17:15:21.154617 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:21.144142 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"670199c105c11a949f324aa937df5522c7ee9bff0b95204e0ca9c22d9ce3c414"} err="failed to get container status \"670199c105c11a949f324aa937df5522c7ee9bff0b95204e0ca9c22d9ce3c414\": rpc error: code = NotFound desc = could not find container \"670199c105c11a949f324aa937df5522c7ee9bff0b95204e0ca9c22d9ce3c414\": container with ID starting with 670199c105c11a949f324aa937df5522c7ee9bff0b95204e0ca9c22d9ce3c414 not found: ID does not exist" Oct 30 17:15:21.155201 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:21.155166 5052 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4036d90-5e91-477e-b81f-91c2dceee6c9-istiod-ca-cert\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:15:21.155201 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:21.155201 5052 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4036d90-5e91-477e-b81f-91c2dceee6c9-credential-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:15:21.155415 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:21.155216 5052 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4036d90-5e91-477e-b81f-91c2dceee6c9-workload-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:15:21.155415 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:21.155229 5052 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4036d90-5e91-477e-b81f-91c2dceee6c9-istio-token\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:15:21.155415 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:21.155239 5052 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4036d90-5e91-477e-b81f-91c2dceee6c9-istio-envoy\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:15:21.155415 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:21.155253 5052 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4036d90-5e91-477e-b81f-91c2dceee6c9-workload-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:15:21.155415 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:21.155265 5052 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4036d90-5e91-477e-b81f-91c2dceee6c9-istio-podinfo\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:15:21.155415 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:21.155278 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkpv6\" (UniqueName: \"kubernetes.io/projected/e4036d90-5e91-477e-b81f-91c2dceee6c9-kube-api-access-mkpv6\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:15:21.180709 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:21.180680 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-cb57p"] Oct 30 17:15:21.186647 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:21.186620 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-cb57p"] Oct 30 17:15:22.111181 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:22.111155 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-g6lll" Oct 30 17:15:22.134748 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:22.134721 5052 generic.go:356] "Generic (PLEG): container finished" podID="0057b70d-3924-4e84-87f5-4cfc7e7babf5" containerID="48313a13198f7585ade8b6eff18fc89e7ac1427dc5757db7a6e919cbeeedb20c" exitCode=0 Oct 30 17:15:22.135101 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:22.134766 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-g6lll" Oct 30 17:15:22.135101 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:22.134807 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-g6lll" event={"ID":"0057b70d-3924-4e84-87f5-4cfc7e7babf5","Type":"ContainerDied","Data":"48313a13198f7585ade8b6eff18fc89e7ac1427dc5757db7a6e919cbeeedb20c"} Oct 30 17:15:22.135101 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:22.134856 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-g6lll" event={"ID":"0057b70d-3924-4e84-87f5-4cfc7e7babf5","Type":"ContainerDied","Data":"241da2d4113cdfd665baa79a0b454d61186610d354f4e6555076577405bec0cf"} Oct 30 17:15:22.135101 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:22.134879 5052 scope.go:122] "RemoveContainer" containerID="48313a13198f7585ade8b6eff18fc89e7ac1427dc5757db7a6e919cbeeedb20c" Oct 30 17:15:22.145807 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:22.145787 5052 scope.go:122] "RemoveContainer" containerID="48313a13198f7585ade8b6eff18fc89e7ac1427dc5757db7a6e919cbeeedb20c" Oct 30 17:15:22.146053 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:15:22.146030 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48313a13198f7585ade8b6eff18fc89e7ac1427dc5757db7a6e919cbeeedb20c\": container with ID starting with 48313a13198f7585ade8b6eff18fc89e7ac1427dc5757db7a6e919cbeeedb20c not found: ID does not exist" containerID="48313a13198f7585ade8b6eff18fc89e7ac1427dc5757db7a6e919cbeeedb20c" Oct 30 17:15:22.146111 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:22.146062 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48313a13198f7585ade8b6eff18fc89e7ac1427dc5757db7a6e919cbeeedb20c"} err="failed to get container status \"48313a13198f7585ade8b6eff18fc89e7ac1427dc5757db7a6e919cbeeedb20c\": rpc error: code = NotFound desc = could not find container \"48313a13198f7585ade8b6eff18fc89e7ac1427dc5757db7a6e919cbeeedb20c\": container with ID starting with 48313a13198f7585ade8b6eff18fc89e7ac1427dc5757db7a6e919cbeeedb20c not found: ID does not exist" Oct 30 17:15:22.167812 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:22.167790 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0057b70d-3924-4e84-87f5-4cfc7e7babf5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0057b70d-3924-4e84-87f5-4cfc7e7babf5-istio-envoy\") pod \"0057b70d-3924-4e84-87f5-4cfc7e7babf5\" (UID: \"0057b70d-3924-4e84-87f5-4cfc7e7babf5\") " Oct 30 17:15:22.167903 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:22.167869 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0057b70d-3924-4e84-87f5-4cfc7e7babf5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0057b70d-3924-4e84-87f5-4cfc7e7babf5-istiod-ca-cert\") pod \"0057b70d-3924-4e84-87f5-4cfc7e7babf5\" (UID: \"0057b70d-3924-4e84-87f5-4cfc7e7babf5\") " Oct 30 17:15:22.167995 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:22.167977 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0057b70d-3924-4e84-87f5-4cfc7e7babf5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0057b70d-3924-4e84-87f5-4cfc7e7babf5-workload-socket\") pod \"0057b70d-3924-4e84-87f5-4cfc7e7babf5\" (UID: \"0057b70d-3924-4e84-87f5-4cfc7e7babf5\") " Oct 30 17:15:22.168055 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:22.168040 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0057b70d-3924-4e84-87f5-4cfc7e7babf5-kube-api-access-zd5h2\" (UniqueName: \"kubernetes.io/projected/0057b70d-3924-4e84-87f5-4cfc7e7babf5-kube-api-access-zd5h2\") pod \"0057b70d-3924-4e84-87f5-4cfc7e7babf5\" (UID: \"0057b70d-3924-4e84-87f5-4cfc7e7babf5\") " Oct 30 17:15:22.168140 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:22.168123 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0057b70d-3924-4e84-87f5-4cfc7e7babf5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0057b70d-3924-4e84-87f5-4cfc7e7babf5-istio-podinfo\") pod \"0057b70d-3924-4e84-87f5-4cfc7e7babf5\" (UID: \"0057b70d-3924-4e84-87f5-4cfc7e7babf5\") " Oct 30 17:15:22.168237 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:22.168218 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0057b70d-3924-4e84-87f5-4cfc7e7babf5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0057b70d-3924-4e84-87f5-4cfc7e7babf5-workload-certs\") pod \"0057b70d-3924-4e84-87f5-4cfc7e7babf5\" (UID: \"0057b70d-3924-4e84-87f5-4cfc7e7babf5\") " Oct 30 17:15:22.168298 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:22.168226 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0057b70d-3924-4e84-87f5-4cfc7e7babf5-workload-socket" pod "0057b70d-3924-4e84-87f5-4cfc7e7babf5" (UID: "0057b70d-3924-4e84-87f5-4cfc7e7babf5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:15:22.168298 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:22.168233 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0057b70d-3924-4e84-87f5-4cfc7e7babf5-istiod-ca-cert" pod "0057b70d-3924-4e84-87f5-4cfc7e7babf5" (UID: "0057b70d-3924-4e84-87f5-4cfc7e7babf5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:15:22.168298 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:22.168253 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0057b70d-3924-4e84-87f5-4cfc7e7babf5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0057b70d-3924-4e84-87f5-4cfc7e7babf5-istio-data\") pod \"0057b70d-3924-4e84-87f5-4cfc7e7babf5\" (UID: \"0057b70d-3924-4e84-87f5-4cfc7e7babf5\") " Oct 30 17:15:22.168471 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:22.168321 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0057b70d-3924-4e84-87f5-4cfc7e7babf5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0057b70d-3924-4e84-87f5-4cfc7e7babf5-credential-socket\") pod \"0057b70d-3924-4e84-87f5-4cfc7e7babf5\" (UID: \"0057b70d-3924-4e84-87f5-4cfc7e7babf5\") " Oct 30 17:15:22.168471 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:22.168352 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0057b70d-3924-4e84-87f5-4cfc7e7babf5-istio-token\" (UniqueName: \"kubernetes.io/projected/0057b70d-3924-4e84-87f5-4cfc7e7babf5-istio-token\") pod \"0057b70d-3924-4e84-87f5-4cfc7e7babf5\" (UID: \"0057b70d-3924-4e84-87f5-4cfc7e7babf5\") " Oct 30 17:15:22.168471 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:22.168444 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0057b70d-3924-4e84-87f5-4cfc7e7babf5-workload-certs" pod "0057b70d-3924-4e84-87f5-4cfc7e7babf5" (UID: "0057b70d-3924-4e84-87f5-4cfc7e7babf5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:15:22.168648 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:22.168570 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0057b70d-3924-4e84-87f5-4cfc7e7babf5-istio-data" pod "0057b70d-3924-4e84-87f5-4cfc7e7babf5" (UID: "0057b70d-3924-4e84-87f5-4cfc7e7babf5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:15:22.168740 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:22.168714 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0057b70d-3924-4e84-87f5-4cfc7e7babf5-credential-socket" pod "0057b70d-3924-4e84-87f5-4cfc7e7babf5" (UID: "0057b70d-3924-4e84-87f5-4cfc7e7babf5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:15:22.168950 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:22.168931 5052 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0057b70d-3924-4e84-87f5-4cfc7e7babf5-workload-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:15:22.169023 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:22.168955 5052 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0057b70d-3924-4e84-87f5-4cfc7e7babf5-istio-data\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:15:22.169023 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:22.168970 5052 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0057b70d-3924-4e84-87f5-4cfc7e7babf5-credential-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:15:22.169023 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:22.168983 5052 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0057b70d-3924-4e84-87f5-4cfc7e7babf5-istiod-ca-cert\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:15:22.169023 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:22.168997 5052 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0057b70d-3924-4e84-87f5-4cfc7e7babf5-workload-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:15:22.170193 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:22.170168 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0057b70d-3924-4e84-87f5-4cfc7e7babf5-istio-envoy" pod "0057b70d-3924-4e84-87f5-4cfc7e7babf5" (UID: "0057b70d-3924-4e84-87f5-4cfc7e7babf5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:15:22.170300 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:22.170230 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0057b70d-3924-4e84-87f5-4cfc7e7babf5-istio-podinfo" pod "0057b70d-3924-4e84-87f5-4cfc7e7babf5" (UID: "0057b70d-3924-4e84-87f5-4cfc7e7babf5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:15:22.170354 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:22.170306 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0057b70d-3924-4e84-87f5-4cfc7e7babf5-kube-api-access-zd5h2" pod "0057b70d-3924-4e84-87f5-4cfc7e7babf5" (UID: "0057b70d-3924-4e84-87f5-4cfc7e7babf5"). InnerVolumeSpecName "kube-api-access-zd5h2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:22.170506 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:22.170485 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0057b70d-3924-4e84-87f5-4cfc7e7babf5-istio-token" pod "0057b70d-3924-4e84-87f5-4cfc7e7babf5" (UID: "0057b70d-3924-4e84-87f5-4cfc7e7babf5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:22.270235 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:22.270201 5052 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0057b70d-3924-4e84-87f5-4cfc7e7babf5-istio-token\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:15:22.270235 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:22.270229 5052 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0057b70d-3924-4e84-87f5-4cfc7e7babf5-istio-envoy\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:15:22.270235 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:22.270239 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zd5h2\" (UniqueName: \"kubernetes.io/projected/0057b70d-3924-4e84-87f5-4cfc7e7babf5-kube-api-access-zd5h2\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:15:22.270468 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:22.270248 5052 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0057b70d-3924-4e84-87f5-4cfc7e7babf5-istio-podinfo\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:15:22.490711 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:22.490680 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-g6lll"] Oct 30 17:15:22.503896 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:22.503868 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-g6lll"] Oct 30 17:15:22.600141 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:22.600109 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0057b70d-3924-4e84-87f5-4cfc7e7babf5" path="/var/lib/kubelet/pods/0057b70d-3924-4e84-87f5-4cfc7e7babf5/volumes" Oct 30 17:15:22.600663 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:22.600643 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4036d90-5e91-477e-b81f-91c2dceee6c9" path="/var/lib/kubelet/pods/e4036d90-5e91-477e-b81f-91c2dceee6c9/volumes" Oct 30 17:15:23.224504 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:23.224471 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-dc74d5-5d4d946fcf-hnnsn"] Oct 30 17:15:23.224934 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:23.224688 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-dc74d5-5d4d946fcf-hnnsn" podUID="552a6bdf-c418-47bd-b72b-20e311760c33" containerName="maas-api" containerID="cri-o://14301e35732ec99a59f5f5454559d03c59d1a96eaf16724b574b719757cdd933" gracePeriod=30 Oct 30 17:15:23.558435 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:23.558342 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-dc74d5-5d4d946fcf-hnnsn" Oct 30 17:15:23.593124 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:23.593097 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/552a6bdf-c418-47bd-b72b-20e311760c33-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/552a6bdf-c418-47bd-b72b-20e311760c33-openshift-service-ca\") pod \"552a6bdf-c418-47bd-b72b-20e311760c33\" (UID: \"552a6bdf-c418-47bd-b72b-20e311760c33\") " Oct 30 17:15:23.593284 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:23.593163 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/552a6bdf-c418-47bd-b72b-20e311760c33-kube-api-access-2gjjv\" (UniqueName: \"kubernetes.io/projected/552a6bdf-c418-47bd-b72b-20e311760c33-kube-api-access-2gjjv\") pod \"552a6bdf-c418-47bd-b72b-20e311760c33\" (UID: \"552a6bdf-c418-47bd-b72b-20e311760c33\") " Oct 30 17:15:23.593284 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:23.593203 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/552a6bdf-c418-47bd-b72b-20e311760c33-metrics-certs\" (UniqueName: \"kubernetes.io/secret/552a6bdf-c418-47bd-b72b-20e311760c33-metrics-certs\") pod \"552a6bdf-c418-47bd-b72b-20e311760c33\" (UID: \"552a6bdf-c418-47bd-b72b-20e311760c33\") " Oct 30 17:15:23.593284 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:23.593247 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/552a6bdf-c418-47bd-b72b-20e311760c33-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/552a6bdf-c418-47bd-b72b-20e311760c33-maas-api-tls\") pod \"552a6bdf-c418-47bd-b72b-20e311760c33\" (UID: \"552a6bdf-c418-47bd-b72b-20e311760c33\") " Oct 30 17:15:23.593442 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:23.593416 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/552a6bdf-c418-47bd-b72b-20e311760c33-openshift-service-ca" pod "552a6bdf-c418-47bd-b72b-20e311760c33" (UID: "552a6bdf-c418-47bd-b72b-20e311760c33"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:15:23.593678 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:23.593650 5052 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/552a6bdf-c418-47bd-b72b-20e311760c33-openshift-service-ca\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:15:23.595381 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:23.595356 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/552a6bdf-c418-47bd-b72b-20e311760c33-metrics-certs" pod "552a6bdf-c418-47bd-b72b-20e311760c33" (UID: "552a6bdf-c418-47bd-b72b-20e311760c33"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:15:23.595486 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:23.595470 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/552a6bdf-c418-47bd-b72b-20e311760c33-kube-api-access-2gjjv" pod "552a6bdf-c418-47bd-b72b-20e311760c33" (UID: "552a6bdf-c418-47bd-b72b-20e311760c33"). InnerVolumeSpecName "kube-api-access-2gjjv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:23.595613 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:23.595575 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/552a6bdf-c418-47bd-b72b-20e311760c33-maas-api-tls" pod "552a6bdf-c418-47bd-b72b-20e311760c33" (UID: "552a6bdf-c418-47bd-b72b-20e311760c33"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:15:23.695632 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:23.695420 5052 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/552a6bdf-c418-47bd-b72b-20e311760c33-maas-api-tls\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:15:23.695632 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:23.695454 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gjjv\" (UniqueName: \"kubernetes.io/projected/552a6bdf-c418-47bd-b72b-20e311760c33-kube-api-access-2gjjv\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:15:23.695632 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:23.695468 5052 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/552a6bdf-c418-47bd-b72b-20e311760c33-metrics-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:15:24.003038 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:24.002992 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-dc74d5-slj4g"] Oct 30 17:15:24.006016 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:24.005993 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-dc74d5-slj4g"] Oct 30 17:15:24.151025 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:24.150986 5052 generic.go:356] "Generic (PLEG): container finished" podID="552a6bdf-c418-47bd-b72b-20e311760c33" containerID="14301e35732ec99a59f5f5454559d03c59d1a96eaf16724b574b719757cdd933" exitCode=0 Oct 30 17:15:24.151176 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:24.151041 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-dc74d5-5d4d946fcf-hnnsn" Oct 30 17:15:24.151176 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:24.151066 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-dc74d5-5d4d946fcf-hnnsn" event={"ID":"552a6bdf-c418-47bd-b72b-20e311760c33","Type":"ContainerDied","Data":"14301e35732ec99a59f5f5454559d03c59d1a96eaf16724b574b719757cdd933"} Oct 30 17:15:24.151176 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:24.151125 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-dc74d5-5d4d946fcf-hnnsn" event={"ID":"552a6bdf-c418-47bd-b72b-20e311760c33","Type":"ContainerDied","Data":"efd05df02fd826768d6271fe66b4acf5f3ebadde31d5d13b66ca4e1d0a3f10e2"} Oct 30 17:15:24.151176 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:24.151151 5052 scope.go:122] "RemoveContainer" containerID="14301e35732ec99a59f5f5454559d03c59d1a96eaf16724b574b719757cdd933" Oct 30 17:15:24.162334 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:24.162313 5052 scope.go:122] "RemoveContainer" containerID="14301e35732ec99a59f5f5454559d03c59d1a96eaf16724b574b719757cdd933" Oct 30 17:15:24.162638 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:15:24.162573 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14301e35732ec99a59f5f5454559d03c59d1a96eaf16724b574b719757cdd933\": container with ID starting with 14301e35732ec99a59f5f5454559d03c59d1a96eaf16724b574b719757cdd933 not found: ID does not exist" containerID="14301e35732ec99a59f5f5454559d03c59d1a96eaf16724b574b719757cdd933" Oct 30 17:15:24.162638 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:24.162615 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14301e35732ec99a59f5f5454559d03c59d1a96eaf16724b574b719757cdd933"} err="failed to get container status \"14301e35732ec99a59f5f5454559d03c59d1a96eaf16724b574b719757cdd933\": rpc error: code = NotFound desc = could not find container \"14301e35732ec99a59f5f5454559d03c59d1a96eaf16724b574b719757cdd933\": container with ID starting with 14301e35732ec99a59f5f5454559d03c59d1a96eaf16724b574b719757cdd933 not found: ID does not exist" Oct 30 17:15:24.207802 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:24.207610 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-dc74d5-5d4d946fcf-hnnsn"] Oct 30 17:15:24.211132 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:24.211110 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-dc74d5-5d4d946fcf-hnnsn"] Oct 30 17:15:24.605406 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:24.605373 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3cf66546-2340-4526-b4e2-9aebde2a705b" path="/var/lib/kubelet/pods/3cf66546-2340-4526-b4e2-9aebde2a705b/volumes" Oct 30 17:15:24.605824 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:24.605809 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="552a6bdf-c418-47bd-b72b-20e311760c33" path="/var/lib/kubelet/pods/552a6bdf-c418-47bd-b72b-20e311760c33/volumes" Oct 30 17:15:27.634926 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:27.634890 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3ed10a-6dcb868f5-2xwbx" podUID="b4d2755a-efbb-41f5-983f-309ee1cff113" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:15:29.106890 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:29.106857 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ddc10d-858465d86c-khzjm" Oct 30 17:15:29.107284 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:29.106942 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-ddc10d-99f47dfb8-w7ktd" Oct 30 17:15:36.842953 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:36.842924 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-4ptgl_973d1ea4-3981-4350-b719-edb7911c23ab/payload-processing/0.log" Oct 30 17:15:37.041065 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:37.041036 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-4ptgl_973d1ea4-3981-4350-b719-edb7911c23ab/payload-processing/0.log" Oct 30 17:15:37.232142 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:37.232109 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-ddc10d-99f47dfb8-w7ktd_b774d2ea-e597-4122-b520-7ccdec91972f/payload-processing/0.log" Oct 30 17:15:37.328953 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:37.328909 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-qq9qb"] Oct 30 17:15:37.329157 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:37.329137 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-qq9qb" podUID="f571ed90-9a31-47d2-8c4b-9bc57a9975b4" containerName="istio-proxy" containerID="cri-o://a5b140946fb5276862c19f77e56a6bbd4bb9251ce4893f5c6cea9e11d34476b3" gracePeriod=30 Oct 30 17:15:37.634170 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:37.634124 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3ed10a-6dcb868f5-2xwbx" podUID="b4d2755a-efbb-41f5-983f-309ee1cff113" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:15:38.558269 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:38.558230 5052 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-qq9qb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 17:15:38.558682 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:38.558287 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-qq9qb" podUID="f571ed90-9a31-47d2-8c4b-9bc57a9975b4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 17:15:38.850827 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:38.850793 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3ed10a-6dcb868f5-2xwbx" Oct 30 17:15:39.033351 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:39.033315 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4d2755a-efbb-41f5-983f-309ee1cff113-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4d2755a-efbb-41f5-983f-309ee1cff113-plugins-config-volume\") pod \"b4d2755a-efbb-41f5-983f-309ee1cff113\" (UID: \"b4d2755a-efbb-41f5-983f-309ee1cff113\") " Oct 30 17:15:39.033564 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:39.033545 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4d2755a-efbb-41f5-983f-309ee1cff113-kube-api-access-whqpx\" (UniqueName: \"kubernetes.io/projected/b4d2755a-efbb-41f5-983f-309ee1cff113-kube-api-access-whqpx\") pod \"b4d2755a-efbb-41f5-983f-309ee1cff113\" (UID: \"b4d2755a-efbb-41f5-983f-309ee1cff113\") " Oct 30 17:15:39.033802 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:39.033779 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4d2755a-efbb-41f5-983f-309ee1cff113-plugins-config-volume" pod "b4d2755a-efbb-41f5-983f-309ee1cff113" (UID: "b4d2755a-efbb-41f5-983f-309ee1cff113"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:15:39.034029 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:39.034008 5052 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4d2755a-efbb-41f5-983f-309ee1cff113-plugins-config-volume\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:15:39.035761 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:39.035729 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4d2755a-efbb-41f5-983f-309ee1cff113-kube-api-access-whqpx" pod "b4d2755a-efbb-41f5-983f-309ee1cff113" (UID: "b4d2755a-efbb-41f5-983f-309ee1cff113"). InnerVolumeSpecName "kube-api-access-whqpx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:39.135103 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:39.135029 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whqpx\" (UniqueName: \"kubernetes.io/projected/b4d2755a-efbb-41f5-983f-309ee1cff113-kube-api-access-whqpx\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:15:39.263910 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:39.263878 5052 generic.go:356] "Generic (PLEG): container finished" podID="b4d2755a-efbb-41f5-983f-309ee1cff113" containerID="510c0776948b928bb418a1d124f9c173d33a14c4345219c0fc13a1b771033a89" exitCode=0 Oct 30 17:15:39.264112 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:39.263926 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3ed10a-6dcb868f5-2xwbx" Oct 30 17:15:39.264112 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:39.263967 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3ed10a-6dcb868f5-2xwbx" event={"ID":"b4d2755a-efbb-41f5-983f-309ee1cff113","Type":"ContainerDied","Data":"510c0776948b928bb418a1d124f9c173d33a14c4345219c0fc13a1b771033a89"} Oct 30 17:15:39.264112 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:39.264023 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3ed10a-6dcb868f5-2xwbx" event={"ID":"b4d2755a-efbb-41f5-983f-309ee1cff113","Type":"ContainerDied","Data":"da93b69d4ad31ee5ff02294a691e0d2187b1e74bcf1b71dc54b58756b5e8dc89"} Oct 30 17:15:39.264112 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:39.264053 5052 scope.go:122] "RemoveContainer" containerID="510c0776948b928bb418a1d124f9c173d33a14c4345219c0fc13a1b771033a89" Oct 30 17:15:39.275262 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:39.275246 5052 scope.go:122] "RemoveContainer" containerID="510c0776948b928bb418a1d124f9c173d33a14c4345219c0fc13a1b771033a89" Oct 30 17:15:39.275530 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:15:39.275512 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"510c0776948b928bb418a1d124f9c173d33a14c4345219c0fc13a1b771033a89\": container with ID starting with 510c0776948b928bb418a1d124f9c173d33a14c4345219c0fc13a1b771033a89 not found: ID does not exist" containerID="510c0776948b928bb418a1d124f9c173d33a14c4345219c0fc13a1b771033a89" Oct 30 17:15:39.275605 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:39.275538 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"510c0776948b928bb418a1d124f9c173d33a14c4345219c0fc13a1b771033a89"} err="failed to get container status \"510c0776948b928bb418a1d124f9c173d33a14c4345219c0fc13a1b771033a89\": rpc error: code = NotFound desc = could not find container \"510c0776948b928bb418a1d124f9c173d33a14c4345219c0fc13a1b771033a89\": container with ID starting with 510c0776948b928bb418a1d124f9c173d33a14c4345219c0fc13a1b771033a89 not found: ID does not exist" Oct 30 17:15:39.320038 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:39.320015 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3ed10a-6dcb868f5-2xwbx"] Oct 30 17:15:39.333307 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:39.333287 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3ed10a-6dcb868f5-2xwbx"] Oct 30 17:15:40.598413 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:40.598373 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4d2755a-efbb-41f5-983f-309ee1cff113" path="/var/lib/kubelet/pods/b4d2755a-efbb-41f5-983f-309ee1cff113/volumes" Oct 30 17:15:42.657681 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:42.657652 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-qq9qb" Oct 30 17:15:42.797391 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:42.797346 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f571ed90-9a31-47d2-8c4b-9bc57a9975b4-istio-token\" (UniqueName: \"kubernetes.io/projected/f571ed90-9a31-47d2-8c4b-9bc57a9975b4-istio-token\") pod \"f571ed90-9a31-47d2-8c4b-9bc57a9975b4\" (UID: \"f571ed90-9a31-47d2-8c4b-9bc57a9975b4\") " Oct 30 17:15:42.797625 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:42.797428 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f571ed90-9a31-47d2-8c4b-9bc57a9975b4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f571ed90-9a31-47d2-8c4b-9bc57a9975b4-workload-socket\") pod \"f571ed90-9a31-47d2-8c4b-9bc57a9975b4\" (UID: \"f571ed90-9a31-47d2-8c4b-9bc57a9975b4\") " Oct 30 17:15:42.797625 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:42.797481 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f571ed90-9a31-47d2-8c4b-9bc57a9975b4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f571ed90-9a31-47d2-8c4b-9bc57a9975b4-istiod-ca-cert\") pod \"f571ed90-9a31-47d2-8c4b-9bc57a9975b4\" (UID: \"f571ed90-9a31-47d2-8c4b-9bc57a9975b4\") " Oct 30 17:15:42.797970 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:42.797553 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f571ed90-9a31-47d2-8c4b-9bc57a9975b4-kube-api-access-dql4s\" (UniqueName: \"kubernetes.io/projected/f571ed90-9a31-47d2-8c4b-9bc57a9975b4-kube-api-access-dql4s\") pod \"f571ed90-9a31-47d2-8c4b-9bc57a9975b4\" (UID: \"f571ed90-9a31-47d2-8c4b-9bc57a9975b4\") " Oct 30 17:15:42.798063 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:42.797976 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f571ed90-9a31-47d2-8c4b-9bc57a9975b4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f571ed90-9a31-47d2-8c4b-9bc57a9975b4-credential-socket\") pod \"f571ed90-9a31-47d2-8c4b-9bc57a9975b4\" (UID: \"f571ed90-9a31-47d2-8c4b-9bc57a9975b4\") " Oct 30 17:15:42.798063 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:42.797808 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f571ed90-9a31-47d2-8c4b-9bc57a9975b4-workload-socket" pod "f571ed90-9a31-47d2-8c4b-9bc57a9975b4" (UID: "f571ed90-9a31-47d2-8c4b-9bc57a9975b4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:15:42.798063 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:42.797906 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f571ed90-9a31-47d2-8c4b-9bc57a9975b4-istiod-ca-cert" pod "f571ed90-9a31-47d2-8c4b-9bc57a9975b4" (UID: "f571ed90-9a31-47d2-8c4b-9bc57a9975b4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:15:42.798249 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:42.798105 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f571ed90-9a31-47d2-8c4b-9bc57a9975b4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f571ed90-9a31-47d2-8c4b-9bc57a9975b4-istio-data\") pod \"f571ed90-9a31-47d2-8c4b-9bc57a9975b4\" (UID: \"f571ed90-9a31-47d2-8c4b-9bc57a9975b4\") " Oct 30 17:15:42.798249 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:42.798158 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f571ed90-9a31-47d2-8c4b-9bc57a9975b4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f571ed90-9a31-47d2-8c4b-9bc57a9975b4-istio-envoy\") pod \"f571ed90-9a31-47d2-8c4b-9bc57a9975b4\" (UID: \"f571ed90-9a31-47d2-8c4b-9bc57a9975b4\") " Oct 30 17:15:42.798361 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:42.798287 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f571ed90-9a31-47d2-8c4b-9bc57a9975b4-credential-socket" pod "f571ed90-9a31-47d2-8c4b-9bc57a9975b4" (UID: "f571ed90-9a31-47d2-8c4b-9bc57a9975b4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:15:42.798361 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:42.798328 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f571ed90-9a31-47d2-8c4b-9bc57a9975b4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f571ed90-9a31-47d2-8c4b-9bc57a9975b4-workload-certs\") pod \"f571ed90-9a31-47d2-8c4b-9bc57a9975b4\" (UID: \"f571ed90-9a31-47d2-8c4b-9bc57a9975b4\") " Oct 30 17:15:42.798471 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:42.798386 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f571ed90-9a31-47d2-8c4b-9bc57a9975b4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f571ed90-9a31-47d2-8c4b-9bc57a9975b4-istio-podinfo\") pod \"f571ed90-9a31-47d2-8c4b-9bc57a9975b4\" (UID: \"f571ed90-9a31-47d2-8c4b-9bc57a9975b4\") " Oct 30 17:15:42.798628 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:42.798604 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f571ed90-9a31-47d2-8c4b-9bc57a9975b4-workload-certs" pod "f571ed90-9a31-47d2-8c4b-9bc57a9975b4" (UID: "f571ed90-9a31-47d2-8c4b-9bc57a9975b4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:15:42.798973 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:42.798950 5052 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f571ed90-9a31-47d2-8c4b-9bc57a9975b4-workload-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:15:42.799057 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:42.798980 5052 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f571ed90-9a31-47d2-8c4b-9bc57a9975b4-workload-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:15:42.799057 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:42.798995 5052 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f571ed90-9a31-47d2-8c4b-9bc57a9975b4-istiod-ca-cert\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:15:42.799057 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:42.799008 5052 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f571ed90-9a31-47d2-8c4b-9bc57a9975b4-credential-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:15:42.799231 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:42.799046 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f571ed90-9a31-47d2-8c4b-9bc57a9975b4-istio-data" pod "f571ed90-9a31-47d2-8c4b-9bc57a9975b4" (UID: "f571ed90-9a31-47d2-8c4b-9bc57a9975b4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:15:42.800127 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:42.800100 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f571ed90-9a31-47d2-8c4b-9bc57a9975b4-istio-token" pod "f571ed90-9a31-47d2-8c4b-9bc57a9975b4" (UID: "f571ed90-9a31-47d2-8c4b-9bc57a9975b4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:42.800781 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:42.800749 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f571ed90-9a31-47d2-8c4b-9bc57a9975b4-istio-podinfo" pod "f571ed90-9a31-47d2-8c4b-9bc57a9975b4" (UID: "f571ed90-9a31-47d2-8c4b-9bc57a9975b4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:15:42.800781 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:42.800753 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f571ed90-9a31-47d2-8c4b-9bc57a9975b4-kube-api-access-dql4s" pod "f571ed90-9a31-47d2-8c4b-9bc57a9975b4" (UID: "f571ed90-9a31-47d2-8c4b-9bc57a9975b4"). InnerVolumeSpecName "kube-api-access-dql4s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:42.801175 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:42.801158 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f571ed90-9a31-47d2-8c4b-9bc57a9975b4-istio-envoy" pod "f571ed90-9a31-47d2-8c4b-9bc57a9975b4" (UID: "f571ed90-9a31-47d2-8c4b-9bc57a9975b4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:15:42.902887 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:42.902706 5052 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f571ed90-9a31-47d2-8c4b-9bc57a9975b4-istio-podinfo\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:15:42.902887 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:42.902748 5052 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f571ed90-9a31-47d2-8c4b-9bc57a9975b4-istio-token\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:15:42.902887 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:42.902762 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dql4s\" (UniqueName: \"kubernetes.io/projected/f571ed90-9a31-47d2-8c4b-9bc57a9975b4-kube-api-access-dql4s\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:15:42.902887 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:42.902777 5052 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f571ed90-9a31-47d2-8c4b-9bc57a9975b4-istio-data\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:15:42.902887 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:42.902793 5052 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f571ed90-9a31-47d2-8c4b-9bc57a9975b4-istio-envoy\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:15:43.295089 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:43.295002 5052 generic.go:356] "Generic (PLEG): container finished" podID="f571ed90-9a31-47d2-8c4b-9bc57a9975b4" containerID="a5b140946fb5276862c19f77e56a6bbd4bb9251ce4893f5c6cea9e11d34476b3" exitCode=0 Oct 30 17:15:43.295089 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:43.295046 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-qq9qb" event={"ID":"f571ed90-9a31-47d2-8c4b-9bc57a9975b4","Type":"ContainerDied","Data":"a5b140946fb5276862c19f77e56a6bbd4bb9251ce4893f5c6cea9e11d34476b3"} Oct 30 17:15:43.295089 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:43.295066 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-qq9qb" Oct 30 17:15:43.295354 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:43.295101 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-qq9qb" event={"ID":"f571ed90-9a31-47d2-8c4b-9bc57a9975b4","Type":"ContainerDied","Data":"b79e8f86366e86afc44f9006fbdbc47d81debcc7c643a90905be645ad1adf857"} Oct 30 17:15:43.295354 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:43.295125 5052 scope.go:122] "RemoveContainer" containerID="a5b140946fb5276862c19f77e56a6bbd4bb9251ce4893f5c6cea9e11d34476b3" Oct 30 17:15:43.310159 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:43.307813 5052 scope.go:122] "RemoveContainer" containerID="a5b140946fb5276862c19f77e56a6bbd4bb9251ce4893f5c6cea9e11d34476b3" Oct 30 17:15:43.310259 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:15:43.310206 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5b140946fb5276862c19f77e56a6bbd4bb9251ce4893f5c6cea9e11d34476b3\": container with ID starting with a5b140946fb5276862c19f77e56a6bbd4bb9251ce4893f5c6cea9e11d34476b3 not found: ID does not exist" containerID="a5b140946fb5276862c19f77e56a6bbd4bb9251ce4893f5c6cea9e11d34476b3" Oct 30 17:15:43.310259 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:43.310236 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5b140946fb5276862c19f77e56a6bbd4bb9251ce4893f5c6cea9e11d34476b3"} err="failed to get container status \"a5b140946fb5276862c19f77e56a6bbd4bb9251ce4893f5c6cea9e11d34476b3\": rpc error: code = NotFound desc = could not find container \"a5b140946fb5276862c19f77e56a6bbd4bb9251ce4893f5c6cea9e11d34476b3\": container with ID starting with a5b140946fb5276862c19f77e56a6bbd4bb9251ce4893f5c6cea9e11d34476b3 not found: ID does not exist" Oct 30 17:15:43.365283 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:43.365253 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-qq9qb"] Oct 30 17:15:43.381043 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:43.381019 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-qq9qb"] Oct 30 17:15:43.640982 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:43.640887 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9j9rt"] Oct 30 17:15:43.641173 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:43.641121 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9j9rt" podUID="9da694ce-2f3b-465d-abe2-675a039d38c0" containerName="istio-proxy" containerID="cri-o://1f57818b297e08c26696f799e8d76487f06c074e4e83db6584c2d4206632b1b0" gracePeriod=30 Oct 30 17:15:44.598036 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:44.597997 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f571ed90-9a31-47d2-8c4b-9bc57a9975b4" path="/var/lib/kubelet/pods/f571ed90-9a31-47d2-8c4b-9bc57a9975b4/volumes" Oct 30 17:15:45.895429 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:45.895402 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4wrf8_4f0f46f4-5cda-4d67-b6f3-47df06eb844c/console-operator/2.log" Oct 30 17:15:45.899033 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:45.899010 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4wrf8_4f0f46f4-5cda-4d67-b6f3-47df06eb844c/console-operator/2.log" Oct 30 17:15:48.845511 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:15:48.845447 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 17:15:48.973403 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:48.973378 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9j9rt" Oct 30 17:15:49.005110 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:49.005085 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9da694ce-2f3b-465d-abe2-675a039d38c0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9da694ce-2f3b-465d-abe2-675a039d38c0-workload-socket\") pod \"9da694ce-2f3b-465d-abe2-675a039d38c0\" (UID: \"9da694ce-2f3b-465d-abe2-675a039d38c0\") " Oct 30 17:15:49.005235 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:49.005223 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9da694ce-2f3b-465d-abe2-675a039d38c0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9da694ce-2f3b-465d-abe2-675a039d38c0-istiod-ca-cert\") pod \"9da694ce-2f3b-465d-abe2-675a039d38c0\" (UID: \"9da694ce-2f3b-465d-abe2-675a039d38c0\") " Oct 30 17:15:49.005297 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:49.005245 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9da694ce-2f3b-465d-abe2-675a039d38c0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9da694ce-2f3b-465d-abe2-675a039d38c0-credential-socket\") pod \"9da694ce-2f3b-465d-abe2-675a039d38c0\" (UID: \"9da694ce-2f3b-465d-abe2-675a039d38c0\") " Oct 30 17:15:49.005297 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:49.005272 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9da694ce-2f3b-465d-abe2-675a039d38c0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9da694ce-2f3b-465d-abe2-675a039d38c0-istio-data\") pod \"9da694ce-2f3b-465d-abe2-675a039d38c0\" (UID: \"9da694ce-2f3b-465d-abe2-675a039d38c0\") " Oct 30 17:15:49.005395 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:49.005301 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9da694ce-2f3b-465d-abe2-675a039d38c0-kube-api-access-4nsbx\" (UniqueName: \"kubernetes.io/projected/9da694ce-2f3b-465d-abe2-675a039d38c0-kube-api-access-4nsbx\") pod \"9da694ce-2f3b-465d-abe2-675a039d38c0\" (UID: \"9da694ce-2f3b-465d-abe2-675a039d38c0\") " Oct 30 17:15:49.005395 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:49.005321 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9da694ce-2f3b-465d-abe2-675a039d38c0-workload-socket" pod "9da694ce-2f3b-465d-abe2-675a039d38c0" (UID: "9da694ce-2f3b-465d-abe2-675a039d38c0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:15:49.005395 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:49.005339 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9da694ce-2f3b-465d-abe2-675a039d38c0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9da694ce-2f3b-465d-abe2-675a039d38c0-istio-podinfo\") pod \"9da694ce-2f3b-465d-abe2-675a039d38c0\" (UID: \"9da694ce-2f3b-465d-abe2-675a039d38c0\") " Oct 30 17:15:49.005543 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:49.005408 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9da694ce-2f3b-465d-abe2-675a039d38c0-istio-token\" (UniqueName: \"kubernetes.io/projected/9da694ce-2f3b-465d-abe2-675a039d38c0-istio-token\") pod \"9da694ce-2f3b-465d-abe2-675a039d38c0\" (UID: \"9da694ce-2f3b-465d-abe2-675a039d38c0\") " Oct 30 17:15:49.005543 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:49.005438 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9da694ce-2f3b-465d-abe2-675a039d38c0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9da694ce-2f3b-465d-abe2-675a039d38c0-workload-certs\") pod \"9da694ce-2f3b-465d-abe2-675a039d38c0\" (UID: \"9da694ce-2f3b-465d-abe2-675a039d38c0\") " Oct 30 17:15:49.005665 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:49.005547 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9da694ce-2f3b-465d-abe2-675a039d38c0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9da694ce-2f3b-465d-abe2-675a039d38c0-istio-envoy\") pod \"9da694ce-2f3b-465d-abe2-675a039d38c0\" (UID: \"9da694ce-2f3b-465d-abe2-675a039d38c0\") " Oct 30 17:15:49.005665 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:49.005634 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9da694ce-2f3b-465d-abe2-675a039d38c0-credential-socket" pod "9da694ce-2f3b-465d-abe2-675a039d38c0" (UID: "9da694ce-2f3b-465d-abe2-675a039d38c0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:15:49.005745 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:49.005704 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9da694ce-2f3b-465d-abe2-675a039d38c0-istio-data" pod "9da694ce-2f3b-465d-abe2-675a039d38c0" (UID: "9da694ce-2f3b-465d-abe2-675a039d38c0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:15:49.005808 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:49.005768 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9da694ce-2f3b-465d-abe2-675a039d38c0-workload-certs" pod "9da694ce-2f3b-465d-abe2-675a039d38c0" (UID: "9da694ce-2f3b-465d-abe2-675a039d38c0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:15:49.005868 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:49.005838 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9da694ce-2f3b-465d-abe2-675a039d38c0-istiod-ca-cert" pod "9da694ce-2f3b-465d-abe2-675a039d38c0" (UID: "9da694ce-2f3b-465d-abe2-675a039d38c0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:15:49.006066 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:49.006050 5052 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9da694ce-2f3b-465d-abe2-675a039d38c0-istiod-ca-cert\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:15:49.006166 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:49.006067 5052 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9da694ce-2f3b-465d-abe2-675a039d38c0-credential-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:15:49.006166 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:49.006077 5052 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9da694ce-2f3b-465d-abe2-675a039d38c0-istio-data\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:15:49.006166 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:49.006090 5052 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9da694ce-2f3b-465d-abe2-675a039d38c0-workload-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:15:49.006166 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:49.006102 5052 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9da694ce-2f3b-465d-abe2-675a039d38c0-workload-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:15:49.007826 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:49.007799 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9da694ce-2f3b-465d-abe2-675a039d38c0-istio-podinfo" pod "9da694ce-2f3b-465d-abe2-675a039d38c0" (UID: "9da694ce-2f3b-465d-abe2-675a039d38c0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:15:49.007928 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:49.007867 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9da694ce-2f3b-465d-abe2-675a039d38c0-istio-token" pod "9da694ce-2f3b-465d-abe2-675a039d38c0" (UID: "9da694ce-2f3b-465d-abe2-675a039d38c0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:49.008037 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:49.008021 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9da694ce-2f3b-465d-abe2-675a039d38c0-istio-envoy" pod "9da694ce-2f3b-465d-abe2-675a039d38c0" (UID: "9da694ce-2f3b-465d-abe2-675a039d38c0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:15:49.008174 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:49.008159 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9da694ce-2f3b-465d-abe2-675a039d38c0-kube-api-access-4nsbx" pod "9da694ce-2f3b-465d-abe2-675a039d38c0" (UID: "9da694ce-2f3b-465d-abe2-675a039d38c0"). InnerVolumeSpecName "kube-api-access-4nsbx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:49.106953 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:49.106920 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nsbx\" (UniqueName: \"kubernetes.io/projected/9da694ce-2f3b-465d-abe2-675a039d38c0-kube-api-access-4nsbx\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:15:49.106953 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:49.106953 5052 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9da694ce-2f3b-465d-abe2-675a039d38c0-istio-podinfo\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:15:49.107163 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:49.106963 5052 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9da694ce-2f3b-465d-abe2-675a039d38c0-istio-token\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:15:49.107163 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:49.106972 5052 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9da694ce-2f3b-465d-abe2-675a039d38c0-istio-envoy\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:15:49.352496 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:49.352405 5052 generic.go:356] "Generic (PLEG): container finished" podID="9da694ce-2f3b-465d-abe2-675a039d38c0" containerID="1f57818b297e08c26696f799e8d76487f06c074e4e83db6584c2d4206632b1b0" exitCode=0 Oct 30 17:15:49.352496 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:49.352459 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9j9rt" Oct 30 17:15:49.352756 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:49.352486 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9j9rt" event={"ID":"9da694ce-2f3b-465d-abe2-675a039d38c0","Type":"ContainerDied","Data":"1f57818b297e08c26696f799e8d76487f06c074e4e83db6584c2d4206632b1b0"} Oct 30 17:15:49.352756 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:49.352546 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9j9rt" event={"ID":"9da694ce-2f3b-465d-abe2-675a039d38c0","Type":"ContainerDied","Data":"02f89ac4ab1b757f4c36cd9ea3d44cc02e3e6f12d742b11cef98349979901ed2"} Oct 30 17:15:49.352756 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:49.352570 5052 scope.go:122] "RemoveContainer" containerID="1f57818b297e08c26696f799e8d76487f06c074e4e83db6584c2d4206632b1b0" Oct 30 17:15:49.364241 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:49.364223 5052 scope.go:122] "RemoveContainer" containerID="1f57818b297e08c26696f799e8d76487f06c074e4e83db6584c2d4206632b1b0" Oct 30 17:15:49.364505 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:15:49.364485 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f57818b297e08c26696f799e8d76487f06c074e4e83db6584c2d4206632b1b0\": container with ID starting with 1f57818b297e08c26696f799e8d76487f06c074e4e83db6584c2d4206632b1b0 not found: ID does not exist" containerID="1f57818b297e08c26696f799e8d76487f06c074e4e83db6584c2d4206632b1b0" Oct 30 17:15:49.364557 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:49.364515 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f57818b297e08c26696f799e8d76487f06c074e4e83db6584c2d4206632b1b0"} err="failed to get container status \"1f57818b297e08c26696f799e8d76487f06c074e4e83db6584c2d4206632b1b0\": rpc error: code = NotFound desc = could not find container \"1f57818b297e08c26696f799e8d76487f06c074e4e83db6584c2d4206632b1b0\": container with ID starting with 1f57818b297e08c26696f799e8d76487f06c074e4e83db6584c2d4206632b1b0 not found: ID does not exist" Oct 30 17:15:49.409426 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:49.409402 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9j9rt"] Oct 30 17:15:49.422883 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:49.422860 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9j9rt"] Oct 30 17:15:50.598449 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:50.598414 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9da694ce-2f3b-465d-abe2-675a039d38c0" path="/var/lib/kubelet/pods/9da694ce-2f3b-465d-abe2-675a039d38c0/volumes" Oct 30 17:15:53.482665 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:53.482632 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-97a47d-84c9dc8988-jss9k"] Oct 30 17:15:53.483126 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:53.482851 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-97a47d-84c9dc8988-jss9k" podUID="3b174c28-3805-4b14-9547-cd003db3d27e" containerName="maas-api" containerID="cri-o://7dc247faa20fc01daf58d491375d944528dce23733f12c04efc9701136592eaa" gracePeriod=30 Oct 30 17:15:53.776075 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:53.776048 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-97a47d-84c9dc8988-jss9k" Oct 30 17:15:53.882537 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:53.882499 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b174c28-3805-4b14-9547-cd003db3d27e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b174c28-3805-4b14-9547-cd003db3d27e-metrics-certs\") pod \"3b174c28-3805-4b14-9547-cd003db3d27e\" (UID: \"3b174c28-3805-4b14-9547-cd003db3d27e\") " Oct 30 17:15:53.882728 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:53.882651 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b174c28-3805-4b14-9547-cd003db3d27e-kube-api-access-jlw5v\" (UniqueName: \"kubernetes.io/projected/3b174c28-3805-4b14-9547-cd003db3d27e-kube-api-access-jlw5v\") pod \"3b174c28-3805-4b14-9547-cd003db3d27e\" (UID: \"3b174c28-3805-4b14-9547-cd003db3d27e\") " Oct 30 17:15:53.882828 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:53.882754 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b174c28-3805-4b14-9547-cd003db3d27e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b174c28-3805-4b14-9547-cd003db3d27e-maas-api-tls\") pod \"3b174c28-3805-4b14-9547-cd003db3d27e\" (UID: \"3b174c28-3805-4b14-9547-cd003db3d27e\") " Oct 30 17:15:53.882828 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:53.882803 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b174c28-3805-4b14-9547-cd003db3d27e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b174c28-3805-4b14-9547-cd003db3d27e-openshift-service-ca\") pod \"3b174c28-3805-4b14-9547-cd003db3d27e\" (UID: \"3b174c28-3805-4b14-9547-cd003db3d27e\") " Oct 30 17:15:53.883253 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:53.883228 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b174c28-3805-4b14-9547-cd003db3d27e-openshift-service-ca" pod "3b174c28-3805-4b14-9547-cd003db3d27e" (UID: "3b174c28-3805-4b14-9547-cd003db3d27e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:15:53.884730 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:53.884710 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b174c28-3805-4b14-9547-cd003db3d27e-kube-api-access-jlw5v" pod "3b174c28-3805-4b14-9547-cd003db3d27e" (UID: "3b174c28-3805-4b14-9547-cd003db3d27e"). InnerVolumeSpecName "kube-api-access-jlw5v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:53.885031 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:53.885012 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b174c28-3805-4b14-9547-cd003db3d27e-metrics-certs" pod "3b174c28-3805-4b14-9547-cd003db3d27e" (UID: "3b174c28-3805-4b14-9547-cd003db3d27e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:15:53.885113 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:53.885028 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b174c28-3805-4b14-9547-cd003db3d27e-maas-api-tls" pod "3b174c28-3805-4b14-9547-cd003db3d27e" (UID: "3b174c28-3805-4b14-9547-cd003db3d27e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:15:53.984823 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:53.984721 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlw5v\" (UniqueName: \"kubernetes.io/projected/3b174c28-3805-4b14-9547-cd003db3d27e-kube-api-access-jlw5v\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:15:53.984823 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:53.984766 5052 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b174c28-3805-4b14-9547-cd003db3d27e-maas-api-tls\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:15:53.984823 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:53.984781 5052 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b174c28-3805-4b14-9547-cd003db3d27e-openshift-service-ca\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:15:53.984823 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:53.984794 5052 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b174c28-3805-4b14-9547-cd003db3d27e-metrics-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:15:54.394571 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:54.394530 5052 generic.go:356] "Generic (PLEG): container finished" podID="3b174c28-3805-4b14-9547-cd003db3d27e" containerID="7dc247faa20fc01daf58d491375d944528dce23733f12c04efc9701136592eaa" exitCode=0 Oct 30 17:15:54.394789 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:54.394601 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-97a47d-84c9dc8988-jss9k" Oct 30 17:15:54.394789 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:54.394615 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-97a47d-84c9dc8988-jss9k" event={"ID":"3b174c28-3805-4b14-9547-cd003db3d27e","Type":"ContainerDied","Data":"7dc247faa20fc01daf58d491375d944528dce23733f12c04efc9701136592eaa"} Oct 30 17:15:54.394789 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:54.394706 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-97a47d-84c9dc8988-jss9k" event={"ID":"3b174c28-3805-4b14-9547-cd003db3d27e","Type":"ContainerDied","Data":"921ac23ddbef8c711078fdd4ba723c8b9f21f75677a3162f54b9a13fcb93497b"} Oct 30 17:15:54.394789 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:54.394735 5052 scope.go:122] "RemoveContainer" containerID="7dc247faa20fc01daf58d491375d944528dce23733f12c04efc9701136592eaa" Oct 30 17:15:54.405265 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:54.405240 5052 scope.go:122] "RemoveContainer" containerID="7dc247faa20fc01daf58d491375d944528dce23733f12c04efc9701136592eaa" Oct 30 17:15:54.405546 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:15:54.405520 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7dc247faa20fc01daf58d491375d944528dce23733f12c04efc9701136592eaa\": container with ID starting with 7dc247faa20fc01daf58d491375d944528dce23733f12c04efc9701136592eaa not found: ID does not exist" containerID="7dc247faa20fc01daf58d491375d944528dce23733f12c04efc9701136592eaa" Oct 30 17:15:54.405623 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:54.405547 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dc247faa20fc01daf58d491375d944528dce23733f12c04efc9701136592eaa"} err="failed to get container status \"7dc247faa20fc01daf58d491375d944528dce23733f12c04efc9701136592eaa\": rpc error: code = NotFound desc = could not find container \"7dc247faa20fc01daf58d491375d944528dce23733f12c04efc9701136592eaa\": container with ID starting with 7dc247faa20fc01daf58d491375d944528dce23733f12c04efc9701136592eaa not found: ID does not exist" Oct 30 17:15:54.451141 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:54.451109 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-97a47d-84c9dc8988-jss9k"] Oct 30 17:15:54.463275 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:54.463248 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-97a47d-84c9dc8988-jss9k"] Oct 30 17:15:54.598098 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:54.598066 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b174c28-3805-4b14-9547-cd003db3d27e" path="/var/lib/kubelet/pods/3b174c28-3805-4b14-9547-cd003db3d27e/volumes" Oct 30 17:15:55.362792 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:55.362760 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-0ae3fe-openshift-default-69fdb67df-sbgql"] Oct 30 17:15:55.363028 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:55.362996 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-0ae3fe-openshift-default-69fdb67df-sbgql" podUID="666e4b84-2d87-4557-a8b5-10f890a1740b" containerName="istio-proxy" containerID="cri-o://5e1bb34dcfa3906b934a7cf76809c8d2ad4cf7085c32d857441401563a4ec18a" gracePeriod=30 Oct 30 17:15:58.370877 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:15:58.370844 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cdwcj"] Oct 30 17:16:00.640057 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:00.640002 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-0ae3fe-openshift-default-69fdb67df-sbgql" Oct 30 17:16:00.690905 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:00.690876 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/666e4b84-2d87-4557-a8b5-10f890a1740b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/666e4b84-2d87-4557-a8b5-10f890a1740b-workload-socket\") pod \"666e4b84-2d87-4557-a8b5-10f890a1740b\" (UID: \"666e4b84-2d87-4557-a8b5-10f890a1740b\") " Oct 30 17:16:00.691059 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:00.690937 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/666e4b84-2d87-4557-a8b5-10f890a1740b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/666e4b84-2d87-4557-a8b5-10f890a1740b-istiod-ca-cert\") pod \"666e4b84-2d87-4557-a8b5-10f890a1740b\" (UID: \"666e4b84-2d87-4557-a8b5-10f890a1740b\") " Oct 30 17:16:00.691059 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:00.691022 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/666e4b84-2d87-4557-a8b5-10f890a1740b-kube-api-access-lbpnc\" (UniqueName: \"kubernetes.io/projected/666e4b84-2d87-4557-a8b5-10f890a1740b-kube-api-access-lbpnc\") pod \"666e4b84-2d87-4557-a8b5-10f890a1740b\" (UID: \"666e4b84-2d87-4557-a8b5-10f890a1740b\") " Oct 30 17:16:00.691145 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:00.691124 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/666e4b84-2d87-4557-a8b5-10f890a1740b-workload-socket" pod "666e4b84-2d87-4557-a8b5-10f890a1740b" (UID: "666e4b84-2d87-4557-a8b5-10f890a1740b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:16:00.691207 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:00.691184 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/666e4b84-2d87-4557-a8b5-10f890a1740b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/666e4b84-2d87-4557-a8b5-10f890a1740b-credential-socket\") pod \"666e4b84-2d87-4557-a8b5-10f890a1740b\" (UID: \"666e4b84-2d87-4557-a8b5-10f890a1740b\") " Oct 30 17:16:00.691311 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:00.691295 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/666e4b84-2d87-4557-a8b5-10f890a1740b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/666e4b84-2d87-4557-a8b5-10f890a1740b-workload-certs\") pod \"666e4b84-2d87-4557-a8b5-10f890a1740b\" (UID: \"666e4b84-2d87-4557-a8b5-10f890a1740b\") " Oct 30 17:16:00.691382 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:00.691330 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/666e4b84-2d87-4557-a8b5-10f890a1740b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/666e4b84-2d87-4557-a8b5-10f890a1740b-istio-envoy\") pod \"666e4b84-2d87-4557-a8b5-10f890a1740b\" (UID: \"666e4b84-2d87-4557-a8b5-10f890a1740b\") " Oct 30 17:16:00.691382 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:00.691364 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/666e4b84-2d87-4557-a8b5-10f890a1740b-credential-socket" pod "666e4b84-2d87-4557-a8b5-10f890a1740b" (UID: "666e4b84-2d87-4557-a8b5-10f890a1740b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:16:00.691486 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:00.691414 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/666e4b84-2d87-4557-a8b5-10f890a1740b-istiod-ca-cert" pod "666e4b84-2d87-4557-a8b5-10f890a1740b" (UID: "666e4b84-2d87-4557-a8b5-10f890a1740b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:16:00.691486 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:00.691430 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/666e4b84-2d87-4557-a8b5-10f890a1740b-istio-token\" (UniqueName: \"kubernetes.io/projected/666e4b84-2d87-4557-a8b5-10f890a1740b-istio-token\") pod \"666e4b84-2d87-4557-a8b5-10f890a1740b\" (UID: \"666e4b84-2d87-4557-a8b5-10f890a1740b\") " Oct 30 17:16:00.691486 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:00.691466 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/666e4b84-2d87-4557-a8b5-10f890a1740b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/666e4b84-2d87-4557-a8b5-10f890a1740b-istio-podinfo\") pod \"666e4b84-2d87-4557-a8b5-10f890a1740b\" (UID: \"666e4b84-2d87-4557-a8b5-10f890a1740b\") " Oct 30 17:16:00.691669 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:00.691501 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/666e4b84-2d87-4557-a8b5-10f890a1740b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/666e4b84-2d87-4557-a8b5-10f890a1740b-istio-data\") pod \"666e4b84-2d87-4557-a8b5-10f890a1740b\" (UID: \"666e4b84-2d87-4557-a8b5-10f890a1740b\") " Oct 30 17:16:00.691669 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:00.691502 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/666e4b84-2d87-4557-a8b5-10f890a1740b-workload-certs" pod "666e4b84-2d87-4557-a8b5-10f890a1740b" (UID: "666e4b84-2d87-4557-a8b5-10f890a1740b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:16:00.691837 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:00.691811 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/666e4b84-2d87-4557-a8b5-10f890a1740b-istio-data" pod "666e4b84-2d87-4557-a8b5-10f890a1740b" (UID: "666e4b84-2d87-4557-a8b5-10f890a1740b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:16:00.692265 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:00.692237 5052 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/666e4b84-2d87-4557-a8b5-10f890a1740b-istio-data\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:16:00.692397 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:00.692268 5052 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/666e4b84-2d87-4557-a8b5-10f890a1740b-workload-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:16:00.692397 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:00.692284 5052 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/666e4b84-2d87-4557-a8b5-10f890a1740b-istiod-ca-cert\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:16:00.692397 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:00.692300 5052 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/666e4b84-2d87-4557-a8b5-10f890a1740b-credential-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:16:00.692397 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:00.692314 5052 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/666e4b84-2d87-4557-a8b5-10f890a1740b-workload-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:16:00.693500 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:00.693475 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/666e4b84-2d87-4557-a8b5-10f890a1740b-kube-api-access-lbpnc" pod "666e4b84-2d87-4557-a8b5-10f890a1740b" (UID: "666e4b84-2d87-4557-a8b5-10f890a1740b"). InnerVolumeSpecName "kube-api-access-lbpnc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:16:00.693642 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:00.693557 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/666e4b84-2d87-4557-a8b5-10f890a1740b-istio-podinfo" pod "666e4b84-2d87-4557-a8b5-10f890a1740b" (UID: "666e4b84-2d87-4557-a8b5-10f890a1740b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:16:00.693703 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:00.693636 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/666e4b84-2d87-4557-a8b5-10f890a1740b-istio-envoy" pod "666e4b84-2d87-4557-a8b5-10f890a1740b" (UID: "666e4b84-2d87-4557-a8b5-10f890a1740b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:16:00.693746 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:00.693706 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/666e4b84-2d87-4557-a8b5-10f890a1740b-istio-token" pod "666e4b84-2d87-4557-a8b5-10f890a1740b" (UID: "666e4b84-2d87-4557-a8b5-10f890a1740b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:16:00.793723 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:00.793688 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbpnc\" (UniqueName: \"kubernetes.io/projected/666e4b84-2d87-4557-a8b5-10f890a1740b-kube-api-access-lbpnc\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:16:00.793723 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:00.793717 5052 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/666e4b84-2d87-4557-a8b5-10f890a1740b-istio-envoy\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:16:00.793723 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:00.793726 5052 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/666e4b84-2d87-4557-a8b5-10f890a1740b-istio-token\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:16:00.794020 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:00.793735 5052 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/666e4b84-2d87-4557-a8b5-10f890a1740b-istio-podinfo\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:16:01.441621 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:01.441572 5052 generic.go:356] "Generic (PLEG): container finished" podID="666e4b84-2d87-4557-a8b5-10f890a1740b" containerID="5e1bb34dcfa3906b934a7cf76809c8d2ad4cf7085c32d857441401563a4ec18a" exitCode=0 Oct 30 17:16:01.441819 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:01.441628 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-0ae3fe-openshift-default-69fdb67df-sbgql" Oct 30 17:16:01.441819 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:01.441620 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0ae3fe-openshift-default-69fdb67df-sbgql" event={"ID":"666e4b84-2d87-4557-a8b5-10f890a1740b","Type":"ContainerDied","Data":"5e1bb34dcfa3906b934a7cf76809c8d2ad4cf7085c32d857441401563a4ec18a"} Oct 30 17:16:01.441819 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:01.441754 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0ae3fe-openshift-default-69fdb67df-sbgql" event={"ID":"666e4b84-2d87-4557-a8b5-10f890a1740b","Type":"ContainerDied","Data":"d97758c7a1bed262c7b4b3707d838af14ceaa224a0f7b30f16228cdf98e6ec14"} Oct 30 17:16:01.441819 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:01.441787 5052 scope.go:122] "RemoveContainer" containerID="5e1bb34dcfa3906b934a7cf76809c8d2ad4cf7085c32d857441401563a4ec18a" Oct 30 17:16:01.453330 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:01.453308 5052 scope.go:122] "RemoveContainer" containerID="3e8dd9f08e58cf02aadc5e493e0f6bbb4b3d3a4f4e48c94664612a42fed64aa3" Oct 30 17:16:01.463035 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:01.463018 5052 scope.go:122] "RemoveContainer" containerID="5e1bb34dcfa3906b934a7cf76809c8d2ad4cf7085c32d857441401563a4ec18a" Oct 30 17:16:01.463332 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:16:01.463313 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e1bb34dcfa3906b934a7cf76809c8d2ad4cf7085c32d857441401563a4ec18a\": container with ID starting with 5e1bb34dcfa3906b934a7cf76809c8d2ad4cf7085c32d857441401563a4ec18a not found: ID does not exist" containerID="5e1bb34dcfa3906b934a7cf76809c8d2ad4cf7085c32d857441401563a4ec18a" Oct 30 17:16:01.463435 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:01.463338 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e1bb34dcfa3906b934a7cf76809c8d2ad4cf7085c32d857441401563a4ec18a"} err="failed to get container status \"5e1bb34dcfa3906b934a7cf76809c8d2ad4cf7085c32d857441401563a4ec18a\": rpc error: code = NotFound desc = could not find container \"5e1bb34dcfa3906b934a7cf76809c8d2ad4cf7085c32d857441401563a4ec18a\": container with ID starting with 5e1bb34dcfa3906b934a7cf76809c8d2ad4cf7085c32d857441401563a4ec18a not found: ID does not exist" Oct 30 17:16:01.463435 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:01.463353 5052 scope.go:122] "RemoveContainer" containerID="3e8dd9f08e58cf02aadc5e493e0f6bbb4b3d3a4f4e48c94664612a42fed64aa3" Oct 30 17:16:01.463740 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:16:01.463716 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e8dd9f08e58cf02aadc5e493e0f6bbb4b3d3a4f4e48c94664612a42fed64aa3\": container with ID starting with 3e8dd9f08e58cf02aadc5e493e0f6bbb4b3d3a4f4e48c94664612a42fed64aa3 not found: ID does not exist" containerID="3e8dd9f08e58cf02aadc5e493e0f6bbb4b3d3a4f4e48c94664612a42fed64aa3" Oct 30 17:16:01.463825 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:01.463745 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e8dd9f08e58cf02aadc5e493e0f6bbb4b3d3a4f4e48c94664612a42fed64aa3"} err="failed to get container status \"3e8dd9f08e58cf02aadc5e493e0f6bbb4b3d3a4f4e48c94664612a42fed64aa3\": rpc error: code = NotFound desc = could not find container \"3e8dd9f08e58cf02aadc5e493e0f6bbb4b3d3a4f4e48c94664612a42fed64aa3\": container with ID starting with 3e8dd9f08e58cf02aadc5e493e0f6bbb4b3d3a4f4e48c94664612a42fed64aa3 not found: ID does not exist" Oct 30 17:16:01.514124 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:01.514088 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-0ae3fe-openshift-default-69fdb67df-sbgql"] Oct 30 17:16:01.527335 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:01.527301 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-0ae3fe-openshift-default-69fdb67df-sbgql"] Oct 30 17:16:01.837068 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:01.836962 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-97a47d-openshift-default-5dcd657cdf-8l2c4"] Oct 30 17:16:01.837569 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:01.837182 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-97a47d-openshift-default-5dcd657cdf-8l2c4" podUID="837038f3-5741-4b1f-8bb8-f8c8e234128f" containerName="istio-proxy" containerID="cri-o://232ede6426254611cacbac9aabff3fbe6a196749b4b91b7efacd57f75a59dc04" gracePeriod=30 Oct 30 17:16:02.598933 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:02.598897 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="666e4b84-2d87-4557-a8b5-10f890a1740b" path="/var/lib/kubelet/pods/666e4b84-2d87-4557-a8b5-10f890a1740b/volumes" Oct 30 17:16:07.184420 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:07.184395 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-97a47d-openshift-default-5dcd657cdf-8l2c4" Oct 30 17:16:07.297202 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:07.297165 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/837038f3-5741-4b1f-8bb8-f8c8e234128f-istio-token\" (UniqueName: \"kubernetes.io/projected/837038f3-5741-4b1f-8bb8-f8c8e234128f-istio-token\") pod \"837038f3-5741-4b1f-8bb8-f8c8e234128f\" (UID: \"837038f3-5741-4b1f-8bb8-f8c8e234128f\") " Oct 30 17:16:07.297379 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:07.297274 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/837038f3-5741-4b1f-8bb8-f8c8e234128f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/837038f3-5741-4b1f-8bb8-f8c8e234128f-istio-podinfo\") pod \"837038f3-5741-4b1f-8bb8-f8c8e234128f\" (UID: \"837038f3-5741-4b1f-8bb8-f8c8e234128f\") " Oct 30 17:16:07.297379 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:07.297313 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/837038f3-5741-4b1f-8bb8-f8c8e234128f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/837038f3-5741-4b1f-8bb8-f8c8e234128f-istio-data\") pod \"837038f3-5741-4b1f-8bb8-f8c8e234128f\" (UID: \"837038f3-5741-4b1f-8bb8-f8c8e234128f\") " Oct 30 17:16:07.297379 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:07.297335 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/837038f3-5741-4b1f-8bb8-f8c8e234128f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/837038f3-5741-4b1f-8bb8-f8c8e234128f-workload-socket\") pod \"837038f3-5741-4b1f-8bb8-f8c8e234128f\" (UID: \"837038f3-5741-4b1f-8bb8-f8c8e234128f\") " Oct 30 17:16:07.297379 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:07.297379 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/837038f3-5741-4b1f-8bb8-f8c8e234128f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/837038f3-5741-4b1f-8bb8-f8c8e234128f-credential-socket\") pod \"837038f3-5741-4b1f-8bb8-f8c8e234128f\" (UID: \"837038f3-5741-4b1f-8bb8-f8c8e234128f\") " Oct 30 17:16:07.297635 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:07.297416 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/837038f3-5741-4b1f-8bb8-f8c8e234128f-kube-api-access-k9dwr\" (UniqueName: \"kubernetes.io/projected/837038f3-5741-4b1f-8bb8-f8c8e234128f-kube-api-access-k9dwr\") pod \"837038f3-5741-4b1f-8bb8-f8c8e234128f\" (UID: \"837038f3-5741-4b1f-8bb8-f8c8e234128f\") " Oct 30 17:16:07.297635 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:07.297460 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/837038f3-5741-4b1f-8bb8-f8c8e234128f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/837038f3-5741-4b1f-8bb8-f8c8e234128f-istiod-ca-cert\") pod \"837038f3-5741-4b1f-8bb8-f8c8e234128f\" (UID: \"837038f3-5741-4b1f-8bb8-f8c8e234128f\") " Oct 30 17:16:07.297635 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:07.297483 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/837038f3-5741-4b1f-8bb8-f8c8e234128f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/837038f3-5741-4b1f-8bb8-f8c8e234128f-istio-envoy\") pod \"837038f3-5741-4b1f-8bb8-f8c8e234128f\" (UID: \"837038f3-5741-4b1f-8bb8-f8c8e234128f\") " Oct 30 17:16:07.297804 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:07.297746 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/837038f3-5741-4b1f-8bb8-f8c8e234128f-istio-data" pod "837038f3-5741-4b1f-8bb8-f8c8e234128f" (UID: "837038f3-5741-4b1f-8bb8-f8c8e234128f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:16:07.297869 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:07.297841 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/837038f3-5741-4b1f-8bb8-f8c8e234128f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/837038f3-5741-4b1f-8bb8-f8c8e234128f-workload-certs\") pod \"837038f3-5741-4b1f-8bb8-f8c8e234128f\" (UID: \"837038f3-5741-4b1f-8bb8-f8c8e234128f\") " Oct 30 17:16:07.297930 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:07.297880 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/837038f3-5741-4b1f-8bb8-f8c8e234128f-credential-socket" pod "837038f3-5741-4b1f-8bb8-f8c8e234128f" (UID: "837038f3-5741-4b1f-8bb8-f8c8e234128f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:16:07.297996 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:07.297965 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/837038f3-5741-4b1f-8bb8-f8c8e234128f-istiod-ca-cert" pod "837038f3-5741-4b1f-8bb8-f8c8e234128f" (UID: "837038f3-5741-4b1f-8bb8-f8c8e234128f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:16:07.298173 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:07.298150 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/837038f3-5741-4b1f-8bb8-f8c8e234128f-workload-socket" pod "837038f3-5741-4b1f-8bb8-f8c8e234128f" (UID: "837038f3-5741-4b1f-8bb8-f8c8e234128f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:16:07.298173 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:07.298164 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/837038f3-5741-4b1f-8bb8-f8c8e234128f-workload-certs" pod "837038f3-5741-4b1f-8bb8-f8c8e234128f" (UID: "837038f3-5741-4b1f-8bb8-f8c8e234128f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:16:07.298536 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:07.298443 5052 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/837038f3-5741-4b1f-8bb8-f8c8e234128f-istiod-ca-cert\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:16:07.298536 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:07.298471 5052 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/837038f3-5741-4b1f-8bb8-f8c8e234128f-workload-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:16:07.298536 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:07.298485 5052 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/837038f3-5741-4b1f-8bb8-f8c8e234128f-istio-data\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:16:07.298536 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:07.298499 5052 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/837038f3-5741-4b1f-8bb8-f8c8e234128f-workload-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:16:07.298536 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:07.298512 5052 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/837038f3-5741-4b1f-8bb8-f8c8e234128f-credential-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:16:07.299717 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:07.299676 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/837038f3-5741-4b1f-8bb8-f8c8e234128f-istio-podinfo" pod "837038f3-5741-4b1f-8bb8-f8c8e234128f" (UID: "837038f3-5741-4b1f-8bb8-f8c8e234128f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:16:07.299956 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:07.299930 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/837038f3-5741-4b1f-8bb8-f8c8e234128f-istio-envoy" pod "837038f3-5741-4b1f-8bb8-f8c8e234128f" (UID: "837038f3-5741-4b1f-8bb8-f8c8e234128f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:16:07.300155 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:07.300138 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/837038f3-5741-4b1f-8bb8-f8c8e234128f-istio-token" pod "837038f3-5741-4b1f-8bb8-f8c8e234128f" (UID: "837038f3-5741-4b1f-8bb8-f8c8e234128f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:16:07.300411 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:07.300381 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/837038f3-5741-4b1f-8bb8-f8c8e234128f-kube-api-access-k9dwr" pod "837038f3-5741-4b1f-8bb8-f8c8e234128f" (UID: "837038f3-5741-4b1f-8bb8-f8c8e234128f"). InnerVolumeSpecName "kube-api-access-k9dwr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:16:07.400199 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:07.400168 5052 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/837038f3-5741-4b1f-8bb8-f8c8e234128f-istio-podinfo\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:16:07.400199 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:07.400194 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9dwr\" (UniqueName: \"kubernetes.io/projected/837038f3-5741-4b1f-8bb8-f8c8e234128f-kube-api-access-k9dwr\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:16:07.400199 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:07.400205 5052 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/837038f3-5741-4b1f-8bb8-f8c8e234128f-istio-envoy\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:16:07.400422 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:07.400214 5052 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/837038f3-5741-4b1f-8bb8-f8c8e234128f-istio-token\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:16:07.485747 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:07.485660 5052 generic.go:356] "Generic (PLEG): container finished" podID="837038f3-5741-4b1f-8bb8-f8c8e234128f" containerID="232ede6426254611cacbac9aabff3fbe6a196749b4b91b7efacd57f75a59dc04" exitCode=0 Oct 30 17:16:07.485747 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:07.485706 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-97a47d-openshift-default-5dcd657cdf-8l2c4" Oct 30 17:16:07.485959 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:07.485744 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-97a47d-openshift-default-5dcd657cdf-8l2c4" event={"ID":"837038f3-5741-4b1f-8bb8-f8c8e234128f","Type":"ContainerDied","Data":"232ede6426254611cacbac9aabff3fbe6a196749b4b91b7efacd57f75a59dc04"} Oct 30 17:16:07.485959 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:07.485791 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-97a47d-openshift-default-5dcd657cdf-8l2c4" event={"ID":"837038f3-5741-4b1f-8bb8-f8c8e234128f","Type":"ContainerDied","Data":"00350f34d5be6acf192c00300818919279fa96c41ed425a8f91de8c4c45a4328"} Oct 30 17:16:07.485959 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:07.485814 5052 scope.go:122] "RemoveContainer" containerID="232ede6426254611cacbac9aabff3fbe6a196749b4b91b7efacd57f75a59dc04" Oct 30 17:16:07.496732 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:07.496710 5052 scope.go:122] "RemoveContainer" containerID="8899d0b5a7283c14ed4e600ce5a5a154d462e7f984936dbc51debcd7da8241c2" Oct 30 17:16:07.505738 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:07.505719 5052 scope.go:122] "RemoveContainer" containerID="232ede6426254611cacbac9aabff3fbe6a196749b4b91b7efacd57f75a59dc04" Oct 30 17:16:07.506010 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:16:07.505991 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"232ede6426254611cacbac9aabff3fbe6a196749b4b91b7efacd57f75a59dc04\": container with ID starting with 232ede6426254611cacbac9aabff3fbe6a196749b4b91b7efacd57f75a59dc04 not found: ID does not exist" containerID="232ede6426254611cacbac9aabff3fbe6a196749b4b91b7efacd57f75a59dc04" Oct 30 17:16:07.506067 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:07.506019 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"232ede6426254611cacbac9aabff3fbe6a196749b4b91b7efacd57f75a59dc04"} err="failed to get container status \"232ede6426254611cacbac9aabff3fbe6a196749b4b91b7efacd57f75a59dc04\": rpc error: code = NotFound desc = could not find container \"232ede6426254611cacbac9aabff3fbe6a196749b4b91b7efacd57f75a59dc04\": container with ID starting with 232ede6426254611cacbac9aabff3fbe6a196749b4b91b7efacd57f75a59dc04 not found: ID does not exist" Oct 30 17:16:07.506067 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:07.506045 5052 scope.go:122] "RemoveContainer" containerID="8899d0b5a7283c14ed4e600ce5a5a154d462e7f984936dbc51debcd7da8241c2" Oct 30 17:16:07.506288 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:16:07.506271 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8899d0b5a7283c14ed4e600ce5a5a154d462e7f984936dbc51debcd7da8241c2\": container with ID starting with 8899d0b5a7283c14ed4e600ce5a5a154d462e7f984936dbc51debcd7da8241c2 not found: ID does not exist" containerID="8899d0b5a7283c14ed4e600ce5a5a154d462e7f984936dbc51debcd7da8241c2" Oct 30 17:16:07.506330 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:07.506296 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8899d0b5a7283c14ed4e600ce5a5a154d462e7f984936dbc51debcd7da8241c2"} err="failed to get container status \"8899d0b5a7283c14ed4e600ce5a5a154d462e7f984936dbc51debcd7da8241c2\": rpc error: code = NotFound desc = could not find container \"8899d0b5a7283c14ed4e600ce5a5a154d462e7f984936dbc51debcd7da8241c2\": container with ID starting with 8899d0b5a7283c14ed4e600ce5a5a154d462e7f984936dbc51debcd7da8241c2 not found: ID does not exist" Oct 30 17:16:07.543099 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:07.543075 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-97a47d-openshift-default-5dcd657cdf-8l2c4"] Oct 30 17:16:07.556666 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:07.556636 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-97a47d-openshift-default-5dcd657cdf-8l2c4"] Oct 30 17:16:08.598390 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:08.598352 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="837038f3-5741-4b1f-8bb8-f8c8e234128f" path="/var/lib/kubelet/pods/837038f3-5741-4b1f-8bb8-f8c8e234128f/volumes" Oct 30 17:16:13.002410 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:13.002370 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-05810b-67f9c6857b-7jz5n"] Oct 30 17:16:14.795520 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:14.795488 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-de18d5-openshift-default-748c669-n82k6"] Oct 30 17:16:14.795969 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:14.795748 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-de18d5-openshift-default-748c669-n82k6" podUID="505e6f2b-634b-40ae-b5db-6ced8e1bf322" containerName="istio-proxy" containerID="cri-o://f95be2dbdb5436562e544029b65fb86d2d4b9d9ed635360bd2caa4e9b379fdb7" gracePeriod=30 Oct 30 17:16:18.003594 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:18.003530 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-05810b-67f9c6857b-7jz5n" podUID="6136d90c-0253-4761-843b-f8a3922c93cf" containerName="payload-processing" containerID="cri-o://cb0f06ad0af10ba3297ecb873778a0595554e2ae8d98f8c2e7f76ec0b9039d98" gracePeriod=25 Oct 30 17:16:18.546724 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:18.546687 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-05810b-dc6664f5c-dtrsg"] Oct 30 17:16:18.546948 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:18.546922 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-05810b-dc6664f5c-dtrsg" podUID="e9258d14-705c-40d3-8898-48d08dcf7846" containerName="maas-api" containerID="cri-o://f0ae2c0f6d1986a6a9cd3e99fe10268b2a79f28548d710ae98ab06f9fa5d90ae" gracePeriod=30 Oct 30 17:16:18.858619 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:18.858575 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-05810b-dc6664f5c-dtrsg" Oct 30 17:16:18.869919 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:18.869889 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9258d14-705c-40d3-8898-48d08dcf7846-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9258d14-705c-40d3-8898-48d08dcf7846-maas-api-tls\") pod \"e9258d14-705c-40d3-8898-48d08dcf7846\" (UID: \"e9258d14-705c-40d3-8898-48d08dcf7846\") " Oct 30 17:16:18.870065 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:18.869948 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9258d14-705c-40d3-8898-48d08dcf7846-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9258d14-705c-40d3-8898-48d08dcf7846-openshift-service-ca\") pod \"e9258d14-705c-40d3-8898-48d08dcf7846\" (UID: \"e9258d14-705c-40d3-8898-48d08dcf7846\") " Oct 30 17:16:18.870065 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:18.869969 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9258d14-705c-40d3-8898-48d08dcf7846-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9258d14-705c-40d3-8898-48d08dcf7846-metrics-certs\") pod \"e9258d14-705c-40d3-8898-48d08dcf7846\" (UID: \"e9258d14-705c-40d3-8898-48d08dcf7846\") " Oct 30 17:16:18.870065 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:18.870005 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9258d14-705c-40d3-8898-48d08dcf7846-kube-api-access-p7z9z\" (UniqueName: \"kubernetes.io/projected/e9258d14-705c-40d3-8898-48d08dcf7846-kube-api-access-p7z9z\") pod \"e9258d14-705c-40d3-8898-48d08dcf7846\" (UID: \"e9258d14-705c-40d3-8898-48d08dcf7846\") " Oct 30 17:16:18.870816 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:16:18.870775 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 17:16:18.871095 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:18.871057 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9258d14-705c-40d3-8898-48d08dcf7846-openshift-service-ca" pod "e9258d14-705c-40d3-8898-48d08dcf7846" (UID: "e9258d14-705c-40d3-8898-48d08dcf7846"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:16:18.876067 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:18.872874 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9258d14-705c-40d3-8898-48d08dcf7846-maas-api-tls" pod "e9258d14-705c-40d3-8898-48d08dcf7846" (UID: "e9258d14-705c-40d3-8898-48d08dcf7846"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:16:18.876067 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:18.873033 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9258d14-705c-40d3-8898-48d08dcf7846-kube-api-access-p7z9z" pod "e9258d14-705c-40d3-8898-48d08dcf7846" (UID: "e9258d14-705c-40d3-8898-48d08dcf7846"). InnerVolumeSpecName "kube-api-access-p7z9z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:16:18.878715 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:18.878689 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9258d14-705c-40d3-8898-48d08dcf7846-metrics-certs" pod "e9258d14-705c-40d3-8898-48d08dcf7846" (UID: "e9258d14-705c-40d3-8898-48d08dcf7846"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:16:18.958194 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:18.958154 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-05810b-67f9c6857b-7jz5n" podUID="6136d90c-0253-4761-843b-f8a3922c93cf" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:16:18.972138 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:18.972104 5052 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9258d14-705c-40d3-8898-48d08dcf7846-maas-api-tls\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:16:18.972138 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:18.972132 5052 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9258d14-705c-40d3-8898-48d08dcf7846-openshift-service-ca\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:16:18.972287 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:18.972145 5052 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9258d14-705c-40d3-8898-48d08dcf7846-metrics-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:16:18.972287 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:18.972159 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7z9z\" (UniqueName: \"kubernetes.io/projected/e9258d14-705c-40d3-8898-48d08dcf7846-kube-api-access-p7z9z\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:16:19.568711 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:19.568673 5052 generic.go:356] "Generic (PLEG): container finished" podID="e9258d14-705c-40d3-8898-48d08dcf7846" containerID="f0ae2c0f6d1986a6a9cd3e99fe10268b2a79f28548d710ae98ab06f9fa5d90ae" exitCode=0 Oct 30 17:16:19.569213 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:19.568716 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-05810b-dc6664f5c-dtrsg" Oct 30 17:16:19.569213 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:19.568762 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-05810b-dc6664f5c-dtrsg" event={"ID":"e9258d14-705c-40d3-8898-48d08dcf7846","Type":"ContainerDied","Data":"f0ae2c0f6d1986a6a9cd3e99fe10268b2a79f28548d710ae98ab06f9fa5d90ae"} Oct 30 17:16:19.569213 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:19.568821 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-05810b-dc6664f5c-dtrsg" event={"ID":"e9258d14-705c-40d3-8898-48d08dcf7846","Type":"ContainerDied","Data":"21b5240c7bf721922b4a7bc75f925334d30cc125b592ecfb258e351cb68fa111"} Oct 30 17:16:19.569213 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:19.568845 5052 scope.go:122] "RemoveContainer" containerID="f0ae2c0f6d1986a6a9cd3e99fe10268b2a79f28548d710ae98ab06f9fa5d90ae" Oct 30 17:16:19.580296 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:19.580281 5052 scope.go:122] "RemoveContainer" containerID="f0ae2c0f6d1986a6a9cd3e99fe10268b2a79f28548d710ae98ab06f9fa5d90ae" Oct 30 17:16:19.580594 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:16:19.580563 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0ae2c0f6d1986a6a9cd3e99fe10268b2a79f28548d710ae98ab06f9fa5d90ae\": container with ID starting with f0ae2c0f6d1986a6a9cd3e99fe10268b2a79f28548d710ae98ab06f9fa5d90ae not found: ID does not exist" containerID="f0ae2c0f6d1986a6a9cd3e99fe10268b2a79f28548d710ae98ab06f9fa5d90ae" Oct 30 17:16:19.580642 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:19.580605 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0ae2c0f6d1986a6a9cd3e99fe10268b2a79f28548d710ae98ab06f9fa5d90ae"} err="failed to get container status \"f0ae2c0f6d1986a6a9cd3e99fe10268b2a79f28548d710ae98ab06f9fa5d90ae\": rpc error: code = NotFound desc = could not find container \"f0ae2c0f6d1986a6a9cd3e99fe10268b2a79f28548d710ae98ab06f9fa5d90ae\": container with ID starting with f0ae2c0f6d1986a6a9cd3e99fe10268b2a79f28548d710ae98ab06f9fa5d90ae not found: ID does not exist" Oct 30 17:16:19.625086 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:19.625061 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-05810b-dc6664f5c-dtrsg"] Oct 30 17:16:19.638491 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:19.638468 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-05810b-dc6664f5c-dtrsg"] Oct 30 17:16:20.079806 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:20.079738 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-de18d5-openshift-default-748c669-n82k6" Oct 30 17:16:20.084838 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:20.084814 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/505e6f2b-634b-40ae-b5db-6ced8e1bf322-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/505e6f2b-634b-40ae-b5db-6ced8e1bf322-istio-podinfo\") pod \"505e6f2b-634b-40ae-b5db-6ced8e1bf322\" (UID: \"505e6f2b-634b-40ae-b5db-6ced8e1bf322\") " Oct 30 17:16:20.084967 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:20.084874 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/505e6f2b-634b-40ae-b5db-6ced8e1bf322-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/505e6f2b-634b-40ae-b5db-6ced8e1bf322-credential-socket\") pod \"505e6f2b-634b-40ae-b5db-6ced8e1bf322\" (UID: \"505e6f2b-634b-40ae-b5db-6ced8e1bf322\") " Oct 30 17:16:20.084967 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:20.084917 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/505e6f2b-634b-40ae-b5db-6ced8e1bf322-kube-api-access-4nvgd\" (UniqueName: \"kubernetes.io/projected/505e6f2b-634b-40ae-b5db-6ced8e1bf322-kube-api-access-4nvgd\") pod \"505e6f2b-634b-40ae-b5db-6ced8e1bf322\" (UID: \"505e6f2b-634b-40ae-b5db-6ced8e1bf322\") " Oct 30 17:16:20.085088 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:20.084988 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/505e6f2b-634b-40ae-b5db-6ced8e1bf322-istio-token\" (UniqueName: \"kubernetes.io/projected/505e6f2b-634b-40ae-b5db-6ced8e1bf322-istio-token\") pod \"505e6f2b-634b-40ae-b5db-6ced8e1bf322\" (UID: \"505e6f2b-634b-40ae-b5db-6ced8e1bf322\") " Oct 30 17:16:20.085157 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:20.085092 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/505e6f2b-634b-40ae-b5db-6ced8e1bf322-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/505e6f2b-634b-40ae-b5db-6ced8e1bf322-workload-certs\") pod \"505e6f2b-634b-40ae-b5db-6ced8e1bf322\" (UID: \"505e6f2b-634b-40ae-b5db-6ced8e1bf322\") " Oct 30 17:16:20.085157 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:20.085147 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/505e6f2b-634b-40ae-b5db-6ced8e1bf322-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/505e6f2b-634b-40ae-b5db-6ced8e1bf322-istiod-ca-cert\") pod \"505e6f2b-634b-40ae-b5db-6ced8e1bf322\" (UID: \"505e6f2b-634b-40ae-b5db-6ced8e1bf322\") " Oct 30 17:16:20.085268 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:20.085160 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/505e6f2b-634b-40ae-b5db-6ced8e1bf322-credential-socket" pod "505e6f2b-634b-40ae-b5db-6ced8e1bf322" (UID: "505e6f2b-634b-40ae-b5db-6ced8e1bf322"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:16:20.085268 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:20.085202 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/505e6f2b-634b-40ae-b5db-6ced8e1bf322-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/505e6f2b-634b-40ae-b5db-6ced8e1bf322-istio-data\") pod \"505e6f2b-634b-40ae-b5db-6ced8e1bf322\" (UID: \"505e6f2b-634b-40ae-b5db-6ced8e1bf322\") " Oct 30 17:16:20.085378 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:20.085269 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/505e6f2b-634b-40ae-b5db-6ced8e1bf322-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/505e6f2b-634b-40ae-b5db-6ced8e1bf322-workload-socket\") pod \"505e6f2b-634b-40ae-b5db-6ced8e1bf322\" (UID: \"505e6f2b-634b-40ae-b5db-6ced8e1bf322\") " Oct 30 17:16:20.085378 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:20.085305 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/505e6f2b-634b-40ae-b5db-6ced8e1bf322-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/505e6f2b-634b-40ae-b5db-6ced8e1bf322-istio-envoy\") pod \"505e6f2b-634b-40ae-b5db-6ced8e1bf322\" (UID: \"505e6f2b-634b-40ae-b5db-6ced8e1bf322\") " Oct 30 17:16:20.085483 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:20.085367 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/505e6f2b-634b-40ae-b5db-6ced8e1bf322-workload-certs" pod "505e6f2b-634b-40ae-b5db-6ced8e1bf322" (UID: "505e6f2b-634b-40ae-b5db-6ced8e1bf322"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:16:20.085553 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:20.085530 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/505e6f2b-634b-40ae-b5db-6ced8e1bf322-istio-data" pod "505e6f2b-634b-40ae-b5db-6ced8e1bf322" (UID: "505e6f2b-634b-40ae-b5db-6ced8e1bf322"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:16:20.085642 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:20.085600 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/505e6f2b-634b-40ae-b5db-6ced8e1bf322-workload-socket" pod "505e6f2b-634b-40ae-b5db-6ced8e1bf322" (UID: "505e6f2b-634b-40ae-b5db-6ced8e1bf322"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:16:20.085696 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:20.085673 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/505e6f2b-634b-40ae-b5db-6ced8e1bf322-istiod-ca-cert" pod "505e6f2b-634b-40ae-b5db-6ced8e1bf322" (UID: "505e6f2b-634b-40ae-b5db-6ced8e1bf322"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:16:20.085868 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:20.085847 5052 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/505e6f2b-634b-40ae-b5db-6ced8e1bf322-istiod-ca-cert\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:16:20.085924 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:20.085876 5052 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/505e6f2b-634b-40ae-b5db-6ced8e1bf322-istio-data\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:16:20.085924 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:20.085891 5052 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/505e6f2b-634b-40ae-b5db-6ced8e1bf322-workload-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:16:20.085924 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:20.085906 5052 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/505e6f2b-634b-40ae-b5db-6ced8e1bf322-credential-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:16:20.086029 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:20.085922 5052 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/505e6f2b-634b-40ae-b5db-6ced8e1bf322-workload-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:16:20.087132 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:20.087112 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/505e6f2b-634b-40ae-b5db-6ced8e1bf322-istio-podinfo" pod "505e6f2b-634b-40ae-b5db-6ced8e1bf322" (UID: "505e6f2b-634b-40ae-b5db-6ced8e1bf322"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:16:20.087299 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:20.087281 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/505e6f2b-634b-40ae-b5db-6ced8e1bf322-istio-envoy" pod "505e6f2b-634b-40ae-b5db-6ced8e1bf322" (UID: "505e6f2b-634b-40ae-b5db-6ced8e1bf322"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:16:20.087595 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:20.087560 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/505e6f2b-634b-40ae-b5db-6ced8e1bf322-istio-token" pod "505e6f2b-634b-40ae-b5db-6ced8e1bf322" (UID: "505e6f2b-634b-40ae-b5db-6ced8e1bf322"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:16:20.087689 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:20.087640 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/505e6f2b-634b-40ae-b5db-6ced8e1bf322-kube-api-access-4nvgd" pod "505e6f2b-634b-40ae-b5db-6ced8e1bf322" (UID: "505e6f2b-634b-40ae-b5db-6ced8e1bf322"). InnerVolumeSpecName "kube-api-access-4nvgd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:16:20.187351 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:20.187322 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nvgd\" (UniqueName: \"kubernetes.io/projected/505e6f2b-634b-40ae-b5db-6ced8e1bf322-kube-api-access-4nvgd\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:16:20.187351 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:20.187347 5052 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/505e6f2b-634b-40ae-b5db-6ced8e1bf322-istio-token\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:16:20.187351 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:20.187356 5052 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/505e6f2b-634b-40ae-b5db-6ced8e1bf322-istio-envoy\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:16:20.187618 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:20.187365 5052 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/505e6f2b-634b-40ae-b5db-6ced8e1bf322-istio-podinfo\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:16:20.578937 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:20.578902 5052 generic.go:356] "Generic (PLEG): container finished" podID="505e6f2b-634b-40ae-b5db-6ced8e1bf322" containerID="f95be2dbdb5436562e544029b65fb86d2d4b9d9ed635360bd2caa4e9b379fdb7" exitCode=0 Oct 30 17:16:20.579462 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:20.578955 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-de18d5-openshift-default-748c669-n82k6" Oct 30 17:16:20.579462 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:20.578980 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-de18d5-openshift-default-748c669-n82k6" event={"ID":"505e6f2b-634b-40ae-b5db-6ced8e1bf322","Type":"ContainerDied","Data":"f95be2dbdb5436562e544029b65fb86d2d4b9d9ed635360bd2caa4e9b379fdb7"} Oct 30 17:16:20.579462 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:20.579042 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-de18d5-openshift-default-748c669-n82k6" event={"ID":"505e6f2b-634b-40ae-b5db-6ced8e1bf322","Type":"ContainerDied","Data":"b47d31d9166a135eb965f2a1ad98cac4101fac5729f9d138b3c6ed27079f5d57"} Oct 30 17:16:20.579462 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:20.579073 5052 scope.go:122] "RemoveContainer" containerID="f95be2dbdb5436562e544029b65fb86d2d4b9d9ed635360bd2caa4e9b379fdb7" Oct 30 17:16:20.590798 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:20.590772 5052 scope.go:122] "RemoveContainer" containerID="0e8e12f2d38c3d2d9f6d112c53f873cd3f077945188e70180365ce297c14af94" Oct 30 17:16:20.598390 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:20.598368 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9258d14-705c-40d3-8898-48d08dcf7846" path="/var/lib/kubelet/pods/e9258d14-705c-40d3-8898-48d08dcf7846/volumes" Oct 30 17:16:20.602639 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:20.602569 5052 scope.go:122] "RemoveContainer" containerID="f95be2dbdb5436562e544029b65fb86d2d4b9d9ed635360bd2caa4e9b379fdb7" Oct 30 17:16:20.602904 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:16:20.602883 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f95be2dbdb5436562e544029b65fb86d2d4b9d9ed635360bd2caa4e9b379fdb7\": container with ID starting with f95be2dbdb5436562e544029b65fb86d2d4b9d9ed635360bd2caa4e9b379fdb7 not found: ID does not exist" containerID="f95be2dbdb5436562e544029b65fb86d2d4b9d9ed635360bd2caa4e9b379fdb7" Oct 30 17:16:20.602983 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:20.602918 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f95be2dbdb5436562e544029b65fb86d2d4b9d9ed635360bd2caa4e9b379fdb7"} err="failed to get container status \"f95be2dbdb5436562e544029b65fb86d2d4b9d9ed635360bd2caa4e9b379fdb7\": rpc error: code = NotFound desc = could not find container \"f95be2dbdb5436562e544029b65fb86d2d4b9d9ed635360bd2caa4e9b379fdb7\": container with ID starting with f95be2dbdb5436562e544029b65fb86d2d4b9d9ed635360bd2caa4e9b379fdb7 not found: ID does not exist" Oct 30 17:16:20.602983 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:20.602943 5052 scope.go:122] "RemoveContainer" containerID="0e8e12f2d38c3d2d9f6d112c53f873cd3f077945188e70180365ce297c14af94" Oct 30 17:16:20.603236 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:16:20.603215 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e8e12f2d38c3d2d9f6d112c53f873cd3f077945188e70180365ce297c14af94\": container with ID starting with 0e8e12f2d38c3d2d9f6d112c53f873cd3f077945188e70180365ce297c14af94 not found: ID does not exist" containerID="0e8e12f2d38c3d2d9f6d112c53f873cd3f077945188e70180365ce297c14af94" Oct 30 17:16:20.603277 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:20.603244 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e8e12f2d38c3d2d9f6d112c53f873cd3f077945188e70180365ce297c14af94"} err="failed to get container status \"0e8e12f2d38c3d2d9f6d112c53f873cd3f077945188e70180365ce297c14af94\": rpc error: code = NotFound desc = could not find container \"0e8e12f2d38c3d2d9f6d112c53f873cd3f077945188e70180365ce297c14af94\": container with ID starting with 0e8e12f2d38c3d2d9f6d112c53f873cd3f077945188e70180365ce297c14af94 not found: ID does not exist" Oct 30 17:16:20.656019 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:20.655983 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-de18d5-openshift-default-748c669-n82k6"] Oct 30 17:16:20.668687 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:20.668645 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-de18d5-openshift-default-748c669-n82k6"] Oct 30 17:16:20.913681 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:20.913641 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-fxsm2"] Oct 30 17:16:20.913949 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:20.913902 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-fxsm2" podUID="f0607bb9-cd62-4149-a429-4051390ad2a1" containerName="istio-proxy" containerID="cri-o://fefb376a529947de9ff314f3583738dc55d221e956a1f3663fbce2a5ccc42760" gracePeriod=30 Oct 30 17:16:22.412032 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:22.411989 5052 patch_prober.go:29] interesting pod/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-fxsm2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 17:16:22.412451 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:22.412062 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-fxsm2" podUID="f0607bb9-cd62-4149-a429-4051390ad2a1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 17:16:22.597000 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:22.596966 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="505e6f2b-634b-40ae-b5db-6ced8e1bf322" path="/var/lib/kubelet/pods/505e6f2b-634b-40ae-b5db-6ced8e1bf322/volumes" Oct 30 17:16:24.679894 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:24.679851 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b4214a-6fc8bc5d7b-m6vlq"] Oct 30 17:16:24.680371 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:24.680094 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b4214a-6fc8bc5d7b-m6vlq" podUID="9ccbae02-fb91-4bd2-9e3b-3aa09cd5a415" containerName="maas-api" containerID="cri-o://c1ae373539f07c74c164c136de726c4c48ce9039bf4965f4ed55948d3bea6239" gracePeriod=30 Oct 30 17:16:24.949752 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:24.949689 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b4214a-6fc8bc5d7b-m6vlq" Oct 30 17:16:25.045426 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:25.045393 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ccbae02-fb91-4bd2-9e3b-3aa09cd5a415-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ccbae02-fb91-4bd2-9e3b-3aa09cd5a415-maas-api-tls\") pod \"9ccbae02-fb91-4bd2-9e3b-3aa09cd5a415\" (UID: \"9ccbae02-fb91-4bd2-9e3b-3aa09cd5a415\") " Oct 30 17:16:25.045648 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:25.045436 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ccbae02-fb91-4bd2-9e3b-3aa09cd5a415-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ccbae02-fb91-4bd2-9e3b-3aa09cd5a415-metrics-certs\") pod \"9ccbae02-fb91-4bd2-9e3b-3aa09cd5a415\" (UID: \"9ccbae02-fb91-4bd2-9e3b-3aa09cd5a415\") " Oct 30 17:16:25.045648 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:25.045615 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ccbae02-fb91-4bd2-9e3b-3aa09cd5a415-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ccbae02-fb91-4bd2-9e3b-3aa09cd5a415-openshift-service-ca\") pod \"9ccbae02-fb91-4bd2-9e3b-3aa09cd5a415\" (UID: \"9ccbae02-fb91-4bd2-9e3b-3aa09cd5a415\") " Oct 30 17:16:25.045648 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:25.045646 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ccbae02-fb91-4bd2-9e3b-3aa09cd5a415-kube-api-access-nhf2j\" (UniqueName: \"kubernetes.io/projected/9ccbae02-fb91-4bd2-9e3b-3aa09cd5a415-kube-api-access-nhf2j\") pod \"9ccbae02-fb91-4bd2-9e3b-3aa09cd5a415\" (UID: \"9ccbae02-fb91-4bd2-9e3b-3aa09cd5a415\") " Oct 30 17:16:25.046123 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:25.046095 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ccbae02-fb91-4bd2-9e3b-3aa09cd5a415-openshift-service-ca" pod "9ccbae02-fb91-4bd2-9e3b-3aa09cd5a415" (UID: "9ccbae02-fb91-4bd2-9e3b-3aa09cd5a415"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:16:25.047819 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:25.047778 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ccbae02-fb91-4bd2-9e3b-3aa09cd5a415-metrics-certs" pod "9ccbae02-fb91-4bd2-9e3b-3aa09cd5a415" (UID: "9ccbae02-fb91-4bd2-9e3b-3aa09cd5a415"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:16:25.049049 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:25.049025 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ccbae02-fb91-4bd2-9e3b-3aa09cd5a415-kube-api-access-nhf2j" pod "9ccbae02-fb91-4bd2-9e3b-3aa09cd5a415" (UID: "9ccbae02-fb91-4bd2-9e3b-3aa09cd5a415"). InnerVolumeSpecName "kube-api-access-nhf2j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:16:25.049173 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:25.049113 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ccbae02-fb91-4bd2-9e3b-3aa09cd5a415-maas-api-tls" pod "9ccbae02-fb91-4bd2-9e3b-3aa09cd5a415" (UID: "9ccbae02-fb91-4bd2-9e3b-3aa09cd5a415"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:16:25.152606 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:25.152504 5052 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ccbae02-fb91-4bd2-9e3b-3aa09cd5a415-maas-api-tls\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:16:25.152606 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:25.152551 5052 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ccbae02-fb91-4bd2-9e3b-3aa09cd5a415-metrics-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:16:25.152606 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:25.152567 5052 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ccbae02-fb91-4bd2-9e3b-3aa09cd5a415-openshift-service-ca\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:16:25.152606 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:25.152603 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhf2j\" (UniqueName: \"kubernetes.io/projected/9ccbae02-fb91-4bd2-9e3b-3aa09cd5a415-kube-api-access-nhf2j\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:16:25.611850 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:25.611814 5052 generic.go:356] "Generic (PLEG): container finished" podID="9ccbae02-fb91-4bd2-9e3b-3aa09cd5a415" containerID="c1ae373539f07c74c164c136de726c4c48ce9039bf4965f4ed55948d3bea6239" exitCode=0 Oct 30 17:16:25.612009 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:25.611856 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b4214a-6fc8bc5d7b-m6vlq" Oct 30 17:16:25.612009 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:25.611907 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b4214a-6fc8bc5d7b-m6vlq" event={"ID":"9ccbae02-fb91-4bd2-9e3b-3aa09cd5a415","Type":"ContainerDied","Data":"c1ae373539f07c74c164c136de726c4c48ce9039bf4965f4ed55948d3bea6239"} Oct 30 17:16:25.612009 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:25.611957 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b4214a-6fc8bc5d7b-m6vlq" event={"ID":"9ccbae02-fb91-4bd2-9e3b-3aa09cd5a415","Type":"ContainerDied","Data":"db49d8b10074f005ccd61f899dbd1066e9d8ee4a2178777b276efae3e32d6761"} Oct 30 17:16:25.612009 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:25.611978 5052 scope.go:122] "RemoveContainer" containerID="c1ae373539f07c74c164c136de726c4c48ce9039bf4965f4ed55948d3bea6239" Oct 30 17:16:25.622394 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:25.622375 5052 scope.go:122] "RemoveContainer" containerID="c1ae373539f07c74c164c136de726c4c48ce9039bf4965f4ed55948d3bea6239" Oct 30 17:16:25.622766 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:16:25.622750 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1ae373539f07c74c164c136de726c4c48ce9039bf4965f4ed55948d3bea6239\": container with ID starting with c1ae373539f07c74c164c136de726c4c48ce9039bf4965f4ed55948d3bea6239 not found: ID does not exist" containerID="c1ae373539f07c74c164c136de726c4c48ce9039bf4965f4ed55948d3bea6239" Oct 30 17:16:25.622817 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:25.622773 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1ae373539f07c74c164c136de726c4c48ce9039bf4965f4ed55948d3bea6239"} err="failed to get container status \"c1ae373539f07c74c164c136de726c4c48ce9039bf4965f4ed55948d3bea6239\": rpc error: code = NotFound desc = could not find container \"c1ae373539f07c74c164c136de726c4c48ce9039bf4965f4ed55948d3bea6239\": container with ID starting with c1ae373539f07c74c164c136de726c4c48ce9039bf4965f4ed55948d3bea6239 not found: ID does not exist" Oct 30 17:16:25.667680 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:25.667648 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b4214a-6fc8bc5d7b-m6vlq"] Oct 30 17:16:25.671381 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:25.671358 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b4214a-6fc8bc5d7b-m6vlq"] Oct 30 17:16:26.190564 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:26.190515 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-fxsm2" Oct 30 17:16:26.264625 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:26.264554 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0607bb9-cd62-4149-a429-4051390ad2a1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0607bb9-cd62-4149-a429-4051390ad2a1-istiod-ca-cert\") pod \"f0607bb9-cd62-4149-a429-4051390ad2a1\" (UID: \"f0607bb9-cd62-4149-a429-4051390ad2a1\") " Oct 30 17:16:26.264795 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:26.264681 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0607bb9-cd62-4149-a429-4051390ad2a1-istio-token\" (UniqueName: \"kubernetes.io/projected/f0607bb9-cd62-4149-a429-4051390ad2a1-istio-token\") pod \"f0607bb9-cd62-4149-a429-4051390ad2a1\" (UID: \"f0607bb9-cd62-4149-a429-4051390ad2a1\") " Oct 30 17:16:26.264795 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:26.264754 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0607bb9-cd62-4149-a429-4051390ad2a1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0607bb9-cd62-4149-a429-4051390ad2a1-istio-data\") pod \"f0607bb9-cd62-4149-a429-4051390ad2a1\" (UID: \"f0607bb9-cd62-4149-a429-4051390ad2a1\") " Oct 30 17:16:26.265038 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:26.264821 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f0607bb9-cd62-4149-a429-4051390ad2a1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0607bb9-cd62-4149-a429-4051390ad2a1-istio-podinfo\") pod \"f0607bb9-cd62-4149-a429-4051390ad2a1\" (UID: \"f0607bb9-cd62-4149-a429-4051390ad2a1\") " Oct 30 17:16:26.265038 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:26.264887 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0607bb9-cd62-4149-a429-4051390ad2a1-kube-api-access-tghhx\" (UniqueName: \"kubernetes.io/projected/f0607bb9-cd62-4149-a429-4051390ad2a1-kube-api-access-tghhx\") pod \"f0607bb9-cd62-4149-a429-4051390ad2a1\" (UID: \"f0607bb9-cd62-4149-a429-4051390ad2a1\") " Oct 30 17:16:26.265038 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:26.264906 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0607bb9-cd62-4149-a429-4051390ad2a1-istiod-ca-cert" pod "f0607bb9-cd62-4149-a429-4051390ad2a1" (UID: "f0607bb9-cd62-4149-a429-4051390ad2a1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:16:26.265038 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:26.264966 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0607bb9-cd62-4149-a429-4051390ad2a1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0607bb9-cd62-4149-a429-4051390ad2a1-istio-envoy\") pod \"f0607bb9-cd62-4149-a429-4051390ad2a1\" (UID: \"f0607bb9-cd62-4149-a429-4051390ad2a1\") " Oct 30 17:16:26.265038 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:26.265019 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0607bb9-cd62-4149-a429-4051390ad2a1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0607bb9-cd62-4149-a429-4051390ad2a1-workload-certs\") pod \"f0607bb9-cd62-4149-a429-4051390ad2a1\" (UID: \"f0607bb9-cd62-4149-a429-4051390ad2a1\") " Oct 30 17:16:26.265341 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:26.265091 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0607bb9-cd62-4149-a429-4051390ad2a1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0607bb9-cd62-4149-a429-4051390ad2a1-credential-socket\") pod \"f0607bb9-cd62-4149-a429-4051390ad2a1\" (UID: \"f0607bb9-cd62-4149-a429-4051390ad2a1\") " Oct 30 17:16:26.265341 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:26.265119 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0607bb9-cd62-4149-a429-4051390ad2a1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0607bb9-cd62-4149-a429-4051390ad2a1-workload-socket\") pod \"f0607bb9-cd62-4149-a429-4051390ad2a1\" (UID: \"f0607bb9-cd62-4149-a429-4051390ad2a1\") " Oct 30 17:16:26.265575 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:26.265552 5052 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0607bb9-cd62-4149-a429-4051390ad2a1-istiod-ca-cert\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:16:26.265751 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:26.265690 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0607bb9-cd62-4149-a429-4051390ad2a1-credential-socket" pod "f0607bb9-cd62-4149-a429-4051390ad2a1" (UID: "f0607bb9-cd62-4149-a429-4051390ad2a1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:16:26.265896 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:26.265828 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0607bb9-cd62-4149-a429-4051390ad2a1-workload-socket" pod "f0607bb9-cd62-4149-a429-4051390ad2a1" (UID: "f0607bb9-cd62-4149-a429-4051390ad2a1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:16:26.266055 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:26.266025 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0607bb9-cd62-4149-a429-4051390ad2a1-workload-certs" pod "f0607bb9-cd62-4149-a429-4051390ad2a1" (UID: "f0607bb9-cd62-4149-a429-4051390ad2a1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:16:26.266232 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:26.266095 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0607bb9-cd62-4149-a429-4051390ad2a1-istio-data" pod "f0607bb9-cd62-4149-a429-4051390ad2a1" (UID: "f0607bb9-cd62-4149-a429-4051390ad2a1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:16:26.267301 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:26.267265 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0607bb9-cd62-4149-a429-4051390ad2a1-istio-token" pod "f0607bb9-cd62-4149-a429-4051390ad2a1" (UID: "f0607bb9-cd62-4149-a429-4051390ad2a1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:16:26.267491 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:26.267473 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f0607bb9-cd62-4149-a429-4051390ad2a1-istio-podinfo" pod "f0607bb9-cd62-4149-a429-4051390ad2a1" (UID: "f0607bb9-cd62-4149-a429-4051390ad2a1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:16:26.267868 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:26.267835 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0607bb9-cd62-4149-a429-4051390ad2a1-kube-api-access-tghhx" pod "f0607bb9-cd62-4149-a429-4051390ad2a1" (UID: "f0607bb9-cd62-4149-a429-4051390ad2a1"). InnerVolumeSpecName "kube-api-access-tghhx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:16:26.267979 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:26.267859 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0607bb9-cd62-4149-a429-4051390ad2a1-istio-envoy" pod "f0607bb9-cd62-4149-a429-4051390ad2a1" (UID: "f0607bb9-cd62-4149-a429-4051390ad2a1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:16:26.366755 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:26.366726 5052 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0607bb9-cd62-4149-a429-4051390ad2a1-istio-envoy\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:16:26.366755 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:26.366751 5052 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0607bb9-cd62-4149-a429-4051390ad2a1-workload-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:16:26.366943 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:26.366761 5052 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0607bb9-cd62-4149-a429-4051390ad2a1-credential-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:16:26.366943 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:26.366771 5052 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0607bb9-cd62-4149-a429-4051390ad2a1-workload-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:16:26.366943 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:26.366780 5052 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0607bb9-cd62-4149-a429-4051390ad2a1-istio-token\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:16:26.366943 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:26.366788 5052 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0607bb9-cd62-4149-a429-4051390ad2a1-istio-data\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:16:26.366943 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:26.366795 5052 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0607bb9-cd62-4149-a429-4051390ad2a1-istio-podinfo\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:16:26.366943 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:26.366803 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tghhx\" (UniqueName: \"kubernetes.io/projected/f0607bb9-cd62-4149-a429-4051390ad2a1-kube-api-access-tghhx\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:16:26.388739 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:26.388710 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-263f00/distinct-2-w4-263f00-kserve-8464897cf5-q8ck8"] Oct 30 17:16:26.597915 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:26.597841 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ccbae02-fb91-4bd2-9e3b-3aa09cd5a415" path="/var/lib/kubelet/pods/9ccbae02-fb91-4bd2-9e3b-3aa09cd5a415/volumes" Oct 30 17:16:26.619737 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:26.619712 5052 generic.go:356] "Generic (PLEG): container finished" podID="f0607bb9-cd62-4149-a429-4051390ad2a1" containerID="fefb376a529947de9ff314f3583738dc55d221e956a1f3663fbce2a5ccc42760" exitCode=0 Oct 30 17:16:26.619846 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:26.619754 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-fxsm2" Oct 30 17:16:26.619846 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:26.619801 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-fxsm2" event={"ID":"f0607bb9-cd62-4149-a429-4051390ad2a1","Type":"ContainerDied","Data":"fefb376a529947de9ff314f3583738dc55d221e956a1f3663fbce2a5ccc42760"} Oct 30 17:16:26.619943 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:26.619850 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-fxsm2" event={"ID":"f0607bb9-cd62-4149-a429-4051390ad2a1","Type":"ContainerDied","Data":"b3bd99bdb6981a31749ea961acce1cf46dd8b641345a978e92484df179ae739e"} Oct 30 17:16:26.619943 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:26.619880 5052 scope.go:122] "RemoveContainer" containerID="fefb376a529947de9ff314f3583738dc55d221e956a1f3663fbce2a5ccc42760" Oct 30 17:16:26.630281 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:26.630254 5052 scope.go:122] "RemoveContainer" containerID="fefb376a529947de9ff314f3583738dc55d221e956a1f3663fbce2a5ccc42760" Oct 30 17:16:26.630542 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:16:26.630526 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fefb376a529947de9ff314f3583738dc55d221e956a1f3663fbce2a5ccc42760\": container with ID starting with fefb376a529947de9ff314f3583738dc55d221e956a1f3663fbce2a5ccc42760 not found: ID does not exist" containerID="fefb376a529947de9ff314f3583738dc55d221e956a1f3663fbce2a5ccc42760" Oct 30 17:16:26.630615 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:26.630549 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fefb376a529947de9ff314f3583738dc55d221e956a1f3663fbce2a5ccc42760"} err="failed to get container status \"fefb376a529947de9ff314f3583738dc55d221e956a1f3663fbce2a5ccc42760\": rpc error: code = NotFound desc = could not find container \"fefb376a529947de9ff314f3583738dc55d221e956a1f3663fbce2a5ccc42760\": container with ID starting with fefb376a529947de9ff314f3583738dc55d221e956a1f3663fbce2a5ccc42760 not found: ID does not exist" Oct 30 17:16:26.677268 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:26.677245 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-fxsm2"] Oct 30 17:16:26.680645 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:26.680622 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-fxsm2"] Oct 30 17:16:28.598200 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:28.598159 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0607bb9-cd62-4149-a429-4051390ad2a1" path="/var/lib/kubelet/pods/f0607bb9-cd62-4149-a429-4051390ad2a1/volumes" Oct 30 17:16:28.958939 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:28.958899 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-05810b-67f9c6857b-7jz5n" podUID="6136d90c-0253-4761-843b-f8a3922c93cf" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:16:32.002161 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:32.002113 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-263f00-544fdb49d4-57fv4"] Oct 30 17:16:36.311781 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:36.311748 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ddc10d-99f47dfb8-w7ktd"] Oct 30 17:16:36.312177 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:36.311972 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-ddc10d-99f47dfb8-w7ktd" podUID="b774d2ea-e597-4122-b520-7ccdec91972f" containerName="payload-processing" containerID="cri-o://22c599594fea7cb342ef291f39b94b1b3bbd26fffaa19962f4cada57ad111fdf" gracePeriod=30 Oct 30 17:16:36.348651 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:36.348622 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ddc10d-858465d86c-khzjm"] Oct 30 17:16:36.348881 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:36.348857 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ddc10d-858465d86c-khzjm" podUID="1e6b7be8-b362-4477-86b5-cf8ff0cd266f" containerName="payload-pre-processing" containerID="cri-o://566bd7adb5bd0d2bde443cda2a8b67c8a663a40494132372a366a9b92b9e9b28" gracePeriod=30 Oct 30 17:16:36.660626 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:36.653063 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ddc10d-99f47dfb8-w7ktd" Oct 30 17:16:36.743027 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:36.742989 5052 generic.go:356] "Generic (PLEG): container finished" podID="1e6b7be8-b362-4477-86b5-cf8ff0cd266f" containerID="566bd7adb5bd0d2bde443cda2a8b67c8a663a40494132372a366a9b92b9e9b28" exitCode=0 Oct 30 17:16:36.743171 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:36.743129 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ddc10d-858465d86c-khzjm" event={"ID":"1e6b7be8-b362-4477-86b5-cf8ff0cd266f","Type":"ContainerDied","Data":"566bd7adb5bd0d2bde443cda2a8b67c8a663a40494132372a366a9b92b9e9b28"} Oct 30 17:16:36.745927 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:36.745899 5052 generic.go:356] "Generic (PLEG): container finished" podID="b774d2ea-e597-4122-b520-7ccdec91972f" containerID="22c599594fea7cb342ef291f39b94b1b3bbd26fffaa19962f4cada57ad111fdf" exitCode=0 Oct 30 17:16:36.746051 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:36.745967 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ddc10d-99f47dfb8-w7ktd" event={"ID":"b774d2ea-e597-4122-b520-7ccdec91972f","Type":"ContainerDied","Data":"22c599594fea7cb342ef291f39b94b1b3bbd26fffaa19962f4cada57ad111fdf"} Oct 30 17:16:36.746051 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:36.746005 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ddc10d-99f47dfb8-w7ktd" event={"ID":"b774d2ea-e597-4122-b520-7ccdec91972f","Type":"ContainerDied","Data":"1af04d518bb57339c2f6cbeed71295f337b014f2af8b74da2d09dc2c3b2de1bc"} Oct 30 17:16:36.746051 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:36.746033 5052 scope.go:122] "RemoveContainer" containerID="22c599594fea7cb342ef291f39b94b1b3bbd26fffaa19962f4cada57ad111fdf" Oct 30 17:16:36.746223 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:36.746205 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ddc10d-99f47dfb8-w7ktd" Oct 30 17:16:36.756499 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:36.756483 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ddc10d-858465d86c-khzjm" Oct 30 17:16:36.763426 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:36.763405 5052 scope.go:122] "RemoveContainer" containerID="22c599594fea7cb342ef291f39b94b1b3bbd26fffaa19962f4cada57ad111fdf" Oct 30 17:16:36.763915 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:16:36.763878 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22c599594fea7cb342ef291f39b94b1b3bbd26fffaa19962f4cada57ad111fdf\": container with ID starting with 22c599594fea7cb342ef291f39b94b1b3bbd26fffaa19962f4cada57ad111fdf not found: ID does not exist" containerID="22c599594fea7cb342ef291f39b94b1b3bbd26fffaa19962f4cada57ad111fdf" Oct 30 17:16:36.764018 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:36.763923 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22c599594fea7cb342ef291f39b94b1b3bbd26fffaa19962f4cada57ad111fdf"} err="failed to get container status \"22c599594fea7cb342ef291f39b94b1b3bbd26fffaa19962f4cada57ad111fdf\": rpc error: code = NotFound desc = could not find container \"22c599594fea7cb342ef291f39b94b1b3bbd26fffaa19962f4cada57ad111fdf\": container with ID starting with 22c599594fea7cb342ef291f39b94b1b3bbd26fffaa19962f4cada57ad111fdf not found: ID does not exist" Oct 30 17:16:36.827126 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:36.827096 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b774d2ea-e597-4122-b520-7ccdec91972f-kube-api-access-hkq26\" (UniqueName: \"kubernetes.io/projected/b774d2ea-e597-4122-b520-7ccdec91972f-kube-api-access-hkq26\") pod \"b774d2ea-e597-4122-b520-7ccdec91972f\" (UID: \"b774d2ea-e597-4122-b520-7ccdec91972f\") " Oct 30 17:16:36.827304 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:36.827186 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b774d2ea-e597-4122-b520-7ccdec91972f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b774d2ea-e597-4122-b520-7ccdec91972f-plugins-config-volume\") pod \"b774d2ea-e597-4122-b520-7ccdec91972f\" (UID: \"b774d2ea-e597-4122-b520-7ccdec91972f\") " Oct 30 17:16:36.827545 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:36.827522 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b774d2ea-e597-4122-b520-7ccdec91972f-plugins-config-volume" pod "b774d2ea-e597-4122-b520-7ccdec91972f" (UID: "b774d2ea-e597-4122-b520-7ccdec91972f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:16:36.829240 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:36.829213 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b774d2ea-e597-4122-b520-7ccdec91972f-kube-api-access-hkq26" pod "b774d2ea-e597-4122-b520-7ccdec91972f" (UID: "b774d2ea-e597-4122-b520-7ccdec91972f"). InnerVolumeSpecName "kube-api-access-hkq26". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:16:36.928420 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:36.928386 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e6b7be8-b362-4477-86b5-cf8ff0cd266f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e6b7be8-b362-4477-86b5-cf8ff0cd266f-plugins-config-volume\") pod \"1e6b7be8-b362-4477-86b5-cf8ff0cd266f\" (UID: \"1e6b7be8-b362-4477-86b5-cf8ff0cd266f\") " Oct 30 17:16:36.928616 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:36.928521 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e6b7be8-b362-4477-86b5-cf8ff0cd266f-kube-api-access-lrr8m\" (UniqueName: \"kubernetes.io/projected/1e6b7be8-b362-4477-86b5-cf8ff0cd266f-kube-api-access-lrr8m\") pod \"1e6b7be8-b362-4477-86b5-cf8ff0cd266f\" (UID: \"1e6b7be8-b362-4477-86b5-cf8ff0cd266f\") " Oct 30 17:16:36.928807 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:36.928774 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e6b7be8-b362-4477-86b5-cf8ff0cd266f-plugins-config-volume" pod "1e6b7be8-b362-4477-86b5-cf8ff0cd266f" (UID: "1e6b7be8-b362-4477-86b5-cf8ff0cd266f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:16:36.928997 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:36.928980 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkq26\" (UniqueName: \"kubernetes.io/projected/b774d2ea-e597-4122-b520-7ccdec91972f-kube-api-access-hkq26\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:16:36.929062 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:36.929000 5052 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b774d2ea-e597-4122-b520-7ccdec91972f-plugins-config-volume\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:16:36.929062 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:36.929010 5052 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e6b7be8-b362-4477-86b5-cf8ff0cd266f-plugins-config-volume\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:16:36.930750 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:36.930727 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e6b7be8-b362-4477-86b5-cf8ff0cd266f-kube-api-access-lrr8m" pod "1e6b7be8-b362-4477-86b5-cf8ff0cd266f" (UID: "1e6b7be8-b362-4477-86b5-cf8ff0cd266f"). InnerVolumeSpecName "kube-api-access-lrr8m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:16:37.003104 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:37.003068 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-263f00-544fdb49d4-57fv4" podUID="2ba3e877-170c-44be-aad2-c96e7cf6b99f" containerName="payload-processing" containerID="cri-o://f187b659f6e6225e2b24ce32a0e56339a688c86e53bba2910fbbf8b4d4f779fb" gracePeriod=25 Oct 30 17:16:37.030631 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:37.030601 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrr8m\" (UniqueName: \"kubernetes.io/projected/1e6b7be8-b362-4477-86b5-cf8ff0cd266f-kube-api-access-lrr8m\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:16:37.116380 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:37.116347 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ddc10d-99f47dfb8-w7ktd"] Oct 30 17:16:37.130707 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:37.128867 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ddc10d-99f47dfb8-w7ktd"] Oct 30 17:16:37.283662 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:37.283635 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-263f00-544fdb49d4-57fv4" Oct 30 17:16:37.437224 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:37.437186 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ba3e877-170c-44be-aad2-c96e7cf6b99f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ba3e877-170c-44be-aad2-c96e7cf6b99f-plugins-config-volume\") pod \"2ba3e877-170c-44be-aad2-c96e7cf6b99f\" (UID: \"2ba3e877-170c-44be-aad2-c96e7cf6b99f\") " Oct 30 17:16:37.437696 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:37.437269 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ba3e877-170c-44be-aad2-c96e7cf6b99f-kube-api-access-ls97l\" (UniqueName: \"kubernetes.io/projected/2ba3e877-170c-44be-aad2-c96e7cf6b99f-kube-api-access-ls97l\") pod \"2ba3e877-170c-44be-aad2-c96e7cf6b99f\" (UID: \"2ba3e877-170c-44be-aad2-c96e7cf6b99f\") " Oct 30 17:16:37.437696 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:37.437658 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ba3e877-170c-44be-aad2-c96e7cf6b99f-plugins-config-volume" pod "2ba3e877-170c-44be-aad2-c96e7cf6b99f" (UID: "2ba3e877-170c-44be-aad2-c96e7cf6b99f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:16:37.439491 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:37.439465 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ba3e877-170c-44be-aad2-c96e7cf6b99f-kube-api-access-ls97l" pod "2ba3e877-170c-44be-aad2-c96e7cf6b99f" (UID: "2ba3e877-170c-44be-aad2-c96e7cf6b99f"). InnerVolumeSpecName "kube-api-access-ls97l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:16:37.538509 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:37.538472 5052 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ba3e877-170c-44be-aad2-c96e7cf6b99f-plugins-config-volume\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:16:37.538509 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:37.538504 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ls97l\" (UniqueName: \"kubernetes.io/projected/2ba3e877-170c-44be-aad2-c96e7cf6b99f-kube-api-access-ls97l\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:16:37.754756 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:37.754672 5052 generic.go:356] "Generic (PLEG): container finished" podID="2ba3e877-170c-44be-aad2-c96e7cf6b99f" containerID="f187b659f6e6225e2b24ce32a0e56339a688c86e53bba2910fbbf8b4d4f779fb" exitCode=0 Oct 30 17:16:37.754756 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:37.754703 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-263f00-544fdb49d4-57fv4" Oct 30 17:16:37.754756 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:37.754742 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-263f00-544fdb49d4-57fv4" event={"ID":"2ba3e877-170c-44be-aad2-c96e7cf6b99f","Type":"ContainerDied","Data":"f187b659f6e6225e2b24ce32a0e56339a688c86e53bba2910fbbf8b4d4f779fb"} Oct 30 17:16:37.755022 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:37.754796 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-263f00-544fdb49d4-57fv4" event={"ID":"2ba3e877-170c-44be-aad2-c96e7cf6b99f","Type":"ContainerDied","Data":"b19c9aff620122371e56b2fbb621f4c71b97fcb62d436f89140a65880acf8639"} Oct 30 17:16:37.755022 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:37.754826 5052 scope.go:122] "RemoveContainer" containerID="f187b659f6e6225e2b24ce32a0e56339a688c86e53bba2910fbbf8b4d4f779fb" Oct 30 17:16:37.756442 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:37.756422 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ddc10d-858465d86c-khzjm" event={"ID":"1e6b7be8-b362-4477-86b5-cf8ff0cd266f","Type":"ContainerDied","Data":"9f637c21cb81bee5df3e1e7931b70ee5aee2e5bececdc9dfba62c9404e119f19"} Oct 30 17:16:37.756542 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:37.756435 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ddc10d-858465d86c-khzjm" Oct 30 17:16:37.766640 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:37.766616 5052 scope.go:122] "RemoveContainer" containerID="f187b659f6e6225e2b24ce32a0e56339a688c86e53bba2910fbbf8b4d4f779fb" Oct 30 17:16:37.766927 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:16:37.766905 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f187b659f6e6225e2b24ce32a0e56339a688c86e53bba2910fbbf8b4d4f779fb\": container with ID starting with f187b659f6e6225e2b24ce32a0e56339a688c86e53bba2910fbbf8b4d4f779fb not found: ID does not exist" containerID="f187b659f6e6225e2b24ce32a0e56339a688c86e53bba2910fbbf8b4d4f779fb" Oct 30 17:16:37.767006 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:37.766935 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f187b659f6e6225e2b24ce32a0e56339a688c86e53bba2910fbbf8b4d4f779fb"} err="failed to get container status \"f187b659f6e6225e2b24ce32a0e56339a688c86e53bba2910fbbf8b4d4f779fb\": rpc error: code = NotFound desc = could not find container \"f187b659f6e6225e2b24ce32a0e56339a688c86e53bba2910fbbf8b4d4f779fb\": container with ID starting with f187b659f6e6225e2b24ce32a0e56339a688c86e53bba2910fbbf8b4d4f779fb not found: ID does not exist" Oct 30 17:16:37.767006 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:37.766953 5052 scope.go:122] "RemoveContainer" containerID="566bd7adb5bd0d2bde443cda2a8b67c8a663a40494132372a366a9b92b9e9b28" Oct 30 17:16:37.815124 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:37.815103 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-263f00-544fdb49d4-57fv4"] Oct 30 17:16:37.819028 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:37.819007 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-263f00-544fdb49d4-57fv4"] Oct 30 17:16:37.856906 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:37.856883 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ddc10d-858465d86c-khzjm"] Oct 30 17:16:37.859030 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:37.859011 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ddc10d-858465d86c-khzjm"] Oct 30 17:16:38.024918 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:16:38.024887 5052 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6136d90c_0253_4761_843b_f8a3922c93cf.slice/crio-cb0f06ad0af10ba3297ecb873778a0595554e2ae8d98f8c2e7f76ec0b9039d98.scope\": RecentStats: unable to find data in memory cache]" Oct 30 17:16:38.273139 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:38.273078 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-05810b-67f9c6857b-7jz5n" Oct 30 17:16:38.448555 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:38.448519 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6136d90c-0253-4761-843b-f8a3922c93cf-kube-api-access-9sfwk\" (UniqueName: \"kubernetes.io/projected/6136d90c-0253-4761-843b-f8a3922c93cf-kube-api-access-9sfwk\") pod \"6136d90c-0253-4761-843b-f8a3922c93cf\" (UID: \"6136d90c-0253-4761-843b-f8a3922c93cf\") " Oct 30 17:16:38.449088 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:38.448673 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6136d90c-0253-4761-843b-f8a3922c93cf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6136d90c-0253-4761-843b-f8a3922c93cf-plugins-config-volume\") pod \"6136d90c-0253-4761-843b-f8a3922c93cf\" (UID: \"6136d90c-0253-4761-843b-f8a3922c93cf\") " Oct 30 17:16:38.449172 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:38.449111 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6136d90c-0253-4761-843b-f8a3922c93cf-plugins-config-volume" pod "6136d90c-0253-4761-843b-f8a3922c93cf" (UID: "6136d90c-0253-4761-843b-f8a3922c93cf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:16:38.451396 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:38.451366 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6136d90c-0253-4761-843b-f8a3922c93cf-kube-api-access-9sfwk" pod "6136d90c-0253-4761-843b-f8a3922c93cf" (UID: "6136d90c-0253-4761-843b-f8a3922c93cf"). InnerVolumeSpecName "kube-api-access-9sfwk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:16:38.550450 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:38.550366 5052 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6136d90c-0253-4761-843b-f8a3922c93cf-plugins-config-volume\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:16:38.550450 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:38.550405 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9sfwk\" (UniqueName: \"kubernetes.io/projected/6136d90c-0253-4761-843b-f8a3922c93cf-kube-api-access-9sfwk\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:16:38.596993 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:38.596962 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e6b7be8-b362-4477-86b5-cf8ff0cd266f" path="/var/lib/kubelet/pods/1e6b7be8-b362-4477-86b5-cf8ff0cd266f/volumes" Oct 30 17:16:38.597474 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:38.597448 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ba3e877-170c-44be-aad2-c96e7cf6b99f" path="/var/lib/kubelet/pods/2ba3e877-170c-44be-aad2-c96e7cf6b99f/volumes" Oct 30 17:16:38.597841 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:38.597827 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b774d2ea-e597-4122-b520-7ccdec91972f" path="/var/lib/kubelet/pods/b774d2ea-e597-4122-b520-7ccdec91972f/volumes" Oct 30 17:16:38.764218 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:38.764182 5052 generic.go:356] "Generic (PLEG): container finished" podID="6136d90c-0253-4761-843b-f8a3922c93cf" containerID="cb0f06ad0af10ba3297ecb873778a0595554e2ae8d98f8c2e7f76ec0b9039d98" exitCode=0 Oct 30 17:16:38.764408 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:38.764235 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-05810b-67f9c6857b-7jz5n" Oct 30 17:16:38.764408 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:38.764274 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-05810b-67f9c6857b-7jz5n" event={"ID":"6136d90c-0253-4761-843b-f8a3922c93cf","Type":"ContainerDied","Data":"cb0f06ad0af10ba3297ecb873778a0595554e2ae8d98f8c2e7f76ec0b9039d98"} Oct 30 17:16:38.764408 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:38.764337 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-05810b-67f9c6857b-7jz5n" event={"ID":"6136d90c-0253-4761-843b-f8a3922c93cf","Type":"ContainerDied","Data":"976f7c568909d2daf7838efbdd580d5dd2280311901f18e59593d5d92cd24411"} Oct 30 17:16:38.764408 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:38.764370 5052 scope.go:122] "RemoveContainer" containerID="cb0f06ad0af10ba3297ecb873778a0595554e2ae8d98f8c2e7f76ec0b9039d98" Oct 30 17:16:38.774812 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:38.774781 5052 scope.go:122] "RemoveContainer" containerID="cb0f06ad0af10ba3297ecb873778a0595554e2ae8d98f8c2e7f76ec0b9039d98" Oct 30 17:16:38.775138 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:16:38.775115 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb0f06ad0af10ba3297ecb873778a0595554e2ae8d98f8c2e7f76ec0b9039d98\": container with ID starting with cb0f06ad0af10ba3297ecb873778a0595554e2ae8d98f8c2e7f76ec0b9039d98 not found: ID does not exist" containerID="cb0f06ad0af10ba3297ecb873778a0595554e2ae8d98f8c2e7f76ec0b9039d98" Oct 30 17:16:38.775220 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:38.775147 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb0f06ad0af10ba3297ecb873778a0595554e2ae8d98f8c2e7f76ec0b9039d98"} err="failed to get container status \"cb0f06ad0af10ba3297ecb873778a0595554e2ae8d98f8c2e7f76ec0b9039d98\": rpc error: code = NotFound desc = could not find container \"cb0f06ad0af10ba3297ecb873778a0595554e2ae8d98f8c2e7f76ec0b9039d98\": container with ID starting with cb0f06ad0af10ba3297ecb873778a0595554e2ae8d98f8c2e7f76ec0b9039d98 not found: ID does not exist" Oct 30 17:16:38.825404 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:38.825329 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-05810b-67f9c6857b-7jz5n"] Oct 30 17:16:38.838864 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:38.838840 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-05810b-67f9c6857b-7jz5n"] Oct 30 17:16:40.596930 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:40.596889 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6136d90c-0253-4761-843b-f8a3922c93cf" path="/var/lib/kubelet/pods/6136d90c-0253-4761-843b-f8a3922c93cf/volumes" Oct 30 17:16:41.427607 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:41.420042 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-263f00/distinct-2-w4-263f00-kserve-8464897cf5-q8ck8" podUID="15ed43ef-5cdd-489a-a70e-063239822ff3" containerName="main" containerID="cri-o://b44b08923a9939b80b5c8728022b13331d54380e8c8f77e8a0de2063f821bc3b" gracePeriod=45 Oct 30 17:16:41.545706 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:41.545676 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cdwcj"] Oct 30 17:16:41.695251 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:41.695175 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-263f00/distinct-2-w4-263f00-kserve-8464897cf5-q8ck8" Oct 30 17:16:41.786046 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:41.785949 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15ed43ef-5cdd-489a-a70e-063239822ff3-kube-api-access-fx2b7\" (UniqueName: \"kubernetes.io/projected/15ed43ef-5cdd-489a-a70e-063239822ff3-kube-api-access-fx2b7\") pod \"15ed43ef-5cdd-489a-a70e-063239822ff3\" (UID: \"15ed43ef-5cdd-489a-a70e-063239822ff3\") " Oct 30 17:16:41.786046 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:41.786026 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15ed43ef-5cdd-489a-a70e-063239822ff3-tls-certs\" (UniqueName: \"kubernetes.io/secret/15ed43ef-5cdd-489a-a70e-063239822ff3-tls-certs\") pod \"15ed43ef-5cdd-489a-a70e-063239822ff3\" (UID: \"15ed43ef-5cdd-489a-a70e-063239822ff3\") " Oct 30 17:16:41.786422 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:41.786090 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15ed43ef-5cdd-489a-a70e-063239822ff3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15ed43ef-5cdd-489a-a70e-063239822ff3-model-cache\") pod \"15ed43ef-5cdd-489a-a70e-063239822ff3\" (UID: \"15ed43ef-5cdd-489a-a70e-063239822ff3\") " Oct 30 17:16:41.786422 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:41.786136 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15ed43ef-5cdd-489a-a70e-063239822ff3-home\" (UniqueName: \"kubernetes.io/empty-dir/15ed43ef-5cdd-489a-a70e-063239822ff3-home\") pod \"15ed43ef-5cdd-489a-a70e-063239822ff3\" (UID: \"15ed43ef-5cdd-489a-a70e-063239822ff3\") " Oct 30 17:16:41.786422 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:41.786244 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15ed43ef-5cdd-489a-a70e-063239822ff3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15ed43ef-5cdd-489a-a70e-063239822ff3-tmp-dir\") pod \"15ed43ef-5cdd-489a-a70e-063239822ff3\" (UID: \"15ed43ef-5cdd-489a-a70e-063239822ff3\") " Oct 30 17:16:41.786422 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:41.786274 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15ed43ef-5cdd-489a-a70e-063239822ff3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/15ed43ef-5cdd-489a-a70e-063239822ff3-dshm\") pod \"15ed43ef-5cdd-489a-a70e-063239822ff3\" (UID: \"15ed43ef-5cdd-489a-a70e-063239822ff3\") " Oct 30 17:16:41.788777 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:41.788744 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15ed43ef-5cdd-489a-a70e-063239822ff3-kube-api-access-fx2b7" pod "15ed43ef-5cdd-489a-a70e-063239822ff3" (UID: "15ed43ef-5cdd-489a-a70e-063239822ff3"). InnerVolumeSpecName "kube-api-access-fx2b7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:16:41.789212 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:41.789178 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15ed43ef-5cdd-489a-a70e-063239822ff3-dshm" pod "15ed43ef-5cdd-489a-a70e-063239822ff3" (UID: "15ed43ef-5cdd-489a-a70e-063239822ff3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:16:41.789928 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:41.789905 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15ed43ef-5cdd-489a-a70e-063239822ff3-home" pod "15ed43ef-5cdd-489a-a70e-063239822ff3" (UID: "15ed43ef-5cdd-489a-a70e-063239822ff3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:16:41.790288 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:41.790213 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15ed43ef-5cdd-489a-a70e-063239822ff3-model-cache" pod "15ed43ef-5cdd-489a-a70e-063239822ff3" (UID: "15ed43ef-5cdd-489a-a70e-063239822ff3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:16:41.790676 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:41.790601 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15ed43ef-5cdd-489a-a70e-063239822ff3-tmp-dir" pod "15ed43ef-5cdd-489a-a70e-063239822ff3" (UID: "15ed43ef-5cdd-489a-a70e-063239822ff3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:16:41.791400 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:41.791375 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15ed43ef-5cdd-489a-a70e-063239822ff3-tls-certs" pod "15ed43ef-5cdd-489a-a70e-063239822ff3" (UID: "15ed43ef-5cdd-489a-a70e-063239822ff3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:16:41.791735 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:41.791707 5052 generic.go:356] "Generic (PLEG): container finished" podID="15ed43ef-5cdd-489a-a70e-063239822ff3" containerID="b44b08923a9939b80b5c8728022b13331d54380e8c8f77e8a0de2063f821bc3b" exitCode=0 Oct 30 17:16:41.791873 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:41.791854 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-263f00/distinct-2-w4-263f00-kserve-8464897cf5-q8ck8" event={"ID":"15ed43ef-5cdd-489a-a70e-063239822ff3","Type":"ContainerDied","Data":"b44b08923a9939b80b5c8728022b13331d54380e8c8f77e8a0de2063f821bc3b"} Oct 30 17:16:41.791944 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:41.791895 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-263f00/distinct-2-w4-263f00-kserve-8464897cf5-q8ck8" event={"ID":"15ed43ef-5cdd-489a-a70e-063239822ff3","Type":"ContainerDied","Data":"638469e57a3882f5a49934f8851d1fb748ac6f241ee08e4b2c7878637a053529"} Oct 30 17:16:41.791944 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:41.791921 5052 scope.go:122] "RemoveContainer" containerID="b44b08923a9939b80b5c8728022b13331d54380e8c8f77e8a0de2063f821bc3b" Oct 30 17:16:41.792054 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:41.792026 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-263f00/distinct-2-w4-263f00-kserve-8464897cf5-q8ck8" Oct 30 17:16:41.822808 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:41.822714 5052 scope.go:122] "RemoveContainer" containerID="b44b08923a9939b80b5c8728022b13331d54380e8c8f77e8a0de2063f821bc3b" Oct 30 17:16:41.823137 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:16:41.823067 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b44b08923a9939b80b5c8728022b13331d54380e8c8f77e8a0de2063f821bc3b\": container with ID starting with b44b08923a9939b80b5c8728022b13331d54380e8c8f77e8a0de2063f821bc3b not found: ID does not exist" containerID="b44b08923a9939b80b5c8728022b13331d54380e8c8f77e8a0de2063f821bc3b" Oct 30 17:16:41.823137 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:41.823109 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b44b08923a9939b80b5c8728022b13331d54380e8c8f77e8a0de2063f821bc3b"} err="failed to get container status \"b44b08923a9939b80b5c8728022b13331d54380e8c8f77e8a0de2063f821bc3b\": rpc error: code = NotFound desc = could not find container \"b44b08923a9939b80b5c8728022b13331d54380e8c8f77e8a0de2063f821bc3b\": container with ID starting with b44b08923a9939b80b5c8728022b13331d54380e8c8f77e8a0de2063f821bc3b not found: ID does not exist" Oct 30 17:16:41.859142 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:41.859117 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-263f00/distinct-2-w4-263f00-kserve-8464897cf5-q8ck8"] Oct 30 17:16:41.862844 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:41.862817 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-263f00/distinct-2-w4-263f00-kserve-8464897cf5-q8ck8"] Oct 30 17:16:41.888216 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:41.888191 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fx2b7\" (UniqueName: \"kubernetes.io/projected/15ed43ef-5cdd-489a-a70e-063239822ff3-kube-api-access-fx2b7\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:16:41.888216 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:41.888214 5052 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15ed43ef-5cdd-489a-a70e-063239822ff3-tls-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:16:41.888353 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:41.888224 5052 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15ed43ef-5cdd-489a-a70e-063239822ff3-model-cache\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:16:41.888353 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:41.888234 5052 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15ed43ef-5cdd-489a-a70e-063239822ff3-home\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:16:41.888353 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:41.888241 5052 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15ed43ef-5cdd-489a-a70e-063239822ff3-tmp-dir\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:16:41.888353 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:41.888248 5052 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15ed43ef-5cdd-489a-a70e-063239822ff3-dshm\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:16:42.597480 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:42.597448 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15ed43ef-5cdd-489a-a70e-063239822ff3" path="/var/lib/kubelet/pods/15ed43ef-5cdd-489a-a70e-063239822ff3/volumes" Oct 30 17:16:44.580571 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:44.580532 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-ddc10d-openshift-default-56bfdd795c-rjzgr"] Oct 30 17:16:44.581094 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:44.580810 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-ddc10d-openshift-default-56bfdd795c-rjzgr" podUID="0261ebf2-6dfd-4df5-b5de-4b7207ca761b" containerName="istio-proxy" containerID="cri-o://c0565b2a4bdbe56e618b4c6badff19e973ba08732d43eec64ab1afa4a4d38ce9" gracePeriod=30 Oct 30 17:16:46.442890 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:46.442850 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-tjv7v"] Oct 30 17:16:46.443384 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:46.443113 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-tjv7v" podUID="39f0bd01-e73f-4f31-a76e-ad6788799f3f" containerName="istio-proxy" containerID="cri-o://afb120f27cfb49af3561b6bddbcabecc7a0086ef4bac62ffe8af92505f99290d" gracePeriod=30 Oct 30 17:16:48.840756 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:16:48.840720 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 17:16:49.845327 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:49.845262 5052 generic.go:356] "Generic (PLEG): container finished" podID="0261ebf2-6dfd-4df5-b5de-4b7207ca761b" containerID="c0565b2a4bdbe56e618b4c6badff19e973ba08732d43eec64ab1afa4a4d38ce9" exitCode=0 Oct 30 17:16:49.845748 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:49.845328 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ddc10d-openshift-default-56bfdd795c-rjzgr" event={"ID":"0261ebf2-6dfd-4df5-b5de-4b7207ca761b","Type":"ContainerDied","Data":"c0565b2a4bdbe56e618b4c6badff19e973ba08732d43eec64ab1afa4a4d38ce9"} Oct 30 17:16:49.845748 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:49.845381 5052 scope.go:122] "RemoveContainer" containerID="679ff824e61208b26c583fee45e391f0e33d81c23c74fcb9f7cc6f1a87d566ab" Oct 30 17:16:49.866676 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:49.866655 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-ddc10d-openshift-default-56bfdd795c-rjzgr" Oct 30 17:16:49.977352 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:49.977318 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0261ebf2-6dfd-4df5-b5de-4b7207ca761b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0261ebf2-6dfd-4df5-b5de-4b7207ca761b-istio-data\") pod \"0261ebf2-6dfd-4df5-b5de-4b7207ca761b\" (UID: \"0261ebf2-6dfd-4df5-b5de-4b7207ca761b\") " Oct 30 17:16:49.977494 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:49.977361 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0261ebf2-6dfd-4df5-b5de-4b7207ca761b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0261ebf2-6dfd-4df5-b5de-4b7207ca761b-istio-podinfo\") pod \"0261ebf2-6dfd-4df5-b5de-4b7207ca761b\" (UID: \"0261ebf2-6dfd-4df5-b5de-4b7207ca761b\") " Oct 30 17:16:49.977494 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:49.977391 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0261ebf2-6dfd-4df5-b5de-4b7207ca761b-istio-token\" (UniqueName: \"kubernetes.io/projected/0261ebf2-6dfd-4df5-b5de-4b7207ca761b-istio-token\") pod \"0261ebf2-6dfd-4df5-b5de-4b7207ca761b\" (UID: \"0261ebf2-6dfd-4df5-b5de-4b7207ca761b\") " Oct 30 17:16:49.977494 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:49.977434 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0261ebf2-6dfd-4df5-b5de-4b7207ca761b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0261ebf2-6dfd-4df5-b5de-4b7207ca761b-istio-envoy\") pod \"0261ebf2-6dfd-4df5-b5de-4b7207ca761b\" (UID: \"0261ebf2-6dfd-4df5-b5de-4b7207ca761b\") " Oct 30 17:16:49.977494 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:49.977490 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0261ebf2-6dfd-4df5-b5de-4b7207ca761b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0261ebf2-6dfd-4df5-b5de-4b7207ca761b-credential-socket\") pod \"0261ebf2-6dfd-4df5-b5de-4b7207ca761b\" (UID: \"0261ebf2-6dfd-4df5-b5de-4b7207ca761b\") " Oct 30 17:16:49.977752 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:49.977552 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0261ebf2-6dfd-4df5-b5de-4b7207ca761b-kube-api-access-8rqqb\" (UniqueName: \"kubernetes.io/projected/0261ebf2-6dfd-4df5-b5de-4b7207ca761b-kube-api-access-8rqqb\") pod \"0261ebf2-6dfd-4df5-b5de-4b7207ca761b\" (UID: \"0261ebf2-6dfd-4df5-b5de-4b7207ca761b\") " Oct 30 17:16:49.977752 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:49.977627 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0261ebf2-6dfd-4df5-b5de-4b7207ca761b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0261ebf2-6dfd-4df5-b5de-4b7207ca761b-istiod-ca-cert\") pod \"0261ebf2-6dfd-4df5-b5de-4b7207ca761b\" (UID: \"0261ebf2-6dfd-4df5-b5de-4b7207ca761b\") " Oct 30 17:16:49.977752 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:49.977653 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0261ebf2-6dfd-4df5-b5de-4b7207ca761b-istio-data" pod "0261ebf2-6dfd-4df5-b5de-4b7207ca761b" (UID: "0261ebf2-6dfd-4df5-b5de-4b7207ca761b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:16:49.977752 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:49.977659 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0261ebf2-6dfd-4df5-b5de-4b7207ca761b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0261ebf2-6dfd-4df5-b5de-4b7207ca761b-workload-socket\") pod \"0261ebf2-6dfd-4df5-b5de-4b7207ca761b\" (UID: \"0261ebf2-6dfd-4df5-b5de-4b7207ca761b\") " Oct 30 17:16:49.977752 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:49.977688 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0261ebf2-6dfd-4df5-b5de-4b7207ca761b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0261ebf2-6dfd-4df5-b5de-4b7207ca761b-workload-certs\") pod \"0261ebf2-6dfd-4df5-b5de-4b7207ca761b\" (UID: \"0261ebf2-6dfd-4df5-b5de-4b7207ca761b\") " Oct 30 17:16:49.978021 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:49.977866 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0261ebf2-6dfd-4df5-b5de-4b7207ca761b-credential-socket" pod "0261ebf2-6dfd-4df5-b5de-4b7207ca761b" (UID: "0261ebf2-6dfd-4df5-b5de-4b7207ca761b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:16:49.978086 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:49.978064 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0261ebf2-6dfd-4df5-b5de-4b7207ca761b-workload-certs" pod "0261ebf2-6dfd-4df5-b5de-4b7207ca761b" (UID: "0261ebf2-6dfd-4df5-b5de-4b7207ca761b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:16:49.978132 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:49.978086 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0261ebf2-6dfd-4df5-b5de-4b7207ca761b-workload-socket" pod "0261ebf2-6dfd-4df5-b5de-4b7207ca761b" (UID: "0261ebf2-6dfd-4df5-b5de-4b7207ca761b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:16:49.978132 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:49.978090 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0261ebf2-6dfd-4df5-b5de-4b7207ca761b-istiod-ca-cert" pod "0261ebf2-6dfd-4df5-b5de-4b7207ca761b" (UID: "0261ebf2-6dfd-4df5-b5de-4b7207ca761b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:16:49.978275 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:49.978257 5052 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0261ebf2-6dfd-4df5-b5de-4b7207ca761b-istio-data\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:16:49.978323 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:49.978283 5052 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0261ebf2-6dfd-4df5-b5de-4b7207ca761b-credential-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:16:49.978323 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:49.978298 5052 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0261ebf2-6dfd-4df5-b5de-4b7207ca761b-istiod-ca-cert\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:16:49.978323 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:49.978312 5052 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0261ebf2-6dfd-4df5-b5de-4b7207ca761b-workload-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:16:49.978428 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:49.978325 5052 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0261ebf2-6dfd-4df5-b5de-4b7207ca761b-workload-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:16:49.979822 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:49.979799 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0261ebf2-6dfd-4df5-b5de-4b7207ca761b-istio-token" pod "0261ebf2-6dfd-4df5-b5de-4b7207ca761b" (UID: "0261ebf2-6dfd-4df5-b5de-4b7207ca761b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:16:49.979938 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:49.979877 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0261ebf2-6dfd-4df5-b5de-4b7207ca761b-kube-api-access-8rqqb" pod "0261ebf2-6dfd-4df5-b5de-4b7207ca761b" (UID: "0261ebf2-6dfd-4df5-b5de-4b7207ca761b"). InnerVolumeSpecName "kube-api-access-8rqqb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:16:49.980062 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:49.980045 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0261ebf2-6dfd-4df5-b5de-4b7207ca761b-istio-envoy" pod "0261ebf2-6dfd-4df5-b5de-4b7207ca761b" (UID: "0261ebf2-6dfd-4df5-b5de-4b7207ca761b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:16:49.980123 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:49.980051 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0261ebf2-6dfd-4df5-b5de-4b7207ca761b-istio-podinfo" pod "0261ebf2-6dfd-4df5-b5de-4b7207ca761b" (UID: "0261ebf2-6dfd-4df5-b5de-4b7207ca761b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:16:50.079102 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:50.079070 5052 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0261ebf2-6dfd-4df5-b5de-4b7207ca761b-istio-podinfo\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:16:50.079102 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:50.079098 5052 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0261ebf2-6dfd-4df5-b5de-4b7207ca761b-istio-token\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:16:50.079102 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:50.079107 5052 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0261ebf2-6dfd-4df5-b5de-4b7207ca761b-istio-envoy\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:16:50.079391 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:50.079117 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rqqb\" (UniqueName: \"kubernetes.io/projected/0261ebf2-6dfd-4df5-b5de-4b7207ca761b-kube-api-access-8rqqb\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:16:50.313182 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:50.313152 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-0b2348-777785f486-tcq45"] Oct 30 17:16:50.313455 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:50.313407 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-0b2348-777785f486-tcq45" podUID="0f58433e-03f7-4e7c-95b5-3d9871cce2f5" containerName="maas-api" containerID="cri-o://3203f0db909bf86e8cc44e6502b4bf7bd64847ae8d534960420e6ad6e531c69c" gracePeriod=30 Oct 30 17:16:50.578172 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:50.578138 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-0b2348-777785f486-tcq45" Oct 30 17:16:50.684842 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:50.684757 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f58433e-03f7-4e7c-95b5-3d9871cce2f5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f58433e-03f7-4e7c-95b5-3d9871cce2f5-metrics-certs\") pod \"0f58433e-03f7-4e7c-95b5-3d9871cce2f5\" (UID: \"0f58433e-03f7-4e7c-95b5-3d9871cce2f5\") " Oct 30 17:16:50.684842 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:50.684801 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f58433e-03f7-4e7c-95b5-3d9871cce2f5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f58433e-03f7-4e7c-95b5-3d9871cce2f5-openshift-service-ca\") pod \"0f58433e-03f7-4e7c-95b5-3d9871cce2f5\" (UID: \"0f58433e-03f7-4e7c-95b5-3d9871cce2f5\") " Oct 30 17:16:50.685090 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:50.684975 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f58433e-03f7-4e7c-95b5-3d9871cce2f5-kube-api-access-xg5jp\" (UniqueName: \"kubernetes.io/projected/0f58433e-03f7-4e7c-95b5-3d9871cce2f5-kube-api-access-xg5jp\") pod \"0f58433e-03f7-4e7c-95b5-3d9871cce2f5\" (UID: \"0f58433e-03f7-4e7c-95b5-3d9871cce2f5\") " Oct 30 17:16:50.685090 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:50.685024 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f58433e-03f7-4e7c-95b5-3d9871cce2f5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f58433e-03f7-4e7c-95b5-3d9871cce2f5-maas-api-tls\") pod \"0f58433e-03f7-4e7c-95b5-3d9871cce2f5\" (UID: \"0f58433e-03f7-4e7c-95b5-3d9871cce2f5\") " Oct 30 17:16:50.685414 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:50.685244 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f58433e-03f7-4e7c-95b5-3d9871cce2f5-openshift-service-ca" pod "0f58433e-03f7-4e7c-95b5-3d9871cce2f5" (UID: "0f58433e-03f7-4e7c-95b5-3d9871cce2f5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:16:50.685906 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:50.685880 5052 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f58433e-03f7-4e7c-95b5-3d9871cce2f5-openshift-service-ca\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:16:50.687151 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:50.687126 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f58433e-03f7-4e7c-95b5-3d9871cce2f5-metrics-certs" pod "0f58433e-03f7-4e7c-95b5-3d9871cce2f5" (UID: "0f58433e-03f7-4e7c-95b5-3d9871cce2f5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:16:50.687371 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:50.687351 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f58433e-03f7-4e7c-95b5-3d9871cce2f5-kube-api-access-xg5jp" pod "0f58433e-03f7-4e7c-95b5-3d9871cce2f5" (UID: "0f58433e-03f7-4e7c-95b5-3d9871cce2f5"). InnerVolumeSpecName "kube-api-access-xg5jp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:16:50.687444 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:50.687403 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f58433e-03f7-4e7c-95b5-3d9871cce2f5-maas-api-tls" pod "0f58433e-03f7-4e7c-95b5-3d9871cce2f5" (UID: "0f58433e-03f7-4e7c-95b5-3d9871cce2f5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:16:50.791798 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:50.791753 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xg5jp\" (UniqueName: \"kubernetes.io/projected/0f58433e-03f7-4e7c-95b5-3d9871cce2f5-kube-api-access-xg5jp\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:16:50.791798 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:50.791797 5052 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f58433e-03f7-4e7c-95b5-3d9871cce2f5-maas-api-tls\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:16:50.792035 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:50.791812 5052 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f58433e-03f7-4e7c-95b5-3d9871cce2f5-metrics-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:16:50.852050 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:50.852018 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-ddc10d-openshift-default-56bfdd795c-rjzgr" Oct 30 17:16:50.852540 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:50.852023 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ddc10d-openshift-default-56bfdd795c-rjzgr" event={"ID":"0261ebf2-6dfd-4df5-b5de-4b7207ca761b","Type":"ContainerDied","Data":"b5ef86695b30809c68ab785bd492d34cbeacb52fd5175c944c2454cb41e50e7f"} Oct 30 17:16:50.852540 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:50.852165 5052 scope.go:122] "RemoveContainer" containerID="c0565b2a4bdbe56e618b4c6badff19e973ba08732d43eec64ab1afa4a4d38ce9" Oct 30 17:16:50.853482 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:50.853460 5052 generic.go:356] "Generic (PLEG): container finished" podID="0f58433e-03f7-4e7c-95b5-3d9871cce2f5" containerID="3203f0db909bf86e8cc44e6502b4bf7bd64847ae8d534960420e6ad6e531c69c" exitCode=0 Oct 30 17:16:50.853557 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:50.853502 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-0b2348-777785f486-tcq45" Oct 30 17:16:50.853662 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:50.853552 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-0b2348-777785f486-tcq45" event={"ID":"0f58433e-03f7-4e7c-95b5-3d9871cce2f5","Type":"ContainerDied","Data":"3203f0db909bf86e8cc44e6502b4bf7bd64847ae8d534960420e6ad6e531c69c"} Oct 30 17:16:50.853662 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:50.853640 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-0b2348-777785f486-tcq45" event={"ID":"0f58433e-03f7-4e7c-95b5-3d9871cce2f5","Type":"ContainerDied","Data":"2b7b4fd283778928c9dd8c3273219592816128353d42322f09d6eec6a9d43c25"} Oct 30 17:16:50.862781 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:50.862760 5052 scope.go:122] "RemoveContainer" containerID="3203f0db909bf86e8cc44e6502b4bf7bd64847ae8d534960420e6ad6e531c69c" Oct 30 17:16:50.871917 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:50.871896 5052 scope.go:122] "RemoveContainer" containerID="3203f0db909bf86e8cc44e6502b4bf7bd64847ae8d534960420e6ad6e531c69c" Oct 30 17:16:50.872197 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:16:50.872180 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3203f0db909bf86e8cc44e6502b4bf7bd64847ae8d534960420e6ad6e531c69c\": container with ID starting with 3203f0db909bf86e8cc44e6502b4bf7bd64847ae8d534960420e6ad6e531c69c not found: ID does not exist" containerID="3203f0db909bf86e8cc44e6502b4bf7bd64847ae8d534960420e6ad6e531c69c" Oct 30 17:16:50.872252 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:50.872205 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3203f0db909bf86e8cc44e6502b4bf7bd64847ae8d534960420e6ad6e531c69c"} err="failed to get container status \"3203f0db909bf86e8cc44e6502b4bf7bd64847ae8d534960420e6ad6e531c69c\": rpc error: code = NotFound desc = could not find container \"3203f0db909bf86e8cc44e6502b4bf7bd64847ae8d534960420e6ad6e531c69c\": container with ID starting with 3203f0db909bf86e8cc44e6502b4bf7bd64847ae8d534960420e6ad6e531c69c not found: ID does not exist" Oct 30 17:16:50.941153 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:50.941052 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-ddc10d-openshift-default-56bfdd795c-rjzgr"] Oct 30 17:16:50.944871 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:50.944844 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-ddc10d-openshift-default-56bfdd795c-rjzgr"] Oct 30 17:16:50.980079 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:50.980037 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-0b2348-777785f486-tcq45"] Oct 30 17:16:50.985990 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:50.985961 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-0b2348-777785f486-tcq45"] Oct 30 17:16:51.708033 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:51.707982 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-tjv7v" Oct 30 17:16:51.803877 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:51.803843 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39f0bd01-e73f-4f31-a76e-ad6788799f3f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39f0bd01-e73f-4f31-a76e-ad6788799f3f-workload-certs\") pod \"39f0bd01-e73f-4f31-a76e-ad6788799f3f\" (UID: \"39f0bd01-e73f-4f31-a76e-ad6788799f3f\") " Oct 30 17:16:51.804059 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:51.803908 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39f0bd01-e73f-4f31-a76e-ad6788799f3f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39f0bd01-e73f-4f31-a76e-ad6788799f3f-workload-socket\") pod \"39f0bd01-e73f-4f31-a76e-ad6788799f3f\" (UID: \"39f0bd01-e73f-4f31-a76e-ad6788799f3f\") " Oct 30 17:16:51.804059 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:51.803942 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39f0bd01-e73f-4f31-a76e-ad6788799f3f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39f0bd01-e73f-4f31-a76e-ad6788799f3f-istio-data\") pod \"39f0bd01-e73f-4f31-a76e-ad6788799f3f\" (UID: \"39f0bd01-e73f-4f31-a76e-ad6788799f3f\") " Oct 30 17:16:51.804059 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:51.803972 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39f0bd01-e73f-4f31-a76e-ad6788799f3f-istio-token\" (UniqueName: \"kubernetes.io/projected/39f0bd01-e73f-4f31-a76e-ad6788799f3f-istio-token\") pod \"39f0bd01-e73f-4f31-a76e-ad6788799f3f\" (UID: \"39f0bd01-e73f-4f31-a76e-ad6788799f3f\") " Oct 30 17:16:51.804059 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:51.804026 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39f0bd01-e73f-4f31-a76e-ad6788799f3f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39f0bd01-e73f-4f31-a76e-ad6788799f3f-istiod-ca-cert\") pod \"39f0bd01-e73f-4f31-a76e-ad6788799f3f\" (UID: \"39f0bd01-e73f-4f31-a76e-ad6788799f3f\") " Oct 30 17:16:51.804284 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:51.804118 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39f0bd01-e73f-4f31-a76e-ad6788799f3f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39f0bd01-e73f-4f31-a76e-ad6788799f3f-credential-socket\") pod \"39f0bd01-e73f-4f31-a76e-ad6788799f3f\" (UID: \"39f0bd01-e73f-4f31-a76e-ad6788799f3f\") " Oct 30 17:16:51.804284 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:51.804144 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/39f0bd01-e73f-4f31-a76e-ad6788799f3f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39f0bd01-e73f-4f31-a76e-ad6788799f3f-istio-podinfo\") pod \"39f0bd01-e73f-4f31-a76e-ad6788799f3f\" (UID: \"39f0bd01-e73f-4f31-a76e-ad6788799f3f\") " Oct 30 17:16:51.804284 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:51.804189 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39f0bd01-e73f-4f31-a76e-ad6788799f3f-workload-certs" pod "39f0bd01-e73f-4f31-a76e-ad6788799f3f" (UID: "39f0bd01-e73f-4f31-a76e-ad6788799f3f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:16:51.804284 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:51.804198 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39f0bd01-e73f-4f31-a76e-ad6788799f3f-kube-api-access-nqw26\" (UniqueName: \"kubernetes.io/projected/39f0bd01-e73f-4f31-a76e-ad6788799f3f-kube-api-access-nqw26\") pod \"39f0bd01-e73f-4f31-a76e-ad6788799f3f\" (UID: \"39f0bd01-e73f-4f31-a76e-ad6788799f3f\") " Oct 30 17:16:51.804504 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:51.804293 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39f0bd01-e73f-4f31-a76e-ad6788799f3f-workload-socket" pod "39f0bd01-e73f-4f31-a76e-ad6788799f3f" (UID: "39f0bd01-e73f-4f31-a76e-ad6788799f3f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:16:51.804504 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:51.804302 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39f0bd01-e73f-4f31-a76e-ad6788799f3f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39f0bd01-e73f-4f31-a76e-ad6788799f3f-istio-envoy\") pod \"39f0bd01-e73f-4f31-a76e-ad6788799f3f\" (UID: \"39f0bd01-e73f-4f31-a76e-ad6788799f3f\") " Oct 30 17:16:51.804504 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:51.804491 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39f0bd01-e73f-4f31-a76e-ad6788799f3f-credential-socket" pod "39f0bd01-e73f-4f31-a76e-ad6788799f3f" (UID: "39f0bd01-e73f-4f31-a76e-ad6788799f3f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:16:51.804722 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:51.804524 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39f0bd01-e73f-4f31-a76e-ad6788799f3f-istio-data" pod "39f0bd01-e73f-4f31-a76e-ad6788799f3f" (UID: "39f0bd01-e73f-4f31-a76e-ad6788799f3f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:16:51.805027 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:51.804827 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39f0bd01-e73f-4f31-a76e-ad6788799f3f-istiod-ca-cert" pod "39f0bd01-e73f-4f31-a76e-ad6788799f3f" (UID: "39f0bd01-e73f-4f31-a76e-ad6788799f3f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:16:51.805027 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:51.804950 5052 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39f0bd01-e73f-4f31-a76e-ad6788799f3f-workload-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:16:51.805027 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:51.804970 5052 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39f0bd01-e73f-4f31-a76e-ad6788799f3f-workload-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:16:51.805027 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:51.804986 5052 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39f0bd01-e73f-4f31-a76e-ad6788799f3f-istio-data\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:16:51.805027 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:51.805001 5052 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39f0bd01-e73f-4f31-a76e-ad6788799f3f-istiod-ca-cert\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:16:51.805027 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:51.805013 5052 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39f0bd01-e73f-4f31-a76e-ad6788799f3f-credential-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:16:51.812622 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:51.807712 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/39f0bd01-e73f-4f31-a76e-ad6788799f3f-istio-podinfo" pod "39f0bd01-e73f-4f31-a76e-ad6788799f3f" (UID: "39f0bd01-e73f-4f31-a76e-ad6788799f3f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:16:51.812622 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:51.807720 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39f0bd01-e73f-4f31-a76e-ad6788799f3f-istio-envoy" pod "39f0bd01-e73f-4f31-a76e-ad6788799f3f" (UID: "39f0bd01-e73f-4f31-a76e-ad6788799f3f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:16:51.812622 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:51.807734 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39f0bd01-e73f-4f31-a76e-ad6788799f3f-istio-token" pod "39f0bd01-e73f-4f31-a76e-ad6788799f3f" (UID: "39f0bd01-e73f-4f31-a76e-ad6788799f3f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:16:51.812622 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:51.807786 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39f0bd01-e73f-4f31-a76e-ad6788799f3f-kube-api-access-nqw26" pod "39f0bd01-e73f-4f31-a76e-ad6788799f3f" (UID: "39f0bd01-e73f-4f31-a76e-ad6788799f3f"). InnerVolumeSpecName "kube-api-access-nqw26". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:16:51.862684 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:51.862653 5052 generic.go:356] "Generic (PLEG): container finished" podID="39f0bd01-e73f-4f31-a76e-ad6788799f3f" containerID="afb120f27cfb49af3561b6bddbcabecc7a0086ef4bac62ffe8af92505f99290d" exitCode=0 Oct 30 17:16:51.863069 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:51.862701 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-tjv7v" Oct 30 17:16:51.863069 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:51.862740 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-tjv7v" event={"ID":"39f0bd01-e73f-4f31-a76e-ad6788799f3f","Type":"ContainerDied","Data":"afb120f27cfb49af3561b6bddbcabecc7a0086ef4bac62ffe8af92505f99290d"} Oct 30 17:16:51.863069 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:51.862794 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-tjv7v" event={"ID":"39f0bd01-e73f-4f31-a76e-ad6788799f3f","Type":"ContainerDied","Data":"fb9e1b9e24e357792a51ee099bdd070911d70d891f05d5db65745f6a5adc3aa9"} Oct 30 17:16:51.863069 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:51.862822 5052 scope.go:122] "RemoveContainer" containerID="afb120f27cfb49af3561b6bddbcabecc7a0086ef4bac62ffe8af92505f99290d" Oct 30 17:16:51.873294 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:51.873276 5052 scope.go:122] "RemoveContainer" containerID="afb120f27cfb49af3561b6bddbcabecc7a0086ef4bac62ffe8af92505f99290d" Oct 30 17:16:51.873565 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:16:51.873547 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afb120f27cfb49af3561b6bddbcabecc7a0086ef4bac62ffe8af92505f99290d\": container with ID starting with afb120f27cfb49af3561b6bddbcabecc7a0086ef4bac62ffe8af92505f99290d not found: ID does not exist" containerID="afb120f27cfb49af3561b6bddbcabecc7a0086ef4bac62ffe8af92505f99290d" Oct 30 17:16:51.873645 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:51.873573 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afb120f27cfb49af3561b6bddbcabecc7a0086ef4bac62ffe8af92505f99290d"} err="failed to get container status \"afb120f27cfb49af3561b6bddbcabecc7a0086ef4bac62ffe8af92505f99290d\": rpc error: code = NotFound desc = could not find container \"afb120f27cfb49af3561b6bddbcabecc7a0086ef4bac62ffe8af92505f99290d\": container with ID starting with afb120f27cfb49af3561b6bddbcabecc7a0086ef4bac62ffe8af92505f99290d not found: ID does not exist" Oct 30 17:16:51.906618 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:51.906593 5052 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39f0bd01-e73f-4f31-a76e-ad6788799f3f-istio-token\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:16:51.906618 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:51.906614 5052 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39f0bd01-e73f-4f31-a76e-ad6788799f3f-istio-podinfo\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:16:51.906745 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:51.906627 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqw26\" (UniqueName: \"kubernetes.io/projected/39f0bd01-e73f-4f31-a76e-ad6788799f3f-kube-api-access-nqw26\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:16:51.906745 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:51.906639 5052 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39f0bd01-e73f-4f31-a76e-ad6788799f3f-istio-envoy\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:16:51.918479 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:51.918459 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-tjv7v"] Oct 30 17:16:51.922494 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:51.922474 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-tjv7v"] Oct 30 17:16:52.597729 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:52.597693 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0261ebf2-6dfd-4df5-b5de-4b7207ca761b" path="/var/lib/kubelet/pods/0261ebf2-6dfd-4df5-b5de-4b7207ca761b/volumes" Oct 30 17:16:52.598195 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:52.598181 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f58433e-03f7-4e7c-95b5-3d9871cce2f5" path="/var/lib/kubelet/pods/0f58433e-03f7-4e7c-95b5-3d9871cce2f5/volumes" Oct 30 17:16:52.598649 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:52.598635 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39f0bd01-e73f-4f31-a76e-ad6788799f3f" path="/var/lib/kubelet/pods/39f0bd01-e73f-4f31-a76e-ad6788799f3f/volumes" Oct 30 17:16:56.342247 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:56.342208 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-263f00-64bd44f46f-9qwbt"] Oct 30 17:16:56.342772 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:56.342744 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-263f00-64bd44f46f-9qwbt" podUID="e98c90ff-9a9a-4e40-aeba-556d98be49a2" containerName="maas-api" containerID="cri-o://9ebcf9cd8aecab35e3019f3833c85e00d7e3c5ad9da1109061d418a396ef186c" gracePeriod=30 Oct 30 17:16:56.381721 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:56.381677 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-263f00-64bd44f46f-9qwbt" podUID="e98c90ff-9a9a-4e40-aeba-556d98be49a2" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.59:8443/health\": dial tcp 10.132.0.59:8443: connect: connection refused" Oct 30 17:16:56.621218 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:56.621145 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-263f00-64bd44f46f-9qwbt" Oct 30 17:16:56.774500 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:56.774431 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e98c90ff-9a9a-4e40-aeba-556d98be49a2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e98c90ff-9a9a-4e40-aeba-556d98be49a2-metrics-certs\") pod \"e98c90ff-9a9a-4e40-aeba-556d98be49a2\" (UID: \"e98c90ff-9a9a-4e40-aeba-556d98be49a2\") " Oct 30 17:16:56.774683 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:56.774558 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e98c90ff-9a9a-4e40-aeba-556d98be49a2-kube-api-access-ldwzb\" (UniqueName: \"kubernetes.io/projected/e98c90ff-9a9a-4e40-aeba-556d98be49a2-kube-api-access-ldwzb\") pod \"e98c90ff-9a9a-4e40-aeba-556d98be49a2\" (UID: \"e98c90ff-9a9a-4e40-aeba-556d98be49a2\") " Oct 30 17:16:56.774753 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:56.774678 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e98c90ff-9a9a-4e40-aeba-556d98be49a2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e98c90ff-9a9a-4e40-aeba-556d98be49a2-openshift-service-ca\") pod \"e98c90ff-9a9a-4e40-aeba-556d98be49a2\" (UID: \"e98c90ff-9a9a-4e40-aeba-556d98be49a2\") " Oct 30 17:16:56.774802 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:56.774771 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e98c90ff-9a9a-4e40-aeba-556d98be49a2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e98c90ff-9a9a-4e40-aeba-556d98be49a2-maas-api-tls\") pod \"e98c90ff-9a9a-4e40-aeba-556d98be49a2\" (UID: \"e98c90ff-9a9a-4e40-aeba-556d98be49a2\") " Oct 30 17:16:56.795223 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:56.780122 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e98c90ff-9a9a-4e40-aeba-556d98be49a2-maas-api-tls" pod "e98c90ff-9a9a-4e40-aeba-556d98be49a2" (UID: "e98c90ff-9a9a-4e40-aeba-556d98be49a2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:16:56.795223 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:56.785763 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e98c90ff-9a9a-4e40-aeba-556d98be49a2-openshift-service-ca" pod "e98c90ff-9a9a-4e40-aeba-556d98be49a2" (UID: "e98c90ff-9a9a-4e40-aeba-556d98be49a2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:16:56.797359 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:56.796947 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e98c90ff-9a9a-4e40-aeba-556d98be49a2-kube-api-access-ldwzb" pod "e98c90ff-9a9a-4e40-aeba-556d98be49a2" (UID: "e98c90ff-9a9a-4e40-aeba-556d98be49a2"). InnerVolumeSpecName "kube-api-access-ldwzb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:16:56.823419 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:56.821351 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e98c90ff-9a9a-4e40-aeba-556d98be49a2-metrics-certs" pod "e98c90ff-9a9a-4e40-aeba-556d98be49a2" (UID: "e98c90ff-9a9a-4e40-aeba-556d98be49a2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:16:56.876570 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:56.876485 5052 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e98c90ff-9a9a-4e40-aeba-556d98be49a2-openshift-service-ca\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:16:56.876570 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:56.876517 5052 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e98c90ff-9a9a-4e40-aeba-556d98be49a2-maas-api-tls\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:16:56.876570 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:56.876531 5052 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e98c90ff-9a9a-4e40-aeba-556d98be49a2-metrics-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:16:56.876570 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:56.876543 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldwzb\" (UniqueName: \"kubernetes.io/projected/e98c90ff-9a9a-4e40-aeba-556d98be49a2-kube-api-access-ldwzb\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:16:56.894026 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:56.893990 5052 generic.go:356] "Generic (PLEG): container finished" podID="e98c90ff-9a9a-4e40-aeba-556d98be49a2" containerID="9ebcf9cd8aecab35e3019f3833c85e00d7e3c5ad9da1109061d418a396ef186c" exitCode=0 Oct 30 17:16:56.894175 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:56.894041 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-263f00-64bd44f46f-9qwbt" Oct 30 17:16:56.894175 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:56.894081 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-263f00-64bd44f46f-9qwbt" event={"ID":"e98c90ff-9a9a-4e40-aeba-556d98be49a2","Type":"ContainerDied","Data":"9ebcf9cd8aecab35e3019f3833c85e00d7e3c5ad9da1109061d418a396ef186c"} Oct 30 17:16:56.894175 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:56.894138 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-263f00-64bd44f46f-9qwbt" event={"ID":"e98c90ff-9a9a-4e40-aeba-556d98be49a2","Type":"ContainerDied","Data":"ee1618a69c3e552e086beb91f1389e4ea4f30388a1d692dd84d495b7bd1b800a"} Oct 30 17:16:56.894338 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:56.894175 5052 scope.go:122] "RemoveContainer" containerID="9ebcf9cd8aecab35e3019f3833c85e00d7e3c5ad9da1109061d418a396ef186c" Oct 30 17:16:56.904941 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:56.904917 5052 scope.go:122] "RemoveContainer" containerID="9ebcf9cd8aecab35e3019f3833c85e00d7e3c5ad9da1109061d418a396ef186c" Oct 30 17:16:56.905213 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:16:56.905194 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ebcf9cd8aecab35e3019f3833c85e00d7e3c5ad9da1109061d418a396ef186c\": container with ID starting with 9ebcf9cd8aecab35e3019f3833c85e00d7e3c5ad9da1109061d418a396ef186c not found: ID does not exist" containerID="9ebcf9cd8aecab35e3019f3833c85e00d7e3c5ad9da1109061d418a396ef186c" Oct 30 17:16:56.905289 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:56.905224 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ebcf9cd8aecab35e3019f3833c85e00d7e3c5ad9da1109061d418a396ef186c"} err="failed to get container status \"9ebcf9cd8aecab35e3019f3833c85e00d7e3c5ad9da1109061d418a396ef186c\": rpc error: code = NotFound desc = could not find container \"9ebcf9cd8aecab35e3019f3833c85e00d7e3c5ad9da1109061d418a396ef186c\": container with ID starting with 9ebcf9cd8aecab35e3019f3833c85e00d7e3c5ad9da1109061d418a396ef186c not found: ID does not exist" Oct 30 17:16:56.949511 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:56.949472 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-263f00-64bd44f46f-9qwbt"] Oct 30 17:16:56.962340 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:56.962320 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-263f00-64bd44f46f-9qwbt"] Oct 30 17:16:58.607156 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:16:58.607113 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e98c90ff-9a9a-4e40-aeba-556d98be49a2" path="/var/lib/kubelet/pods/e98c90ff-9a9a-4e40-aeba-556d98be49a2/volumes" Oct 30 17:17:11.098242 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:11.098203 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cdwcj"] Oct 30 17:17:15.846786 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:15.846745 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-x5p9m"] Oct 30 17:17:15.847296 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:15.847012 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-x5p9m" podUID="d6c3dee8-d01e-445d-b8a4-96b7f57493ec" containerName="istio-proxy" containerID="cri-o://b6db49b82fee9c70b44841dcc72fbd0811f6ef15f1630ab7b4ea93e4e3d52770" gracePeriod=30 Oct 30 17:17:18.840766 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:17:18.840723 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 17:17:19.752419 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:19.752380 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-a8c22d-7ff9d946b7-xqlzn"] Oct 30 17:17:19.752706 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:19.752675 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a8c22d-7ff9d946b7-xqlzn" podUID="e68b78bc-50ac-43ee-8354-a11b921e2713" containerName="maas-api" containerID="cri-o://ae5569953f52787236943cca4f21042f58288bb70da33906cca7983c628910e1" gracePeriod=30 Oct 30 17:17:20.027792 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:20.027763 5052 generic.go:356] "Generic (PLEG): container finished" podID="e68b78bc-50ac-43ee-8354-a11b921e2713" containerID="ae5569953f52787236943cca4f21042f58288bb70da33906cca7983c628910e1" exitCode=0 Oct 30 17:17:20.028134 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:20.027807 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a8c22d-7ff9d946b7-xqlzn" event={"ID":"e68b78bc-50ac-43ee-8354-a11b921e2713","Type":"ContainerDied","Data":"ae5569953f52787236943cca4f21042f58288bb70da33906cca7983c628910e1"} Oct 30 17:17:20.028134 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:20.027844 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a8c22d-7ff9d946b7-xqlzn" event={"ID":"e68b78bc-50ac-43ee-8354-a11b921e2713","Type":"ContainerDied","Data":"a3e426afe8275ced5dc11eeacf8af02324bb2f87fbd4c33696ee6f8413359041"} Oct 30 17:17:20.028134 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:20.027862 5052 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3e426afe8275ced5dc11eeacf8af02324bb2f87fbd4c33696ee6f8413359041" Oct 30 17:17:20.031649 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:20.031628 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a8c22d-7ff9d946b7-xqlzn" Oct 30 17:17:20.169641 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:20.169603 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e68b78bc-50ac-43ee-8354-a11b921e2713-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e68b78bc-50ac-43ee-8354-a11b921e2713-metrics-certs\") pod \"e68b78bc-50ac-43ee-8354-a11b921e2713\" (UID: \"e68b78bc-50ac-43ee-8354-a11b921e2713\") " Oct 30 17:17:20.169820 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:20.169696 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e68b78bc-50ac-43ee-8354-a11b921e2713-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e68b78bc-50ac-43ee-8354-a11b921e2713-maas-api-tls\") pod \"e68b78bc-50ac-43ee-8354-a11b921e2713\" (UID: \"e68b78bc-50ac-43ee-8354-a11b921e2713\") " Oct 30 17:17:20.169820 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:20.169763 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e68b78bc-50ac-43ee-8354-a11b921e2713-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e68b78bc-50ac-43ee-8354-a11b921e2713-openshift-service-ca\") pod \"e68b78bc-50ac-43ee-8354-a11b921e2713\" (UID: \"e68b78bc-50ac-43ee-8354-a11b921e2713\") " Oct 30 17:17:20.169820 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:20.169812 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e68b78bc-50ac-43ee-8354-a11b921e2713-kube-api-access-dwvvh\" (UniqueName: \"kubernetes.io/projected/e68b78bc-50ac-43ee-8354-a11b921e2713-kube-api-access-dwvvh\") pod \"e68b78bc-50ac-43ee-8354-a11b921e2713\" (UID: \"e68b78bc-50ac-43ee-8354-a11b921e2713\") " Oct 30 17:17:20.170235 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:20.170198 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e68b78bc-50ac-43ee-8354-a11b921e2713-openshift-service-ca" pod "e68b78bc-50ac-43ee-8354-a11b921e2713" (UID: "e68b78bc-50ac-43ee-8354-a11b921e2713"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:17:20.172013 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:20.171973 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e68b78bc-50ac-43ee-8354-a11b921e2713-metrics-certs" pod "e68b78bc-50ac-43ee-8354-a11b921e2713" (UID: "e68b78bc-50ac-43ee-8354-a11b921e2713"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:17:20.172618 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:20.172576 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e68b78bc-50ac-43ee-8354-a11b921e2713-maas-api-tls" pod "e68b78bc-50ac-43ee-8354-a11b921e2713" (UID: "e68b78bc-50ac-43ee-8354-a11b921e2713"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:17:20.172734 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:20.172627 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e68b78bc-50ac-43ee-8354-a11b921e2713-kube-api-access-dwvvh" pod "e68b78bc-50ac-43ee-8354-a11b921e2713" (UID: "e68b78bc-50ac-43ee-8354-a11b921e2713"). InnerVolumeSpecName "kube-api-access-dwvvh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:17:20.272062 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:20.271964 5052 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e68b78bc-50ac-43ee-8354-a11b921e2713-metrics-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:17:20.272062 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:20.272009 5052 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e68b78bc-50ac-43ee-8354-a11b921e2713-maas-api-tls\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:17:20.272062 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:20.272024 5052 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e68b78bc-50ac-43ee-8354-a11b921e2713-openshift-service-ca\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:17:20.272062 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:20.272037 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwvvh\" (UniqueName: \"kubernetes.io/projected/e68b78bc-50ac-43ee-8354-a11b921e2713-kube-api-access-dwvvh\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:17:20.343084 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:20.343039 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-750dcb/distinct-w2-750dcb-kserve-86bbdfcbb-bpn77"] Oct 30 17:17:21.043084 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:21.040543 5052 generic.go:356] "Generic (PLEG): container finished" podID="d6c3dee8-d01e-445d-b8a4-96b7f57493ec" containerID="b6db49b82fee9c70b44841dcc72fbd0811f6ef15f1630ab7b4ea93e4e3d52770" exitCode=0 Oct 30 17:17:21.043084 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:21.040729 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a8c22d-7ff9d946b7-xqlzn" Oct 30 17:17:21.043084 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:21.041316 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-x5p9m" event={"ID":"d6c3dee8-d01e-445d-b8a4-96b7f57493ec","Type":"ContainerDied","Data":"b6db49b82fee9c70b44841dcc72fbd0811f6ef15f1630ab7b4ea93e4e3d52770"} Oct 30 17:17:21.043084 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:21.041410 5052 scope.go:122] "RemoveContainer" containerID="6e801b7a1159f0947dc899e0f8a3597f7d099583f63a7e759acb93e810488d39" Oct 30 17:17:21.110324 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:21.110298 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-x5p9m" Oct 30 17:17:21.112294 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:21.112273 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-a8c22d-7ff9d946b7-xqlzn"] Oct 30 17:17:21.125530 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:21.125468 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-a8c22d-7ff9d946b7-xqlzn"] Oct 30 17:17:21.181324 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:21.181294 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6c3dee8-d01e-445d-b8a4-96b7f57493ec-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6c3dee8-d01e-445d-b8a4-96b7f57493ec-workload-socket\") pod \"d6c3dee8-d01e-445d-b8a4-96b7f57493ec\" (UID: \"d6c3dee8-d01e-445d-b8a4-96b7f57493ec\") " Oct 30 17:17:21.181491 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:21.181382 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6c3dee8-d01e-445d-b8a4-96b7f57493ec-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6c3dee8-d01e-445d-b8a4-96b7f57493ec-istiod-ca-cert\") pod \"d6c3dee8-d01e-445d-b8a4-96b7f57493ec\" (UID: \"d6c3dee8-d01e-445d-b8a4-96b7f57493ec\") " Oct 30 17:17:21.181491 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:21.181469 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6c3dee8-d01e-445d-b8a4-96b7f57493ec-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6c3dee8-d01e-445d-b8a4-96b7f57493ec-workload-certs\") pod \"d6c3dee8-d01e-445d-b8a4-96b7f57493ec\" (UID: \"d6c3dee8-d01e-445d-b8a4-96b7f57493ec\") " Oct 30 17:17:21.181625 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:21.181511 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6c3dee8-d01e-445d-b8a4-96b7f57493ec-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6c3dee8-d01e-445d-b8a4-96b7f57493ec-istio-data\") pod \"d6c3dee8-d01e-445d-b8a4-96b7f57493ec\" (UID: \"d6c3dee8-d01e-445d-b8a4-96b7f57493ec\") " Oct 30 17:17:21.181625 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:21.181565 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6c3dee8-d01e-445d-b8a4-96b7f57493ec-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6c3dee8-d01e-445d-b8a4-96b7f57493ec-credential-socket\") pod \"d6c3dee8-d01e-445d-b8a4-96b7f57493ec\" (UID: \"d6c3dee8-d01e-445d-b8a4-96b7f57493ec\") " Oct 30 17:17:21.181759 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:21.181632 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6c3dee8-d01e-445d-b8a4-96b7f57493ec-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6c3dee8-d01e-445d-b8a4-96b7f57493ec-istio-envoy\") pod \"d6c3dee8-d01e-445d-b8a4-96b7f57493ec\" (UID: \"d6c3dee8-d01e-445d-b8a4-96b7f57493ec\") " Oct 30 17:17:21.181759 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:21.181632 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6c3dee8-d01e-445d-b8a4-96b7f57493ec-workload-socket" pod "d6c3dee8-d01e-445d-b8a4-96b7f57493ec" (UID: "d6c3dee8-d01e-445d-b8a4-96b7f57493ec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:17:21.181759 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:21.181675 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d6c3dee8-d01e-445d-b8a4-96b7f57493ec-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6c3dee8-d01e-445d-b8a4-96b7f57493ec-istio-podinfo\") pod \"d6c3dee8-d01e-445d-b8a4-96b7f57493ec\" (UID: \"d6c3dee8-d01e-445d-b8a4-96b7f57493ec\") " Oct 30 17:17:21.181759 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:21.181733 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6c3dee8-d01e-445d-b8a4-96b7f57493ec-kube-api-access-h9gxx\" (UniqueName: \"kubernetes.io/projected/d6c3dee8-d01e-445d-b8a4-96b7f57493ec-kube-api-access-h9gxx\") pod \"d6c3dee8-d01e-445d-b8a4-96b7f57493ec\" (UID: \"d6c3dee8-d01e-445d-b8a4-96b7f57493ec\") " Oct 30 17:17:21.181989 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:21.181783 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6c3dee8-d01e-445d-b8a4-96b7f57493ec-istio-token\" (UniqueName: \"kubernetes.io/projected/d6c3dee8-d01e-445d-b8a4-96b7f57493ec-istio-token\") pod \"d6c3dee8-d01e-445d-b8a4-96b7f57493ec\" (UID: \"d6c3dee8-d01e-445d-b8a4-96b7f57493ec\") " Oct 30 17:17:21.181989 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:21.181861 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6c3dee8-d01e-445d-b8a4-96b7f57493ec-workload-certs" pod "d6c3dee8-d01e-445d-b8a4-96b7f57493ec" (UID: "d6c3dee8-d01e-445d-b8a4-96b7f57493ec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:17:21.181989 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:21.181926 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6c3dee8-d01e-445d-b8a4-96b7f57493ec-istiod-ca-cert" pod "d6c3dee8-d01e-445d-b8a4-96b7f57493ec" (UID: "d6c3dee8-d01e-445d-b8a4-96b7f57493ec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:17:21.182200 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:21.182172 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6c3dee8-d01e-445d-b8a4-96b7f57493ec-istio-data" pod "d6c3dee8-d01e-445d-b8a4-96b7f57493ec" (UID: "d6c3dee8-d01e-445d-b8a4-96b7f57493ec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:17:21.182282 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:21.182206 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6c3dee8-d01e-445d-b8a4-96b7f57493ec-credential-socket" pod "d6c3dee8-d01e-445d-b8a4-96b7f57493ec" (UID: "d6c3dee8-d01e-445d-b8a4-96b7f57493ec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:17:21.182282 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:21.182263 5052 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6c3dee8-d01e-445d-b8a4-96b7f57493ec-workload-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:17:21.182399 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:21.182286 5052 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6c3dee8-d01e-445d-b8a4-96b7f57493ec-workload-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:17:21.182399 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:21.182302 5052 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6c3dee8-d01e-445d-b8a4-96b7f57493ec-istiod-ca-cert\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:17:21.184508 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:21.184453 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6c3dee8-d01e-445d-b8a4-96b7f57493ec-istio-envoy" pod "d6c3dee8-d01e-445d-b8a4-96b7f57493ec" (UID: "d6c3dee8-d01e-445d-b8a4-96b7f57493ec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:17:21.184666 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:21.184569 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d6c3dee8-d01e-445d-b8a4-96b7f57493ec-istio-podinfo" pod "d6c3dee8-d01e-445d-b8a4-96b7f57493ec" (UID: "d6c3dee8-d01e-445d-b8a4-96b7f57493ec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:17:21.184666 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:21.184600 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6c3dee8-d01e-445d-b8a4-96b7f57493ec-kube-api-access-h9gxx" pod "d6c3dee8-d01e-445d-b8a4-96b7f57493ec" (UID: "d6c3dee8-d01e-445d-b8a4-96b7f57493ec"). InnerVolumeSpecName "kube-api-access-h9gxx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:17:21.184946 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:21.184917 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6c3dee8-d01e-445d-b8a4-96b7f57493ec-istio-token" pod "d6c3dee8-d01e-445d-b8a4-96b7f57493ec" (UID: "d6c3dee8-d01e-445d-b8a4-96b7f57493ec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:17:21.283512 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:21.283476 5052 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6c3dee8-d01e-445d-b8a4-96b7f57493ec-istio-data\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:17:21.283512 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:21.283508 5052 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6c3dee8-d01e-445d-b8a4-96b7f57493ec-credential-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:17:21.283512 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:21.283521 5052 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6c3dee8-d01e-445d-b8a4-96b7f57493ec-istio-envoy\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:17:21.283512 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:21.283529 5052 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6c3dee8-d01e-445d-b8a4-96b7f57493ec-istio-podinfo\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:17:21.283819 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:21.283537 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9gxx\" (UniqueName: \"kubernetes.io/projected/d6c3dee8-d01e-445d-b8a4-96b7f57493ec-kube-api-access-h9gxx\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:17:21.283819 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:21.283546 5052 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6c3dee8-d01e-445d-b8a4-96b7f57493ec-istio-token\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:17:22.047646 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:22.047608 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-x5p9m" event={"ID":"d6c3dee8-d01e-445d-b8a4-96b7f57493ec","Type":"ContainerDied","Data":"cb5d5641f049456fb180ce2ebcd6cc9c9e03816356f63c9a0b3b2ae6dd4501a9"} Oct 30 17:17:22.047646 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:22.047633 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-x5p9m" Oct 30 17:17:22.048149 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:22.047677 5052 scope.go:122] "RemoveContainer" containerID="b6db49b82fee9c70b44841dcc72fbd0811f6ef15f1630ab7b4ea93e4e3d52770" Oct 30 17:17:22.124361 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:22.124336 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-x5p9m"] Oct 30 17:17:22.138003 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:22.137980 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-x5p9m"] Oct 30 17:17:22.597912 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:22.597879 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6c3dee8-d01e-445d-b8a4-96b7f57493ec" path="/var/lib/kubelet/pods/d6c3dee8-d01e-445d-b8a4-96b7f57493ec/volumes" Oct 30 17:17:22.598314 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:22.598297 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e68b78bc-50ac-43ee-8354-a11b921e2713" path="/var/lib/kubelet/pods/e68b78bc-50ac-43ee-8354-a11b921e2713/volumes" Oct 30 17:17:24.329519 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:24.329040 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cdwcj"] Oct 30 17:17:26.659137 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:26.659101 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-9e533f-57fd8bd864-t6vr2"] Oct 30 17:17:26.690369 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:26.690342 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9e533f-85c9dd5979-nrmbf"] Oct 30 17:17:28.559629 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:28.559563 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-bvwpz"] Oct 30 17:17:28.560137 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:28.559815 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-bvwpz" podUID="e8eabbfd-4dce-4087-a350-c55cb7c7baf3" containerName="istio-proxy" containerID="cri-o://e6f611504d91d2a3ed7e8c61e2d6eec3166562879ead1ab42e6e256e155349a9" gracePeriod=30 Oct 30 17:17:28.902066 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:28.902038 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-h5v86"] Oct 30 17:17:28.902314 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:28.902240 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-h5v86" podUID="029cd866-9c51-45c8-9ca4-4c16578e12c9" containerName="istio-proxy" containerID="cri-o://73c8b70b6d202f94562b22334a53f566f4884cd9d971ef55d243aed2ffc55b36" gracePeriod=30 Oct 30 17:17:31.691230 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:31.691165 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9e533f-85c9dd5979-nrmbf" podUID="8054915f-a3d1-44d0-ab47-4ca0161255da" containerName="payload-pre-processing" containerID="cri-o://9c88eb46a20297d4d538b40dbd7a013fa2d48b920ea4aa9562ce897a8daa31c4" gracePeriod=25 Oct 30 17:17:32.078348 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:32.078251 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-9e533f-57fd8bd864-t6vr2" podUID="4963d7c3-cee3-430c-8e1c-156c79a75a3d" containerName="payload-processing" containerID="cri-o://3e52bc09f7330f22593ced39b6dad44b40e6f6b9adb426971337f4627e150ffa" gracePeriod=25 Oct 30 17:17:33.831082 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:33.831021 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-bvwpz" Oct 30 17:17:33.919491 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:33.919455 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8eabbfd-4dce-4087-a350-c55cb7c7baf3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8eabbfd-4dce-4087-a350-c55cb7c7baf3-workload-socket\") pod \"e8eabbfd-4dce-4087-a350-c55cb7c7baf3\" (UID: \"e8eabbfd-4dce-4087-a350-c55cb7c7baf3\") " Oct 30 17:17:33.919689 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:33.919562 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8eabbfd-4dce-4087-a350-c55cb7c7baf3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8eabbfd-4dce-4087-a350-c55cb7c7baf3-workload-certs\") pod \"e8eabbfd-4dce-4087-a350-c55cb7c7baf3\" (UID: \"e8eabbfd-4dce-4087-a350-c55cb7c7baf3\") " Oct 30 17:17:33.919689 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:33.919666 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e8eabbfd-4dce-4087-a350-c55cb7c7baf3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8eabbfd-4dce-4087-a350-c55cb7c7baf3-istio-podinfo\") pod \"e8eabbfd-4dce-4087-a350-c55cb7c7baf3\" (UID: \"e8eabbfd-4dce-4087-a350-c55cb7c7baf3\") " Oct 30 17:17:33.919848 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:33.919701 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8eabbfd-4dce-4087-a350-c55cb7c7baf3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8eabbfd-4dce-4087-a350-c55cb7c7baf3-istio-data\") pod \"e8eabbfd-4dce-4087-a350-c55cb7c7baf3\" (UID: \"e8eabbfd-4dce-4087-a350-c55cb7c7baf3\") " Oct 30 17:17:33.919848 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:33.919735 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8eabbfd-4dce-4087-a350-c55cb7c7baf3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8eabbfd-4dce-4087-a350-c55cb7c7baf3-istio-envoy\") pod \"e8eabbfd-4dce-4087-a350-c55cb7c7baf3\" (UID: \"e8eabbfd-4dce-4087-a350-c55cb7c7baf3\") " Oct 30 17:17:33.921118 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:33.920136 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8eabbfd-4dce-4087-a350-c55cb7c7baf3-istio-token\" (UniqueName: \"kubernetes.io/projected/e8eabbfd-4dce-4087-a350-c55cb7c7baf3-istio-token\") pod \"e8eabbfd-4dce-4087-a350-c55cb7c7baf3\" (UID: \"e8eabbfd-4dce-4087-a350-c55cb7c7baf3\") " Oct 30 17:17:33.921118 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:33.920139 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8eabbfd-4dce-4087-a350-c55cb7c7baf3-workload-socket" pod "e8eabbfd-4dce-4087-a350-c55cb7c7baf3" (UID: "e8eabbfd-4dce-4087-a350-c55cb7c7baf3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:17:33.921118 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:33.920255 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8eabbfd-4dce-4087-a350-c55cb7c7baf3-kube-api-access-j664n\" (UniqueName: \"kubernetes.io/projected/e8eabbfd-4dce-4087-a350-c55cb7c7baf3-kube-api-access-j664n\") pod \"e8eabbfd-4dce-4087-a350-c55cb7c7baf3\" (UID: \"e8eabbfd-4dce-4087-a350-c55cb7c7baf3\") " Oct 30 17:17:33.921118 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:33.920172 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8eabbfd-4dce-4087-a350-c55cb7c7baf3-workload-certs" pod "e8eabbfd-4dce-4087-a350-c55cb7c7baf3" (UID: "e8eabbfd-4dce-4087-a350-c55cb7c7baf3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:17:33.921118 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:33.920233 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8eabbfd-4dce-4087-a350-c55cb7c7baf3-istio-data" pod "e8eabbfd-4dce-4087-a350-c55cb7c7baf3" (UID: "e8eabbfd-4dce-4087-a350-c55cb7c7baf3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:17:33.921118 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:33.920322 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8eabbfd-4dce-4087-a350-c55cb7c7baf3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8eabbfd-4dce-4087-a350-c55cb7c7baf3-istiod-ca-cert\") pod \"e8eabbfd-4dce-4087-a350-c55cb7c7baf3\" (UID: \"e8eabbfd-4dce-4087-a350-c55cb7c7baf3\") " Oct 30 17:17:33.921118 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:33.920352 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8eabbfd-4dce-4087-a350-c55cb7c7baf3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8eabbfd-4dce-4087-a350-c55cb7c7baf3-credential-socket\") pod \"e8eabbfd-4dce-4087-a350-c55cb7c7baf3\" (UID: \"e8eabbfd-4dce-4087-a350-c55cb7c7baf3\") " Oct 30 17:17:33.921118 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:33.920985 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8eabbfd-4dce-4087-a350-c55cb7c7baf3-istiod-ca-cert" pod "e8eabbfd-4dce-4087-a350-c55cb7c7baf3" (UID: "e8eabbfd-4dce-4087-a350-c55cb7c7baf3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:17:33.921118 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:33.921081 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8eabbfd-4dce-4087-a350-c55cb7c7baf3-credential-socket" pod "e8eabbfd-4dce-4087-a350-c55cb7c7baf3" (UID: "e8eabbfd-4dce-4087-a350-c55cb7c7baf3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:17:33.922026 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:33.921927 5052 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8eabbfd-4dce-4087-a350-c55cb7c7baf3-istio-data\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:17:33.922026 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:33.921957 5052 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8eabbfd-4dce-4087-a350-c55cb7c7baf3-istiod-ca-cert\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:17:33.922026 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:33.921971 5052 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8eabbfd-4dce-4087-a350-c55cb7c7baf3-credential-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:17:33.922026 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:33.921985 5052 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8eabbfd-4dce-4087-a350-c55cb7c7baf3-workload-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:17:33.922026 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:33.922001 5052 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8eabbfd-4dce-4087-a350-c55cb7c7baf3-workload-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:17:33.922394 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:33.922374 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e8eabbfd-4dce-4087-a350-c55cb7c7baf3-istio-podinfo" pod "e8eabbfd-4dce-4087-a350-c55cb7c7baf3" (UID: "e8eabbfd-4dce-4087-a350-c55cb7c7baf3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:17:33.922394 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:33.922383 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8eabbfd-4dce-4087-a350-c55cb7c7baf3-istio-envoy" pod "e8eabbfd-4dce-4087-a350-c55cb7c7baf3" (UID: "e8eabbfd-4dce-4087-a350-c55cb7c7baf3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:17:33.922673 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:33.922655 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8eabbfd-4dce-4087-a350-c55cb7c7baf3-istio-token" pod "e8eabbfd-4dce-4087-a350-c55cb7c7baf3" (UID: "e8eabbfd-4dce-4087-a350-c55cb7c7baf3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:17:33.923414 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:33.923387 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8eabbfd-4dce-4087-a350-c55cb7c7baf3-kube-api-access-j664n" pod "e8eabbfd-4dce-4087-a350-c55cb7c7baf3" (UID: "e8eabbfd-4dce-4087-a350-c55cb7c7baf3"). InnerVolumeSpecName "kube-api-access-j664n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:17:34.023351 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:34.023330 5052 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8eabbfd-4dce-4087-a350-c55cb7c7baf3-istio-podinfo\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:17:34.023428 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:34.023357 5052 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8eabbfd-4dce-4087-a350-c55cb7c7baf3-istio-envoy\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:17:34.023428 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:34.023367 5052 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8eabbfd-4dce-4087-a350-c55cb7c7baf3-istio-token\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:17:34.023428 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:34.023377 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j664n\" (UniqueName: \"kubernetes.io/projected/e8eabbfd-4dce-4087-a350-c55cb7c7baf3-kube-api-access-j664n\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:17:34.121546 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:34.121440 5052 generic.go:356] "Generic (PLEG): container finished" podID="e8eabbfd-4dce-4087-a350-c55cb7c7baf3" containerID="e6f611504d91d2a3ed7e8c61e2d6eec3166562879ead1ab42e6e256e155349a9" exitCode=0 Oct 30 17:17:34.121546 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:34.121500 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-bvwpz" Oct 30 17:17:34.121546 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:34.121526 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-bvwpz" event={"ID":"e8eabbfd-4dce-4087-a350-c55cb7c7baf3","Type":"ContainerDied","Data":"e6f611504d91d2a3ed7e8c61e2d6eec3166562879ead1ab42e6e256e155349a9"} Oct 30 17:17:34.121910 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:34.121615 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-bvwpz" event={"ID":"e8eabbfd-4dce-4087-a350-c55cb7c7baf3","Type":"ContainerDied","Data":"c1abf298a0705a2443c0b155474860cdb6a41c74c8b8a0955fa9b6d6108266bf"} Oct 30 17:17:34.121910 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:34.121654 5052 scope.go:122] "RemoveContainer" containerID="e6f611504d91d2a3ed7e8c61e2d6eec3166562879ead1ab42e6e256e155349a9" Oct 30 17:17:34.129478 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:34.129450 5052 generic.go:356] "Generic (PLEG): container finished" podID="029cd866-9c51-45c8-9ca4-4c16578e12c9" containerID="73c8b70b6d202f94562b22334a53f566f4884cd9d971ef55d243aed2ffc55b36" exitCode=0 Oct 30 17:17:34.129666 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:34.129506 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-h5v86" event={"ID":"029cd866-9c51-45c8-9ca4-4c16578e12c9","Type":"ContainerDied","Data":"73c8b70b6d202f94562b22334a53f566f4884cd9d971ef55d243aed2ffc55b36"} Oct 30 17:17:34.140601 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:34.140512 5052 scope.go:122] "RemoveContainer" containerID="e6f611504d91d2a3ed7e8c61e2d6eec3166562879ead1ab42e6e256e155349a9" Oct 30 17:17:34.140860 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:17:34.140837 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6f611504d91d2a3ed7e8c61e2d6eec3166562879ead1ab42e6e256e155349a9\": container with ID starting with e6f611504d91d2a3ed7e8c61e2d6eec3166562879ead1ab42e6e256e155349a9 not found: ID does not exist" containerID="e6f611504d91d2a3ed7e8c61e2d6eec3166562879ead1ab42e6e256e155349a9" Oct 30 17:17:34.140973 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:34.140871 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6f611504d91d2a3ed7e8c61e2d6eec3166562879ead1ab42e6e256e155349a9"} err="failed to get container status \"e6f611504d91d2a3ed7e8c61e2d6eec3166562879ead1ab42e6e256e155349a9\": rpc error: code = NotFound desc = could not find container \"e6f611504d91d2a3ed7e8c61e2d6eec3166562879ead1ab42e6e256e155349a9\": container with ID starting with e6f611504d91d2a3ed7e8c61e2d6eec3166562879ead1ab42e6e256e155349a9 not found: ID does not exist" Oct 30 17:17:34.140973 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:34.140894 5052 scope.go:122] "RemoveContainer" containerID="e3c6f7aeea864977164007842e03e5ab7d8c32ea1a7105943fd2a500d7d7fafd" Oct 30 17:17:34.151216 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:34.151195 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-h5v86" Oct 30 17:17:34.177896 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:34.177870 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-bvwpz"] Oct 30 17:17:34.191359 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:34.191336 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-bvwpz"] Oct 30 17:17:34.329700 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:34.329667 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/029cd866-9c51-45c8-9ca4-4c16578e12c9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/029cd866-9c51-45c8-9ca4-4c16578e12c9-workload-certs\") pod \"029cd866-9c51-45c8-9ca4-4c16578e12c9\" (UID: \"029cd866-9c51-45c8-9ca4-4c16578e12c9\") " Oct 30 17:17:34.329883 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:34.329777 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/029cd866-9c51-45c8-9ca4-4c16578e12c9-istio-token\" (UniqueName: \"kubernetes.io/projected/029cd866-9c51-45c8-9ca4-4c16578e12c9-istio-token\") pod \"029cd866-9c51-45c8-9ca4-4c16578e12c9\" (UID: \"029cd866-9c51-45c8-9ca4-4c16578e12c9\") " Oct 30 17:17:34.329883 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:34.329818 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/029cd866-9c51-45c8-9ca4-4c16578e12c9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/029cd866-9c51-45c8-9ca4-4c16578e12c9-istio-envoy\") pod \"029cd866-9c51-45c8-9ca4-4c16578e12c9\" (UID: \"029cd866-9c51-45c8-9ca4-4c16578e12c9\") " Oct 30 17:17:34.329883 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:34.329841 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/029cd866-9c51-45c8-9ca4-4c16578e12c9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/029cd866-9c51-45c8-9ca4-4c16578e12c9-workload-socket\") pod \"029cd866-9c51-45c8-9ca4-4c16578e12c9\" (UID: \"029cd866-9c51-45c8-9ca4-4c16578e12c9\") " Oct 30 17:17:34.329883 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:34.329876 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/029cd866-9c51-45c8-9ca4-4c16578e12c9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/029cd866-9c51-45c8-9ca4-4c16578e12c9-istio-podinfo\") pod \"029cd866-9c51-45c8-9ca4-4c16578e12c9\" (UID: \"029cd866-9c51-45c8-9ca4-4c16578e12c9\") " Oct 30 17:17:34.330132 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:34.329908 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/029cd866-9c51-45c8-9ca4-4c16578e12c9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/029cd866-9c51-45c8-9ca4-4c16578e12c9-credential-socket\") pod \"029cd866-9c51-45c8-9ca4-4c16578e12c9\" (UID: \"029cd866-9c51-45c8-9ca4-4c16578e12c9\") " Oct 30 17:17:34.330132 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:34.329930 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/029cd866-9c51-45c8-9ca4-4c16578e12c9-kube-api-access-56bhv\" (UniqueName: \"kubernetes.io/projected/029cd866-9c51-45c8-9ca4-4c16578e12c9-kube-api-access-56bhv\") pod \"029cd866-9c51-45c8-9ca4-4c16578e12c9\" (UID: \"029cd866-9c51-45c8-9ca4-4c16578e12c9\") " Oct 30 17:17:34.330132 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:34.329949 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/029cd866-9c51-45c8-9ca4-4c16578e12c9-workload-certs" pod "029cd866-9c51-45c8-9ca4-4c16578e12c9" (UID: "029cd866-9c51-45c8-9ca4-4c16578e12c9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:17:34.330132 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:34.329957 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/029cd866-9c51-45c8-9ca4-4c16578e12c9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/029cd866-9c51-45c8-9ca4-4c16578e12c9-istiod-ca-cert\") pod \"029cd866-9c51-45c8-9ca4-4c16578e12c9\" (UID: \"029cd866-9c51-45c8-9ca4-4c16578e12c9\") " Oct 30 17:17:34.330132 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:34.330024 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/029cd866-9c51-45c8-9ca4-4c16578e12c9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/029cd866-9c51-45c8-9ca4-4c16578e12c9-istio-data\") pod \"029cd866-9c51-45c8-9ca4-4c16578e12c9\" (UID: \"029cd866-9c51-45c8-9ca4-4c16578e12c9\") " Oct 30 17:17:34.330423 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:34.330141 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/029cd866-9c51-45c8-9ca4-4c16578e12c9-workload-socket" pod "029cd866-9c51-45c8-9ca4-4c16578e12c9" (UID: "029cd866-9c51-45c8-9ca4-4c16578e12c9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:17:34.330423 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:34.330311 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/029cd866-9c51-45c8-9ca4-4c16578e12c9-istiod-ca-cert" pod "029cd866-9c51-45c8-9ca4-4c16578e12c9" (UID: "029cd866-9c51-45c8-9ca4-4c16578e12c9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:17:34.330547 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:34.330446 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/029cd866-9c51-45c8-9ca4-4c16578e12c9-istio-data" pod "029cd866-9c51-45c8-9ca4-4c16578e12c9" (UID: "029cd866-9c51-45c8-9ca4-4c16578e12c9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:17:34.330547 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:34.330465 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/029cd866-9c51-45c8-9ca4-4c16578e12c9-credential-socket" pod "029cd866-9c51-45c8-9ca4-4c16578e12c9" (UID: "029cd866-9c51-45c8-9ca4-4c16578e12c9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:17:34.330695 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:34.330628 5052 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/029cd866-9c51-45c8-9ca4-4c16578e12c9-credential-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:17:34.330695 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:34.330645 5052 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/029cd866-9c51-45c8-9ca4-4c16578e12c9-istiod-ca-cert\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:17:34.330695 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:34.330654 5052 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/029cd866-9c51-45c8-9ca4-4c16578e12c9-istio-data\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:17:34.330695 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:34.330662 5052 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/029cd866-9c51-45c8-9ca4-4c16578e12c9-workload-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:17:34.330695 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:34.330670 5052 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/029cd866-9c51-45c8-9ca4-4c16578e12c9-workload-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:17:34.332342 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:34.332314 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/029cd866-9c51-45c8-9ca4-4c16578e12c9-istio-token" pod "029cd866-9c51-45c8-9ca4-4c16578e12c9" (UID: "029cd866-9c51-45c8-9ca4-4c16578e12c9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:17:34.332466 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:34.332450 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/029cd866-9c51-45c8-9ca4-4c16578e12c9-istio-podinfo" pod "029cd866-9c51-45c8-9ca4-4c16578e12c9" (UID: "029cd866-9c51-45c8-9ca4-4c16578e12c9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:17:34.332466 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:34.332457 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/029cd866-9c51-45c8-9ca4-4c16578e12c9-istio-envoy" pod "029cd866-9c51-45c8-9ca4-4c16578e12c9" (UID: "029cd866-9c51-45c8-9ca4-4c16578e12c9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:17:34.332687 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:34.332672 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/029cd866-9c51-45c8-9ca4-4c16578e12c9-kube-api-access-56bhv" pod "029cd866-9c51-45c8-9ca4-4c16578e12c9" (UID: "029cd866-9c51-45c8-9ca4-4c16578e12c9"). InnerVolumeSpecName "kube-api-access-56bhv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:17:34.341412 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:34.341383 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9e533f-85c9dd5979-nrmbf" podUID="8054915f-a3d1-44d0-ab47-4ca0161255da" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:17:34.431834 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:34.431799 5052 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/029cd866-9c51-45c8-9ca4-4c16578e12c9-istio-podinfo\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:17:34.431834 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:34.431832 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56bhv\" (UniqueName: \"kubernetes.io/projected/029cd866-9c51-45c8-9ca4-4c16578e12c9-kube-api-access-56bhv\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:17:34.432038 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:34.431846 5052 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/029cd866-9c51-45c8-9ca4-4c16578e12c9-istio-token\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:17:34.432038 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:34.431858 5052 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/029cd866-9c51-45c8-9ca4-4c16578e12c9-istio-envoy\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:17:34.598719 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:34.598687 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8eabbfd-4dce-4087-a350-c55cb7c7baf3" path="/var/lib/kubelet/pods/e8eabbfd-4dce-4087-a350-c55cb7c7baf3/volumes" Oct 30 17:17:35.143433 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:35.143389 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-h5v86" event={"ID":"029cd866-9c51-45c8-9ca4-4c16578e12c9","Type":"ContainerDied","Data":"4bb7056c10742d4e8e78844d1a85d29ec079404f04bccc5ee33ce7fbd0f429d0"} Oct 30 17:17:35.143979 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:35.143466 5052 scope.go:122] "RemoveContainer" containerID="73c8b70b6d202f94562b22334a53f566f4884cd9d971ef55d243aed2ffc55b36" Oct 30 17:17:35.143979 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:35.143412 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-h5v86" Oct 30 17:17:35.230522 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:35.230483 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-h5v86"] Oct 30 17:17:35.237056 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:35.237028 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-h5v86"] Oct 30 17:17:35.349263 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:35.349223 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-9e533f-57fd8bd864-t6vr2" podUID="4963d7c3-cee3-430c-8e1c-156c79a75a3d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:17:35.364957 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:35.364927 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-750dcb/distinct-w2-750dcb-kserve-86bbdfcbb-bpn77" podUID="c45190af-d795-4e13-8f2f-670b2c597658" containerName="main" containerID="cri-o://947e5ececd27de1adc121ca653afa4e886b8756c0317abac23b33b2f748435c6" gracePeriod=45 Oct 30 17:17:35.624551 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:35.624526 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-750dcb/distinct-w2-750dcb-kserve-86bbdfcbb-bpn77" Oct 30 17:17:35.745862 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:35.745777 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c45190af-d795-4e13-8f2f-670b2c597658-tls-certs\" (UniqueName: \"kubernetes.io/secret/c45190af-d795-4e13-8f2f-670b2c597658-tls-certs\") pod \"c45190af-d795-4e13-8f2f-670b2c597658\" (UID: \"c45190af-d795-4e13-8f2f-670b2c597658\") " Oct 30 17:17:35.745862 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:35.745846 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c45190af-d795-4e13-8f2f-670b2c597658-home\" (UniqueName: \"kubernetes.io/empty-dir/c45190af-d795-4e13-8f2f-670b2c597658-home\") pod \"c45190af-d795-4e13-8f2f-670b2c597658\" (UID: \"c45190af-d795-4e13-8f2f-670b2c597658\") " Oct 30 17:17:35.746109 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:35.745885 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c45190af-d795-4e13-8f2f-670b2c597658-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c45190af-d795-4e13-8f2f-670b2c597658-model-cache\") pod \"c45190af-d795-4e13-8f2f-670b2c597658\" (UID: \"c45190af-d795-4e13-8f2f-670b2c597658\") " Oct 30 17:17:35.746109 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:35.745935 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c45190af-d795-4e13-8f2f-670b2c597658-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c45190af-d795-4e13-8f2f-670b2c597658-tmp-dir\") pod \"c45190af-d795-4e13-8f2f-670b2c597658\" (UID: \"c45190af-d795-4e13-8f2f-670b2c597658\") " Oct 30 17:17:35.746109 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:35.746046 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c45190af-d795-4e13-8f2f-670b2c597658-kube-api-access-rnl2p\" (UniqueName: \"kubernetes.io/projected/c45190af-d795-4e13-8f2f-670b2c597658-kube-api-access-rnl2p\") pod \"c45190af-d795-4e13-8f2f-670b2c597658\" (UID: \"c45190af-d795-4e13-8f2f-670b2c597658\") " Oct 30 17:17:35.746303 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:35.746113 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c45190af-d795-4e13-8f2f-670b2c597658-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c45190af-d795-4e13-8f2f-670b2c597658-dshm\") pod \"c45190af-d795-4e13-8f2f-670b2c597658\" (UID: \"c45190af-d795-4e13-8f2f-670b2c597658\") " Oct 30 17:17:35.746403 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:35.746362 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c45190af-d795-4e13-8f2f-670b2c597658-model-cache" pod "c45190af-d795-4e13-8f2f-670b2c597658" (UID: "c45190af-d795-4e13-8f2f-670b2c597658"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:17:35.747288 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:35.746764 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c45190af-d795-4e13-8f2f-670b2c597658-tmp-dir" pod "c45190af-d795-4e13-8f2f-670b2c597658" (UID: "c45190af-d795-4e13-8f2f-670b2c597658"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:17:35.747288 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:35.747102 5052 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c45190af-d795-4e13-8f2f-670b2c597658-model-cache\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:17:35.747288 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:35.747122 5052 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c45190af-d795-4e13-8f2f-670b2c597658-tmp-dir\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:17:35.747288 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:35.747257 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c45190af-d795-4e13-8f2f-670b2c597658-home" pod "c45190af-d795-4e13-8f2f-670b2c597658" (UID: "c45190af-d795-4e13-8f2f-670b2c597658"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:17:35.748426 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:35.748403 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c45190af-d795-4e13-8f2f-670b2c597658-tls-certs" pod "c45190af-d795-4e13-8f2f-670b2c597658" (UID: "c45190af-d795-4e13-8f2f-670b2c597658"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:17:35.749895 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:35.749870 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c45190af-d795-4e13-8f2f-670b2c597658-dshm" pod "c45190af-d795-4e13-8f2f-670b2c597658" (UID: "c45190af-d795-4e13-8f2f-670b2c597658"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:17:35.750047 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:35.750028 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c45190af-d795-4e13-8f2f-670b2c597658-kube-api-access-rnl2p" pod "c45190af-d795-4e13-8f2f-670b2c597658" (UID: "c45190af-d795-4e13-8f2f-670b2c597658"). InnerVolumeSpecName "kube-api-access-rnl2p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:17:35.848514 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:35.848486 5052 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c45190af-d795-4e13-8f2f-670b2c597658-tls-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:17:35.848514 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:35.848512 5052 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c45190af-d795-4e13-8f2f-670b2c597658-home\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:17:35.848514 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:35.848523 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnl2p\" (UniqueName: \"kubernetes.io/projected/c45190af-d795-4e13-8f2f-670b2c597658-kube-api-access-rnl2p\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:17:35.848784 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:35.848532 5052 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c45190af-d795-4e13-8f2f-670b2c597658-dshm\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:17:36.150138 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:36.150105 5052 generic.go:356] "Generic (PLEG): container finished" podID="c45190af-d795-4e13-8f2f-670b2c597658" containerID="947e5ececd27de1adc121ca653afa4e886b8756c0317abac23b33b2f748435c6" exitCode=0 Oct 30 17:17:36.150647 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:36.150162 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-750dcb/distinct-w2-750dcb-kserve-86bbdfcbb-bpn77" Oct 30 17:17:36.150647 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:36.150184 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-750dcb/distinct-w2-750dcb-kserve-86bbdfcbb-bpn77" event={"ID":"c45190af-d795-4e13-8f2f-670b2c597658","Type":"ContainerDied","Data":"947e5ececd27de1adc121ca653afa4e886b8756c0317abac23b33b2f748435c6"} Oct 30 17:17:36.150647 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:36.150225 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-750dcb/distinct-w2-750dcb-kserve-86bbdfcbb-bpn77" event={"ID":"c45190af-d795-4e13-8f2f-670b2c597658","Type":"ContainerDied","Data":"7a10e73f1b0dd38922611c4e7282b93b247bb01afef8b735c63af8e18536dc43"} Oct 30 17:17:36.150647 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:36.150242 5052 scope.go:122] "RemoveContainer" containerID="947e5ececd27de1adc121ca653afa4e886b8756c0317abac23b33b2f748435c6" Oct 30 17:17:36.166099 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:36.166077 5052 scope.go:122] "RemoveContainer" containerID="947e5ececd27de1adc121ca653afa4e886b8756c0317abac23b33b2f748435c6" Oct 30 17:17:36.166792 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:17:36.166764 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"947e5ececd27de1adc121ca653afa4e886b8756c0317abac23b33b2f748435c6\": container with ID starting with 947e5ececd27de1adc121ca653afa4e886b8756c0317abac23b33b2f748435c6 not found: ID does not exist" containerID="947e5ececd27de1adc121ca653afa4e886b8756c0317abac23b33b2f748435c6" Oct 30 17:17:36.166884 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:36.166800 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"947e5ececd27de1adc121ca653afa4e886b8756c0317abac23b33b2f748435c6"} err="failed to get container status \"947e5ececd27de1adc121ca653afa4e886b8756c0317abac23b33b2f748435c6\": rpc error: code = NotFound desc = could not find container \"947e5ececd27de1adc121ca653afa4e886b8756c0317abac23b33b2f748435c6\": container with ID starting with 947e5ececd27de1adc121ca653afa4e886b8756c0317abac23b33b2f748435c6 not found: ID does not exist" Oct 30 17:17:36.207629 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:36.207597 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-750dcb/distinct-w2-750dcb-kserve-86bbdfcbb-bpn77"] Oct 30 17:17:36.221074 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:36.221044 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-750dcb/distinct-w2-750dcb-kserve-86bbdfcbb-bpn77"] Oct 30 17:17:36.597829 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:36.597751 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="029cd866-9c51-45c8-9ca4-4c16578e12c9" path="/var/lib/kubelet/pods/029cd866-9c51-45c8-9ca4-4c16578e12c9/volumes" Oct 30 17:17:36.598215 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:36.598202 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c45190af-d795-4e13-8f2f-670b2c597658" path="/var/lib/kubelet/pods/c45190af-d795-4e13-8f2f-670b2c597658/volumes" Oct 30 17:17:38.936500 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:38.936451 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cdwcj"] Oct 30 17:17:43.804444 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:43.804405 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-750dcb-56df54cc5-nsm9f"] Oct 30 17:17:43.825981 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:43.804644 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-750dcb-56df54cc5-nsm9f" podUID="d56798b3-ff19-4644-8d23-2967f523427d" containerName="maas-api" containerID="cri-o://cb10f1d06b15e55269145534c2d31d96dd48ea2c8968b2c868250b4727c84c0b" gracePeriod=30 Oct 30 17:17:44.107628 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:44.107601 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-750dcb-56df54cc5-nsm9f" Oct 30 17:17:44.125861 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:44.125834 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d56798b3-ff19-4644-8d23-2967f523427d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d56798b3-ff19-4644-8d23-2967f523427d-openshift-service-ca\") pod \"d56798b3-ff19-4644-8d23-2967f523427d\" (UID: \"d56798b3-ff19-4644-8d23-2967f523427d\") " Oct 30 17:17:44.126054 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:44.125945 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d56798b3-ff19-4644-8d23-2967f523427d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d56798b3-ff19-4644-8d23-2967f523427d-maas-api-tls\") pod \"d56798b3-ff19-4644-8d23-2967f523427d\" (UID: \"d56798b3-ff19-4644-8d23-2967f523427d\") " Oct 30 17:17:44.126054 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:44.125974 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d56798b3-ff19-4644-8d23-2967f523427d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d56798b3-ff19-4644-8d23-2967f523427d-metrics-certs\") pod \"d56798b3-ff19-4644-8d23-2967f523427d\" (UID: \"d56798b3-ff19-4644-8d23-2967f523427d\") " Oct 30 17:17:44.126288 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:44.126266 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d56798b3-ff19-4644-8d23-2967f523427d-openshift-service-ca" pod "d56798b3-ff19-4644-8d23-2967f523427d" (UID: "d56798b3-ff19-4644-8d23-2967f523427d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:17:44.128655 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:44.128625 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d56798b3-ff19-4644-8d23-2967f523427d-maas-api-tls" pod "d56798b3-ff19-4644-8d23-2967f523427d" (UID: "d56798b3-ff19-4644-8d23-2967f523427d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:17:44.128655 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:44.128636 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d56798b3-ff19-4644-8d23-2967f523427d-metrics-certs" pod "d56798b3-ff19-4644-8d23-2967f523427d" (UID: "d56798b3-ff19-4644-8d23-2967f523427d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:17:44.199409 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:44.199369 5052 generic.go:356] "Generic (PLEG): container finished" podID="d56798b3-ff19-4644-8d23-2967f523427d" containerID="cb10f1d06b15e55269145534c2d31d96dd48ea2c8968b2c868250b4727c84c0b" exitCode=0 Oct 30 17:17:44.199634 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:44.199426 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-750dcb-56df54cc5-nsm9f" Oct 30 17:17:44.199634 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:44.199462 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-750dcb-56df54cc5-nsm9f" event={"ID":"d56798b3-ff19-4644-8d23-2967f523427d","Type":"ContainerDied","Data":"cb10f1d06b15e55269145534c2d31d96dd48ea2c8968b2c868250b4727c84c0b"} Oct 30 17:17:44.199634 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:44.199533 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-750dcb-56df54cc5-nsm9f" event={"ID":"d56798b3-ff19-4644-8d23-2967f523427d","Type":"ContainerDied","Data":"ce9db742e719d23196e32f7f014af9f14e87aee1c6296cbddaf122be17d8030d"} Oct 30 17:17:44.199634 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:44.199573 5052 scope.go:122] "RemoveContainer" containerID="cb10f1d06b15e55269145534c2d31d96dd48ea2c8968b2c868250b4727c84c0b" Oct 30 17:17:44.210262 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:44.210242 5052 scope.go:122] "RemoveContainer" containerID="cb10f1d06b15e55269145534c2d31d96dd48ea2c8968b2c868250b4727c84c0b" Oct 30 17:17:44.210555 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:17:44.210534 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb10f1d06b15e55269145534c2d31d96dd48ea2c8968b2c868250b4727c84c0b\": container with ID starting with cb10f1d06b15e55269145534c2d31d96dd48ea2c8968b2c868250b4727c84c0b not found: ID does not exist" containerID="cb10f1d06b15e55269145534c2d31d96dd48ea2c8968b2c868250b4727c84c0b" Oct 30 17:17:44.210658 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:44.210564 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb10f1d06b15e55269145534c2d31d96dd48ea2c8968b2c868250b4727c84c0b"} err="failed to get container status \"cb10f1d06b15e55269145534c2d31d96dd48ea2c8968b2c868250b4727c84c0b\": rpc error: code = NotFound desc = could not find container \"cb10f1d06b15e55269145534c2d31d96dd48ea2c8968b2c868250b4727c84c0b\": container with ID starting with cb10f1d06b15e55269145534c2d31d96dd48ea2c8968b2c868250b4727c84c0b not found: ID does not exist" Oct 30 17:17:44.226520 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:44.226499 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d56798b3-ff19-4644-8d23-2967f523427d-kube-api-access-vsslt\" (UniqueName: \"kubernetes.io/projected/d56798b3-ff19-4644-8d23-2967f523427d-kube-api-access-vsslt\") pod \"d56798b3-ff19-4644-8d23-2967f523427d\" (UID: \"d56798b3-ff19-4644-8d23-2967f523427d\") " Oct 30 17:17:44.226870 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:44.226851 5052 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d56798b3-ff19-4644-8d23-2967f523427d-openshift-service-ca\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:17:44.226944 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:44.226877 5052 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d56798b3-ff19-4644-8d23-2967f523427d-maas-api-tls\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:17:44.226944 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:44.226891 5052 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d56798b3-ff19-4644-8d23-2967f523427d-metrics-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:17:44.228894 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:44.228872 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d56798b3-ff19-4644-8d23-2967f523427d-kube-api-access-vsslt" pod "d56798b3-ff19-4644-8d23-2967f523427d" (UID: "d56798b3-ff19-4644-8d23-2967f523427d"). InnerVolumeSpecName "kube-api-access-vsslt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:17:44.327613 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:44.327559 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsslt\" (UniqueName: \"kubernetes.io/projected/d56798b3-ff19-4644-8d23-2967f523427d-kube-api-access-vsslt\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:17:44.341563 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:44.341478 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9e533f-85c9dd5979-nrmbf" podUID="8054915f-a3d1-44d0-ab47-4ca0161255da" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:17:44.547735 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:44.547704 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-750dcb-56df54cc5-nsm9f"] Oct 30 17:17:44.554031 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:44.554000 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-750dcb-56df54cc5-nsm9f"] Oct 30 17:17:44.597517 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:44.597439 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d56798b3-ff19-4644-8d23-2967f523427d" path="/var/lib/kubelet/pods/d56798b3-ff19-4644-8d23-2967f523427d/volumes" Oct 30 17:17:45.348964 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:45.348923 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-9e533f-57fd8bd864-t6vr2" podUID="4963d7c3-cee3-430c-8e1c-156c79a75a3d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:17:48.840211 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:17:48.840177 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 17:17:50.205486 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:50.205451 5052 scope.go:122] "RemoveContainer" containerID="ae5569953f52787236943cca4f21042f58288bb70da33906cca7983c628910e1" Oct 30 17:17:51.297660 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:51.297626 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cdwcj"] Oct 30 17:17:51.966178 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:51.966137 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9e533f-85c9dd5979-nrmbf" Oct 30 17:17:51.993282 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:51.993255 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8054915f-a3d1-44d0-ab47-4ca0161255da-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8054915f-a3d1-44d0-ab47-4ca0161255da-plugins-config-volume\") pod \"8054915f-a3d1-44d0-ab47-4ca0161255da\" (UID: \"8054915f-a3d1-44d0-ab47-4ca0161255da\") " Oct 30 17:17:51.993421 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:51.993286 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8054915f-a3d1-44d0-ab47-4ca0161255da-kube-api-access-h9rhc\" (UniqueName: \"kubernetes.io/projected/8054915f-a3d1-44d0-ab47-4ca0161255da-kube-api-access-h9rhc\") pod \"8054915f-a3d1-44d0-ab47-4ca0161255da\" (UID: \"8054915f-a3d1-44d0-ab47-4ca0161255da\") " Oct 30 17:17:51.993646 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:51.993625 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8054915f-a3d1-44d0-ab47-4ca0161255da-plugins-config-volume" pod "8054915f-a3d1-44d0-ab47-4ca0161255da" (UID: "8054915f-a3d1-44d0-ab47-4ca0161255da"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:17:51.995295 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:51.995276 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8054915f-a3d1-44d0-ab47-4ca0161255da-kube-api-access-h9rhc" pod "8054915f-a3d1-44d0-ab47-4ca0161255da" (UID: "8054915f-a3d1-44d0-ab47-4ca0161255da"). InnerVolumeSpecName "kube-api-access-h9rhc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:17:52.094441 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:52.094319 5052 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8054915f-a3d1-44d0-ab47-4ca0161255da-plugins-config-volume\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:17:52.094441 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:52.094353 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9rhc\" (UniqueName: \"kubernetes.io/projected/8054915f-a3d1-44d0-ab47-4ca0161255da-kube-api-access-h9rhc\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:17:52.104626 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:52.104595 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-dnwt8"] Oct 30 17:17:52.104878 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:52.104844 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-dnwt8" podUID="c8947ffd-7539-43d8-86d5-fc17dd8c7d6e" containerName="istio-proxy" containerID="cri-o://061347d21117ffeb6f479ab871723e42258058119396dfb814a824800ae970ef" gracePeriod=30 Oct 30 17:17:52.248030 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:52.247990 5052 generic.go:356] "Generic (PLEG): container finished" podID="4963d7c3-cee3-430c-8e1c-156c79a75a3d" containerID="3e52bc09f7330f22593ced39b6dad44b40e6f6b9adb426971337f4627e150ffa" exitCode=0 Oct 30 17:17:52.248203 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:52.248065 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-9e533f-57fd8bd864-t6vr2" event={"ID":"4963d7c3-cee3-430c-8e1c-156c79a75a3d","Type":"ContainerDied","Data":"3e52bc09f7330f22593ced39b6dad44b40e6f6b9adb426971337f4627e150ffa"} Oct 30 17:17:52.249444 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:52.249421 5052 generic.go:356] "Generic (PLEG): container finished" podID="8054915f-a3d1-44d0-ab47-4ca0161255da" containerID="9c88eb46a20297d4d538b40dbd7a013fa2d48b920ea4aa9562ce897a8daa31c4" exitCode=0 Oct 30 17:17:52.249572 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:52.249482 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9e533f-85c9dd5979-nrmbf" Oct 30 17:17:52.249572 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:52.249502 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9e533f-85c9dd5979-nrmbf" event={"ID":"8054915f-a3d1-44d0-ab47-4ca0161255da","Type":"ContainerDied","Data":"9c88eb46a20297d4d538b40dbd7a013fa2d48b920ea4aa9562ce897a8daa31c4"} Oct 30 17:17:52.249572 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:52.249552 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9e533f-85c9dd5979-nrmbf" event={"ID":"8054915f-a3d1-44d0-ab47-4ca0161255da","Type":"ContainerDied","Data":"bc86857811cdcc3c0fb6aff9dd8e1b00f24b4c33fc1c723cd530def86c39f283"} Oct 30 17:17:52.249572 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:52.249590 5052 scope.go:122] "RemoveContainer" containerID="9c88eb46a20297d4d538b40dbd7a013fa2d48b920ea4aa9562ce897a8daa31c4" Oct 30 17:17:52.268802 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:52.268757 5052 scope.go:122] "RemoveContainer" containerID="9c88eb46a20297d4d538b40dbd7a013fa2d48b920ea4aa9562ce897a8daa31c4" Oct 30 17:17:52.270097 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:17:52.269292 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c88eb46a20297d4d538b40dbd7a013fa2d48b920ea4aa9562ce897a8daa31c4\": container with ID starting with 9c88eb46a20297d4d538b40dbd7a013fa2d48b920ea4aa9562ce897a8daa31c4 not found: ID does not exist" containerID="9c88eb46a20297d4d538b40dbd7a013fa2d48b920ea4aa9562ce897a8daa31c4" Oct 30 17:17:52.270097 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:52.269336 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c88eb46a20297d4d538b40dbd7a013fa2d48b920ea4aa9562ce897a8daa31c4"} err="failed to get container status \"9c88eb46a20297d4d538b40dbd7a013fa2d48b920ea4aa9562ce897a8daa31c4\": rpc error: code = NotFound desc = could not find container \"9c88eb46a20297d4d538b40dbd7a013fa2d48b920ea4aa9562ce897a8daa31c4\": container with ID starting with 9c88eb46a20297d4d538b40dbd7a013fa2d48b920ea4aa9562ce897a8daa31c4 not found: ID does not exist" Oct 30 17:17:52.313035 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:52.312994 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9e533f-85c9dd5979-nrmbf"] Oct 30 17:17:52.320307 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:52.320275 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9e533f-85c9dd5979-nrmbf"] Oct 30 17:17:52.338850 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:52.338823 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-9e533f-57fd8bd864-t6vr2" Oct 30 17:17:52.397646 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:52.397612 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4963d7c3-cee3-430c-8e1c-156c79a75a3d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4963d7c3-cee3-430c-8e1c-156c79a75a3d-plugins-config-volume\") pod \"4963d7c3-cee3-430c-8e1c-156c79a75a3d\" (UID: \"4963d7c3-cee3-430c-8e1c-156c79a75a3d\") " Oct 30 17:17:52.397840 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:52.397718 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4963d7c3-cee3-430c-8e1c-156c79a75a3d-kube-api-access-gg4gh\" (UniqueName: \"kubernetes.io/projected/4963d7c3-cee3-430c-8e1c-156c79a75a3d-kube-api-access-gg4gh\") pod \"4963d7c3-cee3-430c-8e1c-156c79a75a3d\" (UID: \"4963d7c3-cee3-430c-8e1c-156c79a75a3d\") " Oct 30 17:17:52.398002 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:52.397980 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4963d7c3-cee3-430c-8e1c-156c79a75a3d-plugins-config-volume" pod "4963d7c3-cee3-430c-8e1c-156c79a75a3d" (UID: "4963d7c3-cee3-430c-8e1c-156c79a75a3d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:17:52.399929 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:52.399902 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4963d7c3-cee3-430c-8e1c-156c79a75a3d-kube-api-access-gg4gh" pod "4963d7c3-cee3-430c-8e1c-156c79a75a3d" (UID: "4963d7c3-cee3-430c-8e1c-156c79a75a3d"). InnerVolumeSpecName "kube-api-access-gg4gh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:17:52.498843 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:52.498763 5052 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4963d7c3-cee3-430c-8e1c-156c79a75a3d-plugins-config-volume\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:17:52.498843 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:52.498791 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gg4gh\" (UniqueName: \"kubernetes.io/projected/4963d7c3-cee3-430c-8e1c-156c79a75a3d-kube-api-access-gg4gh\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:17:52.598083 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:52.598045 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8054915f-a3d1-44d0-ab47-4ca0161255da" path="/var/lib/kubelet/pods/8054915f-a3d1-44d0-ab47-4ca0161255da/volumes" Oct 30 17:17:53.257860 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:53.257825 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-9e533f-57fd8bd864-t6vr2" event={"ID":"4963d7c3-cee3-430c-8e1c-156c79a75a3d","Type":"ContainerDied","Data":"873dcca7d92d2269123cdcfd3c37fa81438dcff6a0936513b4a924f70e9b0b65"} Oct 30 17:17:53.257860 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:53.257853 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-9e533f-57fd8bd864-t6vr2" Oct 30 17:17:53.258090 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:53.257885 5052 scope.go:122] "RemoveContainer" containerID="3e52bc09f7330f22593ced39b6dad44b40e6f6b9adb426971337f4627e150ffa" Oct 30 17:17:53.314184 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:53.314144 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-9e533f-57fd8bd864-t6vr2"] Oct 30 17:17:53.327452 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:53.327426 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-9e533f-57fd8bd864-t6vr2"] Oct 30 17:17:54.597256 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:54.597219 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4963d7c3-cee3-430c-8e1c-156c79a75a3d" path="/var/lib/kubelet/pods/4963d7c3-cee3-430c-8e1c-156c79a75a3d/volumes" Oct 30 17:17:57.284721 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:57.284658 5052 generic.go:356] "Generic (PLEG): container finished" podID="c8947ffd-7539-43d8-86d5-fc17dd8c7d6e" containerID="061347d21117ffeb6f479ab871723e42258058119396dfb814a824800ae970ef" exitCode=0 Oct 30 17:17:57.285061 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:57.284731 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-dnwt8" event={"ID":"c8947ffd-7539-43d8-86d5-fc17dd8c7d6e","Type":"ContainerDied","Data":"061347d21117ffeb6f479ab871723e42258058119396dfb814a824800ae970ef"} Oct 30 17:17:57.369215 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:57.369187 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-dnwt8" Oct 30 17:17:57.446925 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:57.446895 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8947ffd-7539-43d8-86d5-fc17dd8c7d6e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8947ffd-7539-43d8-86d5-fc17dd8c7d6e-credential-socket\") pod \"c8947ffd-7539-43d8-86d5-fc17dd8c7d6e\" (UID: \"c8947ffd-7539-43d8-86d5-fc17dd8c7d6e\") " Oct 30 17:17:57.447091 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:57.446999 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8947ffd-7539-43d8-86d5-fc17dd8c7d6e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8947ffd-7539-43d8-86d5-fc17dd8c7d6e-istio-data\") pod \"c8947ffd-7539-43d8-86d5-fc17dd8c7d6e\" (UID: \"c8947ffd-7539-43d8-86d5-fc17dd8c7d6e\") " Oct 30 17:17:57.447091 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:57.447041 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8947ffd-7539-43d8-86d5-fc17dd8c7d6e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8947ffd-7539-43d8-86d5-fc17dd8c7d6e-workload-socket\") pod \"c8947ffd-7539-43d8-86d5-fc17dd8c7d6e\" (UID: \"c8947ffd-7539-43d8-86d5-fc17dd8c7d6e\") " Oct 30 17:17:57.447091 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:57.447065 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8947ffd-7539-43d8-86d5-fc17dd8c7d6e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8947ffd-7539-43d8-86d5-fc17dd8c7d6e-istio-envoy\") pod \"c8947ffd-7539-43d8-86d5-fc17dd8c7d6e\" (UID: \"c8947ffd-7539-43d8-86d5-fc17dd8c7d6e\") " Oct 30 17:17:57.447091 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:57.447085 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8947ffd-7539-43d8-86d5-fc17dd8c7d6e-istio-token\" (UniqueName: \"kubernetes.io/projected/c8947ffd-7539-43d8-86d5-fc17dd8c7d6e-istio-token\") pod \"c8947ffd-7539-43d8-86d5-fc17dd8c7d6e\" (UID: \"c8947ffd-7539-43d8-86d5-fc17dd8c7d6e\") " Oct 30 17:17:57.447352 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:57.447106 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c8947ffd-7539-43d8-86d5-fc17dd8c7d6e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8947ffd-7539-43d8-86d5-fc17dd8c7d6e-istio-podinfo\") pod \"c8947ffd-7539-43d8-86d5-fc17dd8c7d6e\" (UID: \"c8947ffd-7539-43d8-86d5-fc17dd8c7d6e\") " Oct 30 17:17:57.447352 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:57.447150 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8947ffd-7539-43d8-86d5-fc17dd8c7d6e-kube-api-access-2h7wj\" (UniqueName: \"kubernetes.io/projected/c8947ffd-7539-43d8-86d5-fc17dd8c7d6e-kube-api-access-2h7wj\") pod \"c8947ffd-7539-43d8-86d5-fc17dd8c7d6e\" (UID: \"c8947ffd-7539-43d8-86d5-fc17dd8c7d6e\") " Oct 30 17:17:57.447352 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:57.447185 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8947ffd-7539-43d8-86d5-fc17dd8c7d6e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8947ffd-7539-43d8-86d5-fc17dd8c7d6e-workload-certs\") pod \"c8947ffd-7539-43d8-86d5-fc17dd8c7d6e\" (UID: \"c8947ffd-7539-43d8-86d5-fc17dd8c7d6e\") " Oct 30 17:17:57.447352 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:57.447198 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8947ffd-7539-43d8-86d5-fc17dd8c7d6e-credential-socket" pod "c8947ffd-7539-43d8-86d5-fc17dd8c7d6e" (UID: "c8947ffd-7539-43d8-86d5-fc17dd8c7d6e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:17:57.447352 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:57.447215 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8947ffd-7539-43d8-86d5-fc17dd8c7d6e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8947ffd-7539-43d8-86d5-fc17dd8c7d6e-istiod-ca-cert\") pod \"c8947ffd-7539-43d8-86d5-fc17dd8c7d6e\" (UID: \"c8947ffd-7539-43d8-86d5-fc17dd8c7d6e\") " Oct 30 17:17:57.447352 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:57.447224 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8947ffd-7539-43d8-86d5-fc17dd8c7d6e-istio-data" pod "c8947ffd-7539-43d8-86d5-fc17dd8c7d6e" (UID: "c8947ffd-7539-43d8-86d5-fc17dd8c7d6e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:17:57.447709 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:57.447412 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8947ffd-7539-43d8-86d5-fc17dd8c7d6e-workload-socket" pod "c8947ffd-7539-43d8-86d5-fc17dd8c7d6e" (UID: "c8947ffd-7539-43d8-86d5-fc17dd8c7d6e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:17:57.447775 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:57.447746 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8947ffd-7539-43d8-86d5-fc17dd8c7d6e-workload-certs" pod "c8947ffd-7539-43d8-86d5-fc17dd8c7d6e" (UID: "c8947ffd-7539-43d8-86d5-fc17dd8c7d6e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:17:57.447842 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:57.447805 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8947ffd-7539-43d8-86d5-fc17dd8c7d6e-istiod-ca-cert" pod "c8947ffd-7539-43d8-86d5-fc17dd8c7d6e" (UID: "c8947ffd-7539-43d8-86d5-fc17dd8c7d6e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:17:57.447961 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:57.447944 5052 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8947ffd-7539-43d8-86d5-fc17dd8c7d6e-credential-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:17:57.448034 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:57.447966 5052 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8947ffd-7539-43d8-86d5-fc17dd8c7d6e-istio-data\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:17:57.448034 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:57.447980 5052 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8947ffd-7539-43d8-86d5-fc17dd8c7d6e-workload-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:17:57.449890 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:57.449864 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c8947ffd-7539-43d8-86d5-fc17dd8c7d6e-istio-podinfo" pod "c8947ffd-7539-43d8-86d5-fc17dd8c7d6e" (UID: "c8947ffd-7539-43d8-86d5-fc17dd8c7d6e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:17:57.450037 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:57.450017 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8947ffd-7539-43d8-86d5-fc17dd8c7d6e-kube-api-access-2h7wj" pod "c8947ffd-7539-43d8-86d5-fc17dd8c7d6e" (UID: "c8947ffd-7539-43d8-86d5-fc17dd8c7d6e"). InnerVolumeSpecName "kube-api-access-2h7wj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:17:57.450106 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:57.450089 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8947ffd-7539-43d8-86d5-fc17dd8c7d6e-istio-envoy" pod "c8947ffd-7539-43d8-86d5-fc17dd8c7d6e" (UID: "c8947ffd-7539-43d8-86d5-fc17dd8c7d6e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:17:57.450147 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:57.450103 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8947ffd-7539-43d8-86d5-fc17dd8c7d6e-istio-token" pod "c8947ffd-7539-43d8-86d5-fc17dd8c7d6e" (UID: "c8947ffd-7539-43d8-86d5-fc17dd8c7d6e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:17:57.548996 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:57.548960 5052 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8947ffd-7539-43d8-86d5-fc17dd8c7d6e-istio-envoy\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:17:57.548996 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:57.548987 5052 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8947ffd-7539-43d8-86d5-fc17dd8c7d6e-istio-token\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:17:57.548996 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:57.548996 5052 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8947ffd-7539-43d8-86d5-fc17dd8c7d6e-istio-podinfo\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:17:57.548996 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:57.549004 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2h7wj\" (UniqueName: \"kubernetes.io/projected/c8947ffd-7539-43d8-86d5-fc17dd8c7d6e-kube-api-access-2h7wj\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:17:57.549282 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:57.549015 5052 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8947ffd-7539-43d8-86d5-fc17dd8c7d6e-workload-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:17:57.549282 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:57.549023 5052 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8947ffd-7539-43d8-86d5-fc17dd8c7d6e-istiod-ca-cert\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:17:58.290999 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:58.290969 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-dnwt8" Oct 30 17:17:58.291495 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:58.291003 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-dnwt8" event={"ID":"c8947ffd-7539-43d8-86d5-fc17dd8c7d6e","Type":"ContainerDied","Data":"2ce1471145b6e14b0aaa1141f3bc321551749b6d30bf4ebc1f823a929806562f"} Oct 30 17:17:58.291495 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:58.291066 5052 scope.go:122] "RemoveContainer" containerID="061347d21117ffeb6f479ab871723e42258058119396dfb814a824800ae970ef" Oct 30 17:17:58.354883 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:58.354853 5052 patch_prober.go:29] interesting pod/e2e-worker-w2-750dcb-openshift-default-777f567b9f-dnwt8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:17:58.354995 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:58.354920 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-dnwt8" podUID="c8947ffd-7539-43d8-86d5-fc17dd8c7d6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:17:58.361144 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:58.361117 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-dnwt8"] Oct 30 17:17:58.376681 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:58.376656 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-dnwt8"] Oct 30 17:17:58.595950 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:17:58.595873 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8947ffd-7539-43d8-86d5-fc17dd8c7d6e" path="/var/lib/kubelet/pods/c8947ffd-7539-43d8-86d5-fc17dd8c7d6e/volumes" Oct 30 17:18:03.922964 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:03.922930 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cdwcj"] Oct 30 17:18:06.326203 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:06.326168 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-3e2f96-57dd88bc59-l2k57"] Oct 30 17:18:06.326734 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:06.326475 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3e2f96-57dd88bc59-l2k57" podUID="d7adb9b2-ac3c-440c-8338-f0a4375625d3" containerName="maas-api" containerID="cri-o://4ea7a2413f58480d7336dbb505e6335cf54cb4993d1cc08035b42ce867665604" gracePeriod=30 Oct 30 17:18:06.598433 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:06.598405 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3e2f96-57dd88bc59-l2k57" Oct 30 17:18:06.734203 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:06.734169 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7adb9b2-ac3c-440c-8338-f0a4375625d3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7adb9b2-ac3c-440c-8338-f0a4375625d3-maas-api-tls\") pod \"d7adb9b2-ac3c-440c-8338-f0a4375625d3\" (UID: \"d7adb9b2-ac3c-440c-8338-f0a4375625d3\") " Oct 30 17:18:06.734452 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:06.734262 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7adb9b2-ac3c-440c-8338-f0a4375625d3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7adb9b2-ac3c-440c-8338-f0a4375625d3-metrics-certs\") pod \"d7adb9b2-ac3c-440c-8338-f0a4375625d3\" (UID: \"d7adb9b2-ac3c-440c-8338-f0a4375625d3\") " Oct 30 17:18:06.734452 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:06.734324 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7adb9b2-ac3c-440c-8338-f0a4375625d3-kube-api-access-mvz7h\" (UniqueName: \"kubernetes.io/projected/d7adb9b2-ac3c-440c-8338-f0a4375625d3-kube-api-access-mvz7h\") pod \"d7adb9b2-ac3c-440c-8338-f0a4375625d3\" (UID: \"d7adb9b2-ac3c-440c-8338-f0a4375625d3\") " Oct 30 17:18:06.734452 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:06.734366 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7adb9b2-ac3c-440c-8338-f0a4375625d3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7adb9b2-ac3c-440c-8338-f0a4375625d3-openshift-service-ca\") pod \"d7adb9b2-ac3c-440c-8338-f0a4375625d3\" (UID: \"d7adb9b2-ac3c-440c-8338-f0a4375625d3\") " Oct 30 17:18:06.734846 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:06.734822 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7adb9b2-ac3c-440c-8338-f0a4375625d3-openshift-service-ca" pod "d7adb9b2-ac3c-440c-8338-f0a4375625d3" (UID: "d7adb9b2-ac3c-440c-8338-f0a4375625d3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:18:06.736304 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:06.736282 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7adb9b2-ac3c-440c-8338-f0a4375625d3-maas-api-tls" pod "d7adb9b2-ac3c-440c-8338-f0a4375625d3" (UID: "d7adb9b2-ac3c-440c-8338-f0a4375625d3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:18:06.736409 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:06.736387 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7adb9b2-ac3c-440c-8338-f0a4375625d3-metrics-certs" pod "d7adb9b2-ac3c-440c-8338-f0a4375625d3" (UID: "d7adb9b2-ac3c-440c-8338-f0a4375625d3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:18:06.736409 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:06.736400 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7adb9b2-ac3c-440c-8338-f0a4375625d3-kube-api-access-mvz7h" pod "d7adb9b2-ac3c-440c-8338-f0a4375625d3" (UID: "d7adb9b2-ac3c-440c-8338-f0a4375625d3"). InnerVolumeSpecName "kube-api-access-mvz7h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:18:06.836204 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:06.836130 5052 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7adb9b2-ac3c-440c-8338-f0a4375625d3-maas-api-tls\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:18:06.836204 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:06.836157 5052 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7adb9b2-ac3c-440c-8338-f0a4375625d3-metrics-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:18:06.836204 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:06.836166 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvz7h\" (UniqueName: \"kubernetes.io/projected/d7adb9b2-ac3c-440c-8338-f0a4375625d3-kube-api-access-mvz7h\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:18:06.836204 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:06.836175 5052 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7adb9b2-ac3c-440c-8338-f0a4375625d3-openshift-service-ca\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:18:07.350853 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:07.350816 5052 generic.go:356] "Generic (PLEG): container finished" podID="d7adb9b2-ac3c-440c-8338-f0a4375625d3" containerID="4ea7a2413f58480d7336dbb505e6335cf54cb4993d1cc08035b42ce867665604" exitCode=0 Oct 30 17:18:07.351294 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:07.350860 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3e2f96-57dd88bc59-l2k57" Oct 30 17:18:07.351294 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:07.350879 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3e2f96-57dd88bc59-l2k57" event={"ID":"d7adb9b2-ac3c-440c-8338-f0a4375625d3","Type":"ContainerDied","Data":"4ea7a2413f58480d7336dbb505e6335cf54cb4993d1cc08035b42ce867665604"} Oct 30 17:18:07.351294 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:07.350919 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3e2f96-57dd88bc59-l2k57" event={"ID":"d7adb9b2-ac3c-440c-8338-f0a4375625d3","Type":"ContainerDied","Data":"6a2dc837f0d90dc65fa4d50d8df81b581b5c7a4f2078c864fb2682a8505b7724"} Oct 30 17:18:07.351294 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:07.350940 5052 scope.go:122] "RemoveContainer" containerID="4ea7a2413f58480d7336dbb505e6335cf54cb4993d1cc08035b42ce867665604" Oct 30 17:18:07.361999 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:07.361974 5052 scope.go:122] "RemoveContainer" containerID="4ea7a2413f58480d7336dbb505e6335cf54cb4993d1cc08035b42ce867665604" Oct 30 17:18:07.364203 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:18:07.364177 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ea7a2413f58480d7336dbb505e6335cf54cb4993d1cc08035b42ce867665604\": container with ID starting with 4ea7a2413f58480d7336dbb505e6335cf54cb4993d1cc08035b42ce867665604 not found: ID does not exist" containerID="4ea7a2413f58480d7336dbb505e6335cf54cb4993d1cc08035b42ce867665604" Oct 30 17:18:07.364312 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:07.364212 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ea7a2413f58480d7336dbb505e6335cf54cb4993d1cc08035b42ce867665604"} err="failed to get container status \"4ea7a2413f58480d7336dbb505e6335cf54cb4993d1cc08035b42ce867665604\": rpc error: code = NotFound desc = could not find container \"4ea7a2413f58480d7336dbb505e6335cf54cb4993d1cc08035b42ce867665604\": container with ID starting with 4ea7a2413f58480d7336dbb505e6335cf54cb4993d1cc08035b42ce867665604 not found: ID does not exist" Oct 30 17:18:07.405902 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:07.405878 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-3e2f96-57dd88bc59-l2k57"] Oct 30 17:18:07.419093 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:07.419073 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-3e2f96-57dd88bc59-l2k57"] Oct 30 17:18:08.596944 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:08.596906 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7adb9b2-ac3c-440c-8338-f0a4375625d3" path="/var/lib/kubelet/pods/d7adb9b2-ac3c-440c-8338-f0a4375625d3/volumes" Oct 30 17:18:12.289515 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:12.289432 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cdwcj"] Oct 30 17:18:18.839788 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:18:18.839749 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 17:18:29.327940 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:29.327904 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-575c7b5c5b-lh44m"] Oct 30 17:18:29.328462 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:29.328100 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-575c7b5c5b-lh44m" podUID="c997fe94-1cf0-4316-826e-6bd921636715" containerName="manager" containerID="cri-o://5481a05961ec5bf038b9a3b2e17d1c5e41beb314db8388fb2cdc2cc8099dc91f" gracePeriod=30 Oct 30 17:18:29.464205 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:29.464072 5052 generic.go:356] "Generic (PLEG): container finished" podID="c997fe94-1cf0-4316-826e-6bd921636715" containerID="5481a05961ec5bf038b9a3b2e17d1c5e41beb314db8388fb2cdc2cc8099dc91f" exitCode=0 Oct 30 17:18:29.464205 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:29.464116 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-575c7b5c5b-lh44m" event={"ID":"c997fe94-1cf0-4316-826e-6bd921636715","Type":"ContainerDied","Data":"5481a05961ec5bf038b9a3b2e17d1c5e41beb314db8388fb2cdc2cc8099dc91f"} Oct 30 17:18:29.609407 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:29.609381 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-575c7b5c5b-lh44m" Oct 30 17:18:29.756740 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:29.756697 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c997fe94-1cf0-4316-826e-6bd921636715-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c997fe94-1cf0-4316-826e-6bd921636715-metrics-certs\") pod \"c997fe94-1cf0-4316-826e-6bd921636715\" (UID: \"c997fe94-1cf0-4316-826e-6bd921636715\") " Oct 30 17:18:29.756961 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:29.756840 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c997fe94-1cf0-4316-826e-6bd921636715-webhook-cert\" (UniqueName: \"kubernetes.io/secret/c997fe94-1cf0-4316-826e-6bd921636715-webhook-cert\") pod \"c997fe94-1cf0-4316-826e-6bd921636715\" (UID: \"c997fe94-1cf0-4316-826e-6bd921636715\") " Oct 30 17:18:29.756961 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:29.756885 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c997fe94-1cf0-4316-826e-6bd921636715-kube-api-access-b9bvv\" (UniqueName: \"kubernetes.io/projected/c997fe94-1cf0-4316-826e-6bd921636715-kube-api-access-b9bvv\") pod \"c997fe94-1cf0-4316-826e-6bd921636715\" (UID: \"c997fe94-1cf0-4316-826e-6bd921636715\") " Oct 30 17:18:29.760714 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:29.760687 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c997fe94-1cf0-4316-826e-6bd921636715-webhook-cert" pod "c997fe94-1cf0-4316-826e-6bd921636715" (UID: "c997fe94-1cf0-4316-826e-6bd921636715"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:18:29.762081 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:29.762054 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c997fe94-1cf0-4316-826e-6bd921636715-metrics-certs" pod "c997fe94-1cf0-4316-826e-6bd921636715" (UID: "c997fe94-1cf0-4316-826e-6bd921636715"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:18:29.762250 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:29.762231 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c997fe94-1cf0-4316-826e-6bd921636715-kube-api-access-b9bvv" pod "c997fe94-1cf0-4316-826e-6bd921636715" (UID: "c997fe94-1cf0-4316-826e-6bd921636715"). InnerVolumeSpecName "kube-api-access-b9bvv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:18:29.858529 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:29.858440 5052 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c997fe94-1cf0-4316-826e-6bd921636715-webhook-cert\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:18:29.858529 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:29.858472 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9bvv\" (UniqueName: \"kubernetes.io/projected/c997fe94-1cf0-4316-826e-6bd921636715-kube-api-access-b9bvv\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:18:29.858529 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:29.858485 5052 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c997fe94-1cf0-4316-826e-6bd921636715-metrics-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:18:30.471643 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:30.471607 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-575c7b5c5b-lh44m" Oct 30 17:18:30.472118 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:30.471612 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-575c7b5c5b-lh44m" event={"ID":"c997fe94-1cf0-4316-826e-6bd921636715","Type":"ContainerDied","Data":"cccb83bd0c96b4c272b1f8a8a44014a6b37d943804eb29120b9923c41fc88d6a"} Oct 30 17:18:30.472118 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:30.471776 5052 scope.go:122] "RemoveContainer" containerID="5481a05961ec5bf038b9a3b2e17d1c5e41beb314db8388fb2cdc2cc8099dc91f" Oct 30 17:18:30.526876 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:30.526845 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-575c7b5c5b-lh44m"] Oct 30 17:18:30.540225 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:30.540190 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-575c7b5c5b-lh44m"] Oct 30 17:18:30.597142 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:30.597111 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c997fe94-1cf0-4316-826e-6bd921636715" path="/var/lib/kubelet/pods/c997fe94-1cf0-4316-826e-6bd921636715/volumes" Oct 30 17:18:31.200168 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.200136 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-575c7b5c5b-f64sn"] Oct 30 17:18:31.200542 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.200528 5052 cpu_manager.go:414] "containerMap: removing container" podUID="505e6f2b-634b-40ae-b5db-6ced8e1bf322" containerName="istio-proxy" Oct 30 17:18:31.200634 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.200542 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="505e6f2b-634b-40ae-b5db-6ced8e1bf322" containerName="istio-proxy" Oct 30 17:18:31.200634 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.200552 5052 cpu_manager.go:414] "containerMap: removing container" podUID="e8eabbfd-4dce-4087-a350-c55cb7c7baf3" containerName="istio-proxy" Oct 30 17:18:31.200634 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.200557 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8eabbfd-4dce-4087-a350-c55cb7c7baf3" containerName="istio-proxy" Oct 30 17:18:31.200634 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.200570 5052 cpu_manager.go:414] "containerMap: removing container" podUID="e4036d90-5e91-477e-b81f-91c2dceee6c9" containerName="istio-proxy" Oct 30 17:18:31.200634 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.200575 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4036d90-5e91-477e-b81f-91c2dceee6c9" containerName="istio-proxy" Oct 30 17:18:31.200634 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.200598 5052 cpu_manager.go:414] "containerMap: removing container" podUID="0261ebf2-6dfd-4df5-b5de-4b7207ca761b" containerName="istio-proxy" Oct 30 17:18:31.200634 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.200603 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0261ebf2-6dfd-4df5-b5de-4b7207ca761b" containerName="istio-proxy" Oct 30 17:18:31.200634 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.200609 5052 cpu_manager.go:414] "containerMap: removing container" podUID="7c8c2e2b-b5e5-4e6c-ace8-db859e693dde" containerName="cleanup" Oct 30 17:18:31.200634 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.200615 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c8c2e2b-b5e5-4e6c-ace8-db859e693dde" containerName="cleanup" Oct 30 17:18:31.200634 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.200622 5052 cpu_manager.go:414] "containerMap: removing container" podUID="15ed43ef-5cdd-489a-a70e-063239822ff3" containerName="main" Oct 30 17:18:31.200634 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.200627 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15ed43ef-5cdd-489a-a70e-063239822ff3" containerName="main" Oct 30 17:18:31.200634 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.200634 5052 cpu_manager.go:414] "containerMap: removing container" podUID="1e6b7be8-b362-4477-86b5-cf8ff0cd266f" containerName="payload-pre-processing" Oct 30 17:18:31.201026 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.200640 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e6b7be8-b362-4477-86b5-cf8ff0cd266f" containerName="payload-pre-processing" Oct 30 17:18:31.201026 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.200646 5052 cpu_manager.go:414] "containerMap: removing container" podUID="e9258d14-705c-40d3-8898-48d08dcf7846" containerName="maas-api" Oct 30 17:18:31.201026 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.200651 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9258d14-705c-40d3-8898-48d08dcf7846" containerName="maas-api" Oct 30 17:18:31.201026 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.200658 5052 cpu_manager.go:414] "containerMap: removing container" podUID="4963d7c3-cee3-430c-8e1c-156c79a75a3d" containerName="payload-processing" Oct 30 17:18:31.201026 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.200662 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4963d7c3-cee3-430c-8e1c-156c79a75a3d" containerName="payload-processing" Oct 30 17:18:31.201026 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.200668 5052 cpu_manager.go:414] "containerMap: removing container" podUID="666e4b84-2d87-4557-a8b5-10f890a1740b" containerName="istio-proxy" Oct 30 17:18:31.201026 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.200672 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="666e4b84-2d87-4557-a8b5-10f890a1740b" containerName="istio-proxy" Oct 30 17:18:31.201026 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.200678 5052 cpu_manager.go:414] "containerMap: removing container" podUID="1c5fe268-6035-4334-916e-b13bca739541" containerName="cleanup" Oct 30 17:18:31.201026 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.200683 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c5fe268-6035-4334-916e-b13bca739541" containerName="cleanup" Oct 30 17:18:31.201026 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.200690 5052 cpu_manager.go:414] "containerMap: removing container" podUID="d56798b3-ff19-4644-8d23-2967f523427d" containerName="maas-api" Oct 30 17:18:31.201026 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.200694 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d56798b3-ff19-4644-8d23-2967f523427d" containerName="maas-api" Oct 30 17:18:31.201026 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.200701 5052 cpu_manager.go:414] "containerMap: removing container" podUID="e98c90ff-9a9a-4e40-aeba-556d98be49a2" containerName="maas-api" Oct 30 17:18:31.201026 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.200705 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e98c90ff-9a9a-4e40-aeba-556d98be49a2" containerName="maas-api" Oct 30 17:18:31.201026 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.200711 5052 cpu_manager.go:414] "containerMap: removing container" podUID="3b174c28-3805-4b14-9547-cd003db3d27e" containerName="maas-api" Oct 30 17:18:31.201026 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.200715 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b174c28-3805-4b14-9547-cd003db3d27e" containerName="maas-api" Oct 30 17:18:31.201026 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.200721 5052 cpu_manager.go:414] "containerMap: removing container" podUID="cbb4905a-78c4-4697-84c0-bd1a665ff034" containerName="cleanup" Oct 30 17:18:31.201026 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.200725 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbb4905a-78c4-4697-84c0-bd1a665ff034" containerName="cleanup" Oct 30 17:18:31.201026 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.200730 5052 cpu_manager.go:414] "containerMap: removing container" podUID="7c8c2e2b-b5e5-4e6c-ace8-db859e693dde" containerName="cleanup" Oct 30 17:18:31.201026 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.200734 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c8c2e2b-b5e5-4e6c-ace8-db859e693dde" containerName="cleanup" Oct 30 17:18:31.201026 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.200742 5052 cpu_manager.go:414] "containerMap: removing container" podUID="c45190af-d795-4e13-8f2f-670b2c597658" containerName="main" Oct 30 17:18:31.201026 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.200749 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c45190af-d795-4e13-8f2f-670b2c597658" containerName="main" Oct 30 17:18:31.201026 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.200760 5052 cpu_manager.go:414] "containerMap: removing container" podUID="9da694ce-2f3b-465d-abe2-675a039d38c0" containerName="istio-proxy" Oct 30 17:18:31.201026 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.200766 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9da694ce-2f3b-465d-abe2-675a039d38c0" containerName="istio-proxy" Oct 30 17:18:31.201026 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.200775 5052 cpu_manager.go:414] "containerMap: removing container" podUID="0057b70d-3924-4e84-87f5-4cfc7e7babf5" containerName="istio-proxy" Oct 30 17:18:31.201026 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.200783 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0057b70d-3924-4e84-87f5-4cfc7e7babf5" containerName="istio-proxy" Oct 30 17:18:31.201026 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.200793 5052 cpu_manager.go:414] "containerMap: removing container" podUID="4bcebabf-2dba-4645-b30d-6280c85ca947" containerName="cleanup" Oct 30 17:18:31.201026 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.200801 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bcebabf-2dba-4645-b30d-6280c85ca947" containerName="cleanup" Oct 30 17:18:31.201026 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.200810 5052 cpu_manager.go:414] "containerMap: removing container" podUID="6136d90c-0253-4761-843b-f8a3922c93cf" containerName="payload-processing" Oct 30 17:18:31.201026 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.200815 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6136d90c-0253-4761-843b-f8a3922c93cf" containerName="payload-processing" Oct 30 17:18:31.201026 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.200822 5052 cpu_manager.go:414] "containerMap: removing container" podUID="9ccbae02-fb91-4bd2-9e3b-3aa09cd5a415" containerName="maas-api" Oct 30 17:18:31.201026 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.200827 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ccbae02-fb91-4bd2-9e3b-3aa09cd5a415" containerName="maas-api" Oct 30 17:18:31.201026 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.200833 5052 cpu_manager.go:414] "containerMap: removing container" podUID="7c8c2e2b-b5e5-4e6c-ace8-db859e693dde" containerName="cleanup" Oct 30 17:18:31.201026 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.200837 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c8c2e2b-b5e5-4e6c-ace8-db859e693dde" containerName="cleanup" Oct 30 17:18:31.201026 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.200846 5052 cpu_manager.go:414] "containerMap: removing container" podUID="c997fe94-1cf0-4316-826e-6bd921636715" containerName="manager" Oct 30 17:18:31.201026 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.200850 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c997fe94-1cf0-4316-826e-6bd921636715" containerName="manager" Oct 30 17:18:31.201026 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.200856 5052 cpu_manager.go:414] "containerMap: removing container" podUID="4bcebabf-2dba-4645-b30d-6280c85ca947" containerName="cleanup" Oct 30 17:18:31.201026 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.200860 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bcebabf-2dba-4645-b30d-6280c85ca947" containerName="cleanup" Oct 30 17:18:31.201026 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.200868 5052 cpu_manager.go:414] "containerMap: removing container" podUID="1c5fe268-6035-4334-916e-b13bca739541" containerName="cleanup" Oct 30 17:18:31.201026 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.200873 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c5fe268-6035-4334-916e-b13bca739541" containerName="cleanup" Oct 30 17:18:31.201026 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.200880 5052 cpu_manager.go:414] "containerMap: removing container" podUID="f0607bb9-cd62-4149-a429-4051390ad2a1" containerName="istio-proxy" Oct 30 17:18:31.201026 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.200885 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0607bb9-cd62-4149-a429-4051390ad2a1" containerName="istio-proxy" Oct 30 17:18:31.201026 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.200890 5052 cpu_manager.go:414] "containerMap: removing container" podUID="837038f3-5741-4b1f-8bb8-f8c8e234128f" containerName="istio-proxy" Oct 30 17:18:31.201026 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.200895 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="837038f3-5741-4b1f-8bb8-f8c8e234128f" containerName="istio-proxy" Oct 30 17:18:31.201026 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.200900 5052 cpu_manager.go:414] "containerMap: removing container" podUID="f571ed90-9a31-47d2-8c4b-9bc57a9975b4" containerName="istio-proxy" Oct 30 17:18:31.201026 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.200904 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f571ed90-9a31-47d2-8c4b-9bc57a9975b4" containerName="istio-proxy" Oct 30 17:18:31.201026 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.200910 5052 cpu_manager.go:414] "containerMap: removing container" podUID="24bfdf71-2df9-478f-a6af-eca34fc212ca" containerName="istio-proxy" Oct 30 17:18:31.201026 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.200915 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24bfdf71-2df9-478f-a6af-eca34fc212ca" containerName="istio-proxy" Oct 30 17:18:31.201026 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.200924 5052 cpu_manager.go:414] "containerMap: removing container" podUID="cbb4905a-78c4-4697-84c0-bd1a665ff034" containerName="cleanup" Oct 30 17:18:31.201026 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.200928 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbb4905a-78c4-4697-84c0-bd1a665ff034" containerName="cleanup" Oct 30 17:18:31.201026 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.200934 5052 cpu_manager.go:414] "containerMap: removing container" podUID="39f0bd01-e73f-4f31-a76e-ad6788799f3f" containerName="istio-proxy" Oct 30 17:18:31.201026 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.200938 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39f0bd01-e73f-4f31-a76e-ad6788799f3f" containerName="istio-proxy" Oct 30 17:18:31.201026 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.200946 5052 cpu_manager.go:414] "containerMap: removing container" podUID="0261ebf2-6dfd-4df5-b5de-4b7207ca761b" containerName="istio-proxy" Oct 30 17:18:31.201026 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.200950 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0261ebf2-6dfd-4df5-b5de-4b7207ca761b" containerName="istio-proxy" Oct 30 17:18:31.201026 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.200956 5052 cpu_manager.go:414] "containerMap: removing container" podUID="1c5fe268-6035-4334-916e-b13bca739541" containerName="cleanup" Oct 30 17:18:31.201026 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.200960 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c5fe268-6035-4334-916e-b13bca739541" containerName="cleanup" Oct 30 17:18:31.201026 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.200968 5052 cpu_manager.go:414] "containerMap: removing container" podUID="029cd866-9c51-45c8-9ca4-4c16578e12c9" containerName="istio-proxy" Oct 30 17:18:31.201026 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.200972 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="029cd866-9c51-45c8-9ca4-4c16578e12c9" containerName="istio-proxy" Oct 30 17:18:31.201026 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.200978 5052 cpu_manager.go:414] "containerMap: removing container" podUID="029cd866-9c51-45c8-9ca4-4c16578e12c9" containerName="istio-proxy" Oct 30 17:18:31.201026 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.200982 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="029cd866-9c51-45c8-9ca4-4c16578e12c9" containerName="istio-proxy" Oct 30 17:18:31.201026 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.200988 5052 cpu_manager.go:414] "containerMap: removing container" podUID="8054915f-a3d1-44d0-ab47-4ca0161255da" containerName="payload-pre-processing" Oct 30 17:18:31.201026 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.200993 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8054915f-a3d1-44d0-ab47-4ca0161255da" containerName="payload-pre-processing" Oct 30 17:18:31.201026 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.201001 5052 cpu_manager.go:414] "containerMap: removing container" podUID="b4d2755a-efbb-41f5-983f-309ee1cff113" containerName="payload-pre-processing" Oct 30 17:18:31.201026 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.201005 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4d2755a-efbb-41f5-983f-309ee1cff113" containerName="payload-pre-processing" Oct 30 17:18:31.201026 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.201014 5052 cpu_manager.go:414] "containerMap: removing container" podUID="0f58433e-03f7-4e7c-95b5-3d9871cce2f5" containerName="maas-api" Oct 30 17:18:31.201026 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.201018 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f58433e-03f7-4e7c-95b5-3d9871cce2f5" containerName="maas-api" Oct 30 17:18:31.201026 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.201027 5052 cpu_manager.go:414] "containerMap: removing container" podUID="e68b78bc-50ac-43ee-8354-a11b921e2713" containerName="maas-api" Oct 30 17:18:31.201026 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.201032 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e68b78bc-50ac-43ee-8354-a11b921e2713" containerName="maas-api" Oct 30 17:18:31.201026 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.201038 5052 cpu_manager.go:414] "containerMap: removing container" podUID="b774d2ea-e597-4122-b520-7ccdec91972f" containerName="payload-processing" Oct 30 17:18:31.201026 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.201043 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b774d2ea-e597-4122-b520-7ccdec91972f" containerName="payload-processing" Oct 30 17:18:31.201026 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.201050 5052 cpu_manager.go:414] "containerMap: removing container" podUID="2ba3e877-170c-44be-aad2-c96e7cf6b99f" containerName="payload-processing" Oct 30 17:18:31.201026 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.201055 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ba3e877-170c-44be-aad2-c96e7cf6b99f" containerName="payload-processing" Oct 30 17:18:31.201026 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.201061 5052 cpu_manager.go:414] "containerMap: removing container" podUID="d7adb9b2-ac3c-440c-8338-f0a4375625d3" containerName="maas-api" Oct 30 17:18:31.203426 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.201066 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7adb9b2-ac3c-440c-8338-f0a4375625d3" containerName="maas-api" Oct 30 17:18:31.203426 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.201075 5052 cpu_manager.go:414] "containerMap: removing container" podUID="c8947ffd-7539-43d8-86d5-fc17dd8c7d6e" containerName="istio-proxy" Oct 30 17:18:31.203426 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.201080 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8947ffd-7539-43d8-86d5-fc17dd8c7d6e" containerName="istio-proxy" Oct 30 17:18:31.203426 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.201089 5052 cpu_manager.go:414] "containerMap: removing container" podUID="552a6bdf-c418-47bd-b72b-20e311760c33" containerName="maas-api" Oct 30 17:18:31.203426 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.201093 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="552a6bdf-c418-47bd-b72b-20e311760c33" containerName="maas-api" Oct 30 17:18:31.203426 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.201100 5052 cpu_manager.go:414] "containerMap: removing container" podUID="d6c3dee8-d01e-445d-b8a4-96b7f57493ec" containerName="istio-proxy" Oct 30 17:18:31.203426 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.201104 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6c3dee8-d01e-445d-b8a4-96b7f57493ec" containerName="istio-proxy" Oct 30 17:18:31.203426 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.201110 5052 cpu_manager.go:414] "containerMap: removing container" podUID="d6c3dee8-d01e-445d-b8a4-96b7f57493ec" containerName="istio-proxy" Oct 30 17:18:31.203426 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.201115 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6c3dee8-d01e-445d-b8a4-96b7f57493ec" containerName="istio-proxy" Oct 30 17:18:31.203426 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.201182 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="4963d7c3-cee3-430c-8e1c-156c79a75a3d" containerName="payload-processing" Oct 30 17:18:31.203426 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.201191 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="666e4b84-2d87-4557-a8b5-10f890a1740b" containerName="istio-proxy" Oct 30 17:18:31.203426 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.201196 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c5fe268-6035-4334-916e-b13bca739541" containerName="cleanup" Oct 30 17:18:31.203426 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.201202 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="d56798b3-ff19-4644-8d23-2967f523427d" containerName="maas-api" Oct 30 17:18:31.203426 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.201207 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="e98c90ff-9a9a-4e40-aeba-556d98be49a2" containerName="maas-api" Oct 30 17:18:31.203426 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.201213 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b174c28-3805-4b14-9547-cd003db3d27e" containerName="maas-api" Oct 30 17:18:31.203426 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.201218 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbb4905a-78c4-4697-84c0-bd1a665ff034" containerName="cleanup" Oct 30 17:18:31.203426 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.201222 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c8c2e2b-b5e5-4e6c-ace8-db859e693dde" containerName="cleanup" Oct 30 17:18:31.203426 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.201227 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="c45190af-d795-4e13-8f2f-670b2c597658" containerName="main" Oct 30 17:18:31.203426 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.201232 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="9da694ce-2f3b-465d-abe2-675a039d38c0" containerName="istio-proxy" Oct 30 17:18:31.203426 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.201238 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="0057b70d-3924-4e84-87f5-4cfc7e7babf5" containerName="istio-proxy" Oct 30 17:18:31.203426 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.201243 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bcebabf-2dba-4645-b30d-6280c85ca947" containerName="cleanup" Oct 30 17:18:31.203426 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.201248 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="6136d90c-0253-4761-843b-f8a3922c93cf" containerName="payload-processing" Oct 30 17:18:31.203426 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.201254 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ccbae02-fb91-4bd2-9e3b-3aa09cd5a415" containerName="maas-api" Oct 30 17:18:31.203426 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.201259 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c8c2e2b-b5e5-4e6c-ace8-db859e693dde" containerName="cleanup" Oct 30 17:18:31.203426 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.201264 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="505e6f2b-634b-40ae-b5db-6ced8e1bf322" containerName="istio-proxy" Oct 30 17:18:31.203426 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.201270 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="c997fe94-1cf0-4316-826e-6bd921636715" containerName="manager" Oct 30 17:18:31.203426 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.201274 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="666e4b84-2d87-4557-a8b5-10f890a1740b" containerName="istio-proxy" Oct 30 17:18:31.203426 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.201281 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c5fe268-6035-4334-916e-b13bca739541" containerName="cleanup" Oct 30 17:18:31.203426 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.201286 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0607bb9-cd62-4149-a429-4051390ad2a1" containerName="istio-proxy" Oct 30 17:18:31.203426 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.201292 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="837038f3-5741-4b1f-8bb8-f8c8e234128f" containerName="istio-proxy" Oct 30 17:18:31.203426 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.201297 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="f571ed90-9a31-47d2-8c4b-9bc57a9975b4" containerName="istio-proxy" Oct 30 17:18:31.203426 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.201302 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="24bfdf71-2df9-478f-a6af-eca34fc212ca" containerName="istio-proxy" Oct 30 17:18:31.203426 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.201306 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbb4905a-78c4-4697-84c0-bd1a665ff034" containerName="cleanup" Oct 30 17:18:31.203426 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.201313 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="39f0bd01-e73f-4f31-a76e-ad6788799f3f" containerName="istio-proxy" Oct 30 17:18:31.203426 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.201318 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="0261ebf2-6dfd-4df5-b5de-4b7207ca761b" containerName="istio-proxy" Oct 30 17:18:31.203426 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.201323 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="029cd866-9c51-45c8-9ca4-4c16578e12c9" containerName="istio-proxy" Oct 30 17:18:31.203426 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.201330 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="029cd866-9c51-45c8-9ca4-4c16578e12c9" containerName="istio-proxy" Oct 30 17:18:31.203426 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.201336 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="8054915f-a3d1-44d0-ab47-4ca0161255da" containerName="payload-pre-processing" Oct 30 17:18:31.203426 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.201342 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4d2755a-efbb-41f5-983f-309ee1cff113" containerName="payload-pre-processing" Oct 30 17:18:31.203426 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.201348 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f58433e-03f7-4e7c-95b5-3d9871cce2f5" containerName="maas-api" Oct 30 17:18:31.203426 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.201352 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="e68b78bc-50ac-43ee-8354-a11b921e2713" containerName="maas-api" Oct 30 17:18:31.203426 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.201357 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbb4905a-78c4-4697-84c0-bd1a665ff034" containerName="cleanup" Oct 30 17:18:31.203426 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.201362 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="b774d2ea-e597-4122-b520-7ccdec91972f" containerName="payload-processing" Oct 30 17:18:31.203426 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.201368 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ba3e877-170c-44be-aad2-c96e7cf6b99f" containerName="payload-processing" Oct 30 17:18:31.203426 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.201374 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="837038f3-5741-4b1f-8bb8-f8c8e234128f" containerName="istio-proxy" Oct 30 17:18:31.203426 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.201379 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7adb9b2-ac3c-440c-8338-f0a4375625d3" containerName="maas-api" Oct 30 17:18:31.203426 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.201384 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8947ffd-7539-43d8-86d5-fc17dd8c7d6e" containerName="istio-proxy" Oct 30 17:18:31.203426 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.201389 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bcebabf-2dba-4645-b30d-6280c85ca947" containerName="cleanup" Oct 30 17:18:31.203426 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.201394 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="552a6bdf-c418-47bd-b72b-20e311760c33" containerName="maas-api" Oct 30 17:18:31.203426 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.201400 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6c3dee8-d01e-445d-b8a4-96b7f57493ec" containerName="istio-proxy" Oct 30 17:18:31.203426 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.201405 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6c3dee8-d01e-445d-b8a4-96b7f57493ec" containerName="istio-proxy" Oct 30 17:18:31.203426 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.201411 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8eabbfd-4dce-4087-a350-c55cb7c7baf3" containerName="istio-proxy" Oct 30 17:18:31.203426 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.201416 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4036d90-5e91-477e-b81f-91c2dceee6c9" containerName="istio-proxy" Oct 30 17:18:31.203426 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.201421 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="15ed43ef-5cdd-489a-a70e-063239822ff3" containerName="main" Oct 30 17:18:31.203426 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.201426 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e6b7be8-b362-4477-86b5-cf8ff0cd266f" containerName="payload-pre-processing" Oct 30 17:18:31.203426 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.201430 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9258d14-705c-40d3-8898-48d08dcf7846" containerName="maas-api" Oct 30 17:18:31.205829 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.205812 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-575c7b5c5b-f64sn" Oct 30 17:18:31.222132 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.222113 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 30 17:18:31.235330 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.235305 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-gtz2x\"" Oct 30 17:18:31.235465 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.235375 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 30 17:18:31.235465 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.235453 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 30 17:18:31.237516 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.237488 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-575c7b5c5b-f64sn"] Oct 30 17:18:31.272277 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.272247 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a0c1295c-b9bb-4056-b5c9-ea94169d47e5-webhook-cert\") pod \"maas-controller-575c7b5c5b-f64sn\" (UID: \"a0c1295c-b9bb-4056-b5c9-ea94169d47e5\") " pod="opendatahub/maas-controller-575c7b5c5b-f64sn" Oct 30 17:18:31.272426 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.272284 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tq7px\" (UniqueName: \"kubernetes.io/projected/a0c1295c-b9bb-4056-b5c9-ea94169d47e5-kube-api-access-tq7px\") pod \"maas-controller-575c7b5c5b-f64sn\" (UID: \"a0c1295c-b9bb-4056-b5c9-ea94169d47e5\") " pod="opendatahub/maas-controller-575c7b5c5b-f64sn" Oct 30 17:18:31.272500 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.272463 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0c1295c-b9bb-4056-b5c9-ea94169d47e5-metrics-certs\") pod \"maas-controller-575c7b5c5b-f64sn\" (UID: \"a0c1295c-b9bb-4056-b5c9-ea94169d47e5\") " pod="opendatahub/maas-controller-575c7b5c5b-f64sn" Oct 30 17:18:31.373528 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.373494 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tq7px\" (UniqueName: \"kubernetes.io/projected/a0c1295c-b9bb-4056-b5c9-ea94169d47e5-kube-api-access-tq7px\") pod \"maas-controller-575c7b5c5b-f64sn\" (UID: \"a0c1295c-b9bb-4056-b5c9-ea94169d47e5\") " pod="opendatahub/maas-controller-575c7b5c5b-f64sn" Oct 30 17:18:31.373707 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.373557 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0c1295c-b9bb-4056-b5c9-ea94169d47e5-metrics-certs\") pod \"maas-controller-575c7b5c5b-f64sn\" (UID: \"a0c1295c-b9bb-4056-b5c9-ea94169d47e5\") " pod="opendatahub/maas-controller-575c7b5c5b-f64sn" Oct 30 17:18:31.373707 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.373657 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a0c1295c-b9bb-4056-b5c9-ea94169d47e5-webhook-cert\") pod \"maas-controller-575c7b5c5b-f64sn\" (UID: \"a0c1295c-b9bb-4056-b5c9-ea94169d47e5\") " pod="opendatahub/maas-controller-575c7b5c5b-f64sn" Oct 30 17:18:31.376059 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.376035 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0c1295c-b9bb-4056-b5c9-ea94169d47e5-metrics-certs\") pod \"maas-controller-575c7b5c5b-f64sn\" (UID: \"a0c1295c-b9bb-4056-b5c9-ea94169d47e5\") " pod="opendatahub/maas-controller-575c7b5c5b-f64sn" Oct 30 17:18:31.376149 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.376122 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a0c1295c-b9bb-4056-b5c9-ea94169d47e5-webhook-cert\") pod \"maas-controller-575c7b5c5b-f64sn\" (UID: \"a0c1295c-b9bb-4056-b5c9-ea94169d47e5\") " pod="opendatahub/maas-controller-575c7b5c5b-f64sn" Oct 30 17:18:31.398222 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.398199 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tq7px\" (UniqueName: \"kubernetes.io/projected/a0c1295c-b9bb-4056-b5c9-ea94169d47e5-kube-api-access-tq7px\") pod \"maas-controller-575c7b5c5b-f64sn\" (UID: \"a0c1295c-b9bb-4056-b5c9-ea94169d47e5\") " pod="opendatahub/maas-controller-575c7b5c5b-f64sn" Oct 30 17:18:31.519687 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.519605 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-575c7b5c5b-f64sn" Oct 30 17:18:31.820525 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.820345 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-575c7b5c5b-f64sn"] Oct 30 17:18:31.822988 ip-10-0-131-136 kubenswrapper[5052]: W1030 17:18:31.822957 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0c1295c_b9bb_4056_b5c9_ea94169d47e5.slice/crio-14ae2ad1b73c5a36a053f1ce5d6b6a3ac44b237f546d7c436ce94babbdf70eaa WatchSource:0}: Error finding container 14ae2ad1b73c5a36a053f1ce5d6b6a3ac44b237f546d7c436ce94babbdf70eaa: Status 404 returned error can't find the container with id 14ae2ad1b73c5a36a053f1ce5d6b6a3ac44b237f546d7c436ce94babbdf70eaa Oct 30 17:18:31.824296 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:31.824279 5052 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 17:18:32.483721 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:32.483681 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-575c7b5c5b-f64sn" event={"ID":"a0c1295c-b9bb-4056-b5c9-ea94169d47e5","Type":"ContainerStarted","Data":"06956a75e919ebef5178713769eb881776407ec7cbcacc858346cbac2061a860"} Oct 30 17:18:32.483871 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:32.483742 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-575c7b5c5b-f64sn" event={"ID":"a0c1295c-b9bb-4056-b5c9-ea94169d47e5","Type":"ContainerStarted","Data":"14ae2ad1b73c5a36a053f1ce5d6b6a3ac44b237f546d7c436ce94babbdf70eaa"} Oct 30 17:18:32.483871 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:32.483783 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-575c7b5c5b-f64sn" Oct 30 17:18:32.530157 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:32.530109 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-575c7b5c5b-f64sn" podStartSLOduration=1.050532584 podStartE2EDuration="1.53009536s" totalImagesPullingTime="479.562776ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:18:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:18:32.528376295 +0000 UTC m=+3468.055711528" watchObservedRunningTime="2026-10-30 17:18:32.53009536 +0000 UTC m=+3468.057430579" Oct 30 17:18:43.490704 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:43.490670 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-575c7b5c5b-f64sn" Oct 30 17:18:48.843346 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:18:48.843304 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 17:18:54.078636 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:54.078603 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-flvdc"] Oct 30 17:18:54.079080 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:54.079066 5052 cpu_manager.go:414] "containerMap: removing container" podUID="505e6f2b-634b-40ae-b5db-6ced8e1bf322" containerName="istio-proxy" Oct 30 17:18:54.079133 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:54.079081 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="505e6f2b-634b-40ae-b5db-6ced8e1bf322" containerName="istio-proxy" Oct 30 17:18:54.079133 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:54.079096 5052 cpu_manager.go:414] "containerMap: removing container" podUID="666e4b84-2d87-4557-a8b5-10f890a1740b" containerName="istio-proxy" Oct 30 17:18:54.079133 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:54.079101 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="666e4b84-2d87-4557-a8b5-10f890a1740b" containerName="istio-proxy" Oct 30 17:18:54.079133 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:54.079119 5052 cpu_manager.go:414] "containerMap: removing container" podUID="4bcebabf-2dba-4645-b30d-6280c85ca947" containerName="cleanup" Oct 30 17:18:54.079133 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:54.079127 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bcebabf-2dba-4645-b30d-6280c85ca947" containerName="cleanup" Oct 30 17:18:54.079338 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:54.079137 5052 cpu_manager.go:414] "containerMap: removing container" podUID="837038f3-5741-4b1f-8bb8-f8c8e234128f" containerName="istio-proxy" Oct 30 17:18:54.079338 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:54.079141 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="837038f3-5741-4b1f-8bb8-f8c8e234128f" containerName="istio-proxy" Oct 30 17:18:54.079338 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:54.079150 5052 cpu_manager.go:414] "containerMap: removing container" podUID="cbb4905a-78c4-4697-84c0-bd1a665ff034" containerName="cleanup" Oct 30 17:18:54.079338 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:54.079155 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbb4905a-78c4-4697-84c0-bd1a665ff034" containerName="cleanup" Oct 30 17:18:54.079338 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:54.079220 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="505e6f2b-634b-40ae-b5db-6ced8e1bf322" containerName="istio-proxy" Oct 30 17:18:54.079338 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:54.079229 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c5fe268-6035-4334-916e-b13bca739541" containerName="cleanup" Oct 30 17:18:54.079338 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:54.079238 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="0261ebf2-6dfd-4df5-b5de-4b7207ca761b" containerName="istio-proxy" Oct 30 17:18:54.079338 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:54.079248 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bcebabf-2dba-4645-b30d-6280c85ca947" containerName="cleanup" Oct 30 17:18:54.079338 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:54.079257 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c8c2e2b-b5e5-4e6c-ace8-db859e693dde" containerName="cleanup" Oct 30 17:18:54.083857 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:54.083835 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-flvdc" Oct 30 17:18:54.116591 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:54.116554 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-flvdc"] Oct 30 17:18:54.194412 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:54.194379 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6r6s4\" (UniqueName: \"kubernetes.io/projected/42edda16-6400-45ef-bce3-a5ce362045ed-kube-api-access-6r6s4\") pod \"maas-default-gateway-openshift-default-76f4876667-flvdc\" (UID: \"42edda16-6400-45ef-bce3-a5ce362045ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-flvdc" Oct 30 17:18:54.194576 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:54.194437 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42edda16-6400-45ef-bce3-a5ce362045ed-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-flvdc\" (UID: \"42edda16-6400-45ef-bce3-a5ce362045ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-flvdc" Oct 30 17:18:54.194576 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:54.194525 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42edda16-6400-45ef-bce3-a5ce362045ed-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-flvdc\" (UID: \"42edda16-6400-45ef-bce3-a5ce362045ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-flvdc" Oct 30 17:18:54.194576 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:54.194564 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42edda16-6400-45ef-bce3-a5ce362045ed-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-flvdc\" (UID: \"42edda16-6400-45ef-bce3-a5ce362045ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-flvdc" Oct 30 17:18:54.194764 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:54.194643 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42edda16-6400-45ef-bce3-a5ce362045ed-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-flvdc\" (UID: \"42edda16-6400-45ef-bce3-a5ce362045ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-flvdc" Oct 30 17:18:54.194764 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:54.194678 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42edda16-6400-45ef-bce3-a5ce362045ed-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-flvdc\" (UID: \"42edda16-6400-45ef-bce3-a5ce362045ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-flvdc" Oct 30 17:18:54.194764 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:54.194699 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42edda16-6400-45ef-bce3-a5ce362045ed-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-flvdc\" (UID: \"42edda16-6400-45ef-bce3-a5ce362045ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-flvdc" Oct 30 17:18:54.194764 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:54.194755 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42edda16-6400-45ef-bce3-a5ce362045ed-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-flvdc\" (UID: \"42edda16-6400-45ef-bce3-a5ce362045ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-flvdc" Oct 30 17:18:54.194939 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:54.194829 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42edda16-6400-45ef-bce3-a5ce362045ed-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-flvdc\" (UID: \"42edda16-6400-45ef-bce3-a5ce362045ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-flvdc" Oct 30 17:18:54.295734 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:54.295696 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42edda16-6400-45ef-bce3-a5ce362045ed-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-flvdc\" (UID: \"42edda16-6400-45ef-bce3-a5ce362045ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-flvdc" Oct 30 17:18:54.295734 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:54.295734 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42edda16-6400-45ef-bce3-a5ce362045ed-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-flvdc\" (UID: \"42edda16-6400-45ef-bce3-a5ce362045ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-flvdc" Oct 30 17:18:54.296011 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:54.295754 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42edda16-6400-45ef-bce3-a5ce362045ed-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-flvdc\" (UID: \"42edda16-6400-45ef-bce3-a5ce362045ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-flvdc" Oct 30 17:18:54.296011 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:54.295795 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6r6s4\" (UniqueName: \"kubernetes.io/projected/42edda16-6400-45ef-bce3-a5ce362045ed-kube-api-access-6r6s4\") pod \"maas-default-gateway-openshift-default-76f4876667-flvdc\" (UID: \"42edda16-6400-45ef-bce3-a5ce362045ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-flvdc" Oct 30 17:18:54.296011 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:54.295836 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42edda16-6400-45ef-bce3-a5ce362045ed-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-flvdc\" (UID: \"42edda16-6400-45ef-bce3-a5ce362045ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-flvdc" Oct 30 17:18:54.296011 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:54.295879 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42edda16-6400-45ef-bce3-a5ce362045ed-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-flvdc\" (UID: \"42edda16-6400-45ef-bce3-a5ce362045ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-flvdc" Oct 30 17:18:54.296011 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:54.295917 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42edda16-6400-45ef-bce3-a5ce362045ed-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-flvdc\" (UID: \"42edda16-6400-45ef-bce3-a5ce362045ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-flvdc" Oct 30 17:18:54.296011 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:54.295946 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42edda16-6400-45ef-bce3-a5ce362045ed-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-flvdc\" (UID: \"42edda16-6400-45ef-bce3-a5ce362045ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-flvdc" Oct 30 17:18:54.296011 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:54.295981 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42edda16-6400-45ef-bce3-a5ce362045ed-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-flvdc\" (UID: \"42edda16-6400-45ef-bce3-a5ce362045ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-flvdc" Oct 30 17:18:54.296348 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:54.296171 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42edda16-6400-45ef-bce3-a5ce362045ed-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-flvdc\" (UID: \"42edda16-6400-45ef-bce3-a5ce362045ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-flvdc" Oct 30 17:18:54.296348 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:54.296263 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42edda16-6400-45ef-bce3-a5ce362045ed-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-flvdc\" (UID: \"42edda16-6400-45ef-bce3-a5ce362045ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-flvdc" Oct 30 17:18:54.296449 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:54.296405 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42edda16-6400-45ef-bce3-a5ce362045ed-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-flvdc\" (UID: \"42edda16-6400-45ef-bce3-a5ce362045ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-flvdc" Oct 30 17:18:54.296449 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:54.296431 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42edda16-6400-45ef-bce3-a5ce362045ed-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-flvdc\" (UID: \"42edda16-6400-45ef-bce3-a5ce362045ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-flvdc" Oct 30 17:18:54.296559 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:54.296539 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42edda16-6400-45ef-bce3-a5ce362045ed-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-flvdc\" (UID: \"42edda16-6400-45ef-bce3-a5ce362045ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-flvdc" Oct 30 17:18:54.298108 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:54.298083 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42edda16-6400-45ef-bce3-a5ce362045ed-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-flvdc\" (UID: \"42edda16-6400-45ef-bce3-a5ce362045ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-flvdc" Oct 30 17:18:54.298444 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:54.298425 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42edda16-6400-45ef-bce3-a5ce362045ed-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-flvdc\" (UID: \"42edda16-6400-45ef-bce3-a5ce362045ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-flvdc" Oct 30 17:18:54.320421 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:54.320394 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42edda16-6400-45ef-bce3-a5ce362045ed-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-flvdc\" (UID: \"42edda16-6400-45ef-bce3-a5ce362045ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-flvdc" Oct 30 17:18:54.320545 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:54.320505 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6r6s4\" (UniqueName: \"kubernetes.io/projected/42edda16-6400-45ef-bce3-a5ce362045ed-kube-api-access-6r6s4\") pod \"maas-default-gateway-openshift-default-76f4876667-flvdc\" (UID: \"42edda16-6400-45ef-bce3-a5ce362045ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-flvdc" Oct 30 17:18:54.399141 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:54.399120 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-flvdc" Oct 30 17:18:54.714386 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:54.714355 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-flvdc"] Oct 30 17:18:54.717146 ip-10-0-131-136 kubenswrapper[5052]: W1030 17:18:54.717119 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42edda16_6400_45ef_bce3_a5ce362045ed.slice/crio-1f1bdf20ba5370ee95a254fb9ada80b84cfe8cd4c5b96387b84ece3d1c700ac5 WatchSource:0}: Error finding container 1f1bdf20ba5370ee95a254fb9ada80b84cfe8cd4c5b96387b84ece3d1c700ac5: Status 404 returned error can't find the container with id 1f1bdf20ba5370ee95a254fb9ada80b84cfe8cd4c5b96387b84ece3d1c700ac5 Oct 30 17:18:54.719253 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:54.719221 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:18:54.719312 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:54.719287 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:18:54.719359 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:54.719316 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:18:55.605182 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:55.605140 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-flvdc" event={"ID":"42edda16-6400-45ef-bce3-a5ce362045ed","Type":"ContainerStarted","Data":"be1a58141492dfed2b72217076bc6e1c6fe7165d79543555ea496726d3ebf6f8"} Oct 30 17:18:55.605619 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:55.605197 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-flvdc" event={"ID":"42edda16-6400-45ef-bce3-a5ce362045ed","Type":"ContainerStarted","Data":"1f1bdf20ba5370ee95a254fb9ada80b84cfe8cd4c5b96387b84ece3d1c700ac5"} Oct 30 17:18:55.675952 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:55.675887 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-flvdc" podStartSLOduration=1.6758729209999998 podStartE2EDuration="1.675872921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:18:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:18:55.674266181 +0000 UTC m=+3491.201601401" watchObservedRunningTime="2026-10-30 17:18:55.675872921 +0000 UTC m=+3491.203208134" Oct 30 17:18:56.400178 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:56.400135 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-flvdc" Oct 30 17:18:56.401703 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:56.401673 5052 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-flvdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body= Oct 30 17:18:56.401842 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:56.401724 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-flvdc" podUID="42edda16-6400-45ef-bce3-a5ce362045ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" Oct 30 17:18:57.399640 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:57.399603 5052 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-flvdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body= Oct 30 17:18:57.400026 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:57.399659 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-flvdc" podUID="42edda16-6400-45ef-bce3-a5ce362045ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" Oct 30 17:18:58.399887 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:58.399848 5052 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-flvdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body= Oct 30 17:18:58.400279 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:58.399916 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-flvdc" podUID="42edda16-6400-45ef-bce3-a5ce362045ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" Oct 30 17:18:59.400410 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:59.400377 5052 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-flvdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body= Oct 30 17:18:59.400827 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:18:59.400442 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-flvdc" podUID="42edda16-6400-45ef-bce3-a5ce362045ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" Oct 30 17:19:00.400219 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:19:00.400183 5052 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-flvdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body= Oct 30 17:19:00.400472 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:19:00.400239 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-flvdc" podUID="42edda16-6400-45ef-bce3-a5ce362045ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" Oct 30 17:19:01.400543 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:19:01.400499 5052 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-flvdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body= Oct 30 17:19:01.400953 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:19:01.400609 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-flvdc" podUID="42edda16-6400-45ef-bce3-a5ce362045ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" Oct 30 17:19:03.142522 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:19:03.142480 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-flvdc" Oct 30 17:19:03.143011 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:19:03.142923 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-flvdc" Oct 30 17:19:03.143983 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:19:03.143960 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-flvdc" Oct 30 17:19:12.441831 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:19:12.441793 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-575c7b5c5b-f64sn"] Oct 30 17:19:12.442359 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:19:12.442014 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-575c7b5c5b-f64sn" podUID="a0c1295c-b9bb-4056-b5c9-ea94169d47e5" containerName="manager" containerID="cri-o://06956a75e919ebef5178713769eb881776407ec7cbcacc858346cbac2061a860" gracePeriod=30 Oct 30 17:19:12.690809 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:19:12.690781 5052 generic.go:356] "Generic (PLEG): container finished" podID="a0c1295c-b9bb-4056-b5c9-ea94169d47e5" containerID="06956a75e919ebef5178713769eb881776407ec7cbcacc858346cbac2061a860" exitCode=0 Oct 30 17:19:12.690939 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:19:12.690854 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-575c7b5c5b-f64sn" event={"ID":"a0c1295c-b9bb-4056-b5c9-ea94169d47e5","Type":"ContainerDied","Data":"06956a75e919ebef5178713769eb881776407ec7cbcacc858346cbac2061a860"} Oct 30 17:19:12.690939 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:19:12.690916 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-575c7b5c5b-f64sn" event={"ID":"a0c1295c-b9bb-4056-b5c9-ea94169d47e5","Type":"ContainerDied","Data":"14ae2ad1b73c5a36a053f1ce5d6b6a3ac44b237f546d7c436ce94babbdf70eaa"} Oct 30 17:19:12.691028 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:19:12.690940 5052 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="14ae2ad1b73c5a36a053f1ce5d6b6a3ac44b237f546d7c436ce94babbdf70eaa" Oct 30 17:19:12.699680 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:19:12.699626 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-575c7b5c5b-f64sn" Oct 30 17:19:12.778764 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:19:12.778724 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0c1295c-b9bb-4056-b5c9-ea94169d47e5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0c1295c-b9bb-4056-b5c9-ea94169d47e5-metrics-certs\") pod \"a0c1295c-b9bb-4056-b5c9-ea94169d47e5\" (UID: \"a0c1295c-b9bb-4056-b5c9-ea94169d47e5\") " Oct 30 17:19:12.778945 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:19:12.778780 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0c1295c-b9bb-4056-b5c9-ea94169d47e5-kube-api-access-tq7px\" (UniqueName: \"kubernetes.io/projected/a0c1295c-b9bb-4056-b5c9-ea94169d47e5-kube-api-access-tq7px\") pod \"a0c1295c-b9bb-4056-b5c9-ea94169d47e5\" (UID: \"a0c1295c-b9bb-4056-b5c9-ea94169d47e5\") " Oct 30 17:19:12.778945 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:19:12.778830 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0c1295c-b9bb-4056-b5c9-ea94169d47e5-webhook-cert\" (UniqueName: \"kubernetes.io/secret/a0c1295c-b9bb-4056-b5c9-ea94169d47e5-webhook-cert\") pod \"a0c1295c-b9bb-4056-b5c9-ea94169d47e5\" (UID: \"a0c1295c-b9bb-4056-b5c9-ea94169d47e5\") " Oct 30 17:19:12.781281 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:19:12.781251 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0c1295c-b9bb-4056-b5c9-ea94169d47e5-metrics-certs" pod "a0c1295c-b9bb-4056-b5c9-ea94169d47e5" (UID: "a0c1295c-b9bb-4056-b5c9-ea94169d47e5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:19:12.781411 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:19:12.781317 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0c1295c-b9bb-4056-b5c9-ea94169d47e5-kube-api-access-tq7px" pod "a0c1295c-b9bb-4056-b5c9-ea94169d47e5" (UID: "a0c1295c-b9bb-4056-b5c9-ea94169d47e5"). InnerVolumeSpecName "kube-api-access-tq7px". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:19:12.781706 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:19:12.781676 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0c1295c-b9bb-4056-b5c9-ea94169d47e5-webhook-cert" pod "a0c1295c-b9bb-4056-b5c9-ea94169d47e5" (UID: "a0c1295c-b9bb-4056-b5c9-ea94169d47e5"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:19:12.880167 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:19:12.880128 5052 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0c1295c-b9bb-4056-b5c9-ea94169d47e5-metrics-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:19:12.880167 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:19:12.880159 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tq7px\" (UniqueName: \"kubernetes.io/projected/a0c1295c-b9bb-4056-b5c9-ea94169d47e5-kube-api-access-tq7px\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:19:12.880167 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:19:12.880169 5052 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a0c1295c-b9bb-4056-b5c9-ea94169d47e5-webhook-cert\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:19:13.696136 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:19:13.696107 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-575c7b5c5b-f64sn" Oct 30 17:19:13.763672 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:19:13.763643 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-575c7b5c5b-f64sn"] Oct 30 17:19:13.776445 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:19:13.776420 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-575c7b5c5b-f64sn"] Oct 30 17:19:14.595945 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:19:14.595908 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0c1295c-b9bb-4056-b5c9-ea94169d47e5" path="/var/lib/kubelet/pods/a0c1295c-b9bb-4056-b5c9-ea94169d47e5/volumes" Oct 30 17:19:14.967802 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:19:14.967768 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-575c7b5c5b-p8bxg"] Oct 30 17:19:14.968323 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:19:14.968296 5052 cpu_manager.go:414] "containerMap: removing container" podUID="a0c1295c-b9bb-4056-b5c9-ea94169d47e5" containerName="manager" Oct 30 17:19:14.968425 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:19:14.968324 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0c1295c-b9bb-4056-b5c9-ea94169d47e5" containerName="manager" Oct 30 17:19:14.968472 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:19:14.968431 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0c1295c-b9bb-4056-b5c9-ea94169d47e5" containerName="manager" Oct 30 17:19:14.971746 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:19:14.971730 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-575c7b5c5b-p8bxg" Oct 30 17:19:15.000116 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:19:15.000090 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tz2ln\" (UniqueName: \"kubernetes.io/projected/44aa1bfa-6067-45b4-ada0-6ea89af3a587-kube-api-access-tz2ln\") pod \"maas-controller-575c7b5c5b-p8bxg\" (UID: \"44aa1bfa-6067-45b4-ada0-6ea89af3a587\") " pod="opendatahub/maas-controller-575c7b5c5b-p8bxg" Oct 30 17:19:15.000264 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:19:15.000132 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/44aa1bfa-6067-45b4-ada0-6ea89af3a587-webhook-cert\") pod \"maas-controller-575c7b5c5b-p8bxg\" (UID: \"44aa1bfa-6067-45b4-ada0-6ea89af3a587\") " pod="opendatahub/maas-controller-575c7b5c5b-p8bxg" Oct 30 17:19:15.000355 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:19:15.000339 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44aa1bfa-6067-45b4-ada0-6ea89af3a587-metrics-certs\") pod \"maas-controller-575c7b5c5b-p8bxg\" (UID: \"44aa1bfa-6067-45b4-ada0-6ea89af3a587\") " pod="opendatahub/maas-controller-575c7b5c5b-p8bxg" Oct 30 17:19:15.003055 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:19:15.003035 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 30 17:19:15.003137 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:19:15.003065 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 30 17:19:15.003249 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:19:15.003234 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 30 17:19:15.003293 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:19:15.003266 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-gtz2x\"" Oct 30 17:19:15.005756 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:19:15.005738 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-575c7b5c5b-p8bxg"] Oct 30 17:19:15.101065 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:19:15.101037 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44aa1bfa-6067-45b4-ada0-6ea89af3a587-metrics-certs\") pod \"maas-controller-575c7b5c5b-p8bxg\" (UID: \"44aa1bfa-6067-45b4-ada0-6ea89af3a587\") " pod="opendatahub/maas-controller-575c7b5c5b-p8bxg" Oct 30 17:19:15.101200 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:19:15.101076 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tz2ln\" (UniqueName: \"kubernetes.io/projected/44aa1bfa-6067-45b4-ada0-6ea89af3a587-kube-api-access-tz2ln\") pod \"maas-controller-575c7b5c5b-p8bxg\" (UID: \"44aa1bfa-6067-45b4-ada0-6ea89af3a587\") " pod="opendatahub/maas-controller-575c7b5c5b-p8bxg" Oct 30 17:19:15.101200 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:19:15.101097 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/44aa1bfa-6067-45b4-ada0-6ea89af3a587-webhook-cert\") pod \"maas-controller-575c7b5c5b-p8bxg\" (UID: \"44aa1bfa-6067-45b4-ada0-6ea89af3a587\") " pod="opendatahub/maas-controller-575c7b5c5b-p8bxg" Oct 30 17:19:15.104064 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:19:15.104024 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44aa1bfa-6067-45b4-ada0-6ea89af3a587-metrics-certs\") pod \"maas-controller-575c7b5c5b-p8bxg\" (UID: \"44aa1bfa-6067-45b4-ada0-6ea89af3a587\") " pod="opendatahub/maas-controller-575c7b5c5b-p8bxg" Oct 30 17:19:15.104253 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:19:15.104229 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/44aa1bfa-6067-45b4-ada0-6ea89af3a587-webhook-cert\") pod \"maas-controller-575c7b5c5b-p8bxg\" (UID: \"44aa1bfa-6067-45b4-ada0-6ea89af3a587\") " pod="opendatahub/maas-controller-575c7b5c5b-p8bxg" Oct 30 17:19:15.129863 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:19:15.129832 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tz2ln\" (UniqueName: \"kubernetes.io/projected/44aa1bfa-6067-45b4-ada0-6ea89af3a587-kube-api-access-tz2ln\") pod \"maas-controller-575c7b5c5b-p8bxg\" (UID: \"44aa1bfa-6067-45b4-ada0-6ea89af3a587\") " pod="opendatahub/maas-controller-575c7b5c5b-p8bxg" Oct 30 17:19:15.284286 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:19:15.284207 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-575c7b5c5b-p8bxg" Oct 30 17:19:15.583701 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:19:15.583672 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-575c7b5c5b-p8bxg"] Oct 30 17:19:15.585336 ip-10-0-131-136 kubenswrapper[5052]: W1030 17:19:15.585305 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44aa1bfa_6067_45b4_ada0_6ea89af3a587.slice/crio-4a9de19c27d95a6f8a9cafa5572349d5d58f45b1b231ca36e034bf1e0c97eb99 WatchSource:0}: Error finding container 4a9de19c27d95a6f8a9cafa5572349d5d58f45b1b231ca36e034bf1e0c97eb99: Status 404 returned error can't find the container with id 4a9de19c27d95a6f8a9cafa5572349d5d58f45b1b231ca36e034bf1e0c97eb99 Oct 30 17:19:15.706378 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:19:15.706345 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-575c7b5c5b-p8bxg" event={"ID":"44aa1bfa-6067-45b4-ada0-6ea89af3a587","Type":"ContainerStarted","Data":"4a9de19c27d95a6f8a9cafa5572349d5d58f45b1b231ca36e034bf1e0c97eb99"} Oct 30 17:19:16.714462 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:19:16.714413 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-575c7b5c5b-p8bxg" event={"ID":"44aa1bfa-6067-45b4-ada0-6ea89af3a587","Type":"ContainerStarted","Data":"c557fe55ff13488d907699c74e87b0453b077c57341ea7bec81fdbde22c9a263"} Oct 30 17:19:16.714880 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:19:16.714493 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-575c7b5c5b-p8bxg" Oct 30 17:19:16.761661 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:19:16.761575 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-575c7b5c5b-p8bxg" podStartSLOduration=2.412737138 podStartE2EDuration="2.76155785s" totalImagesPullingTime="348.820712ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:19:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:19:16.760399511 +0000 UTC m=+3512.287734750" watchObservedRunningTime="2026-10-30 17:19:16.76155785 +0000 UTC m=+3512.288893072" Oct 30 17:19:18.839606 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:19:18.839547 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 17:19:27.722048 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:19:27.722013 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-575c7b5c5b-p8bxg" Oct 30 17:19:41.000089 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:19:41.000054 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gd2bm"] Oct 30 17:19:41.000643 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:19:41.000262 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gd2bm" podUID="57007fdd-a5ca-4597-a8c4-901310d35117" containerName="discovery" containerID="cri-o://622a8d32838021cb176634adf4ec426bef77bb5f2d2232a9f9e420da26d0a0f3" gracePeriod=30 Oct 30 17:19:41.263525 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:19:41.263467 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gd2bm" Oct 30 17:19:41.336779 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:19:41.336746 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57007fdd-a5ca-4597-a8c4-901310d35117-istio-token\" (UniqueName: \"kubernetes.io/projected/57007fdd-a5ca-4597-a8c4-901310d35117-istio-token\") pod \"57007fdd-a5ca-4597-a8c4-901310d35117\" (UID: \"57007fdd-a5ca-4597-a8c4-901310d35117\") " Oct 30 17:19:41.336962 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:19:41.336794 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57007fdd-a5ca-4597-a8c4-901310d35117-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/57007fdd-a5ca-4597-a8c4-901310d35117-istio-kubeconfig\") pod \"57007fdd-a5ca-4597-a8c4-901310d35117\" (UID: \"57007fdd-a5ca-4597-a8c4-901310d35117\") " Oct 30 17:19:41.336962 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:19:41.336831 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57007fdd-a5ca-4597-a8c4-901310d35117-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/57007fdd-a5ca-4597-a8c4-901310d35117-local-certs\") pod \"57007fdd-a5ca-4597-a8c4-901310d35117\" (UID: \"57007fdd-a5ca-4597-a8c4-901310d35117\") " Oct 30 17:19:41.336962 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:19:41.336887 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57007fdd-a5ca-4597-a8c4-901310d35117-cacerts\" (UniqueName: \"kubernetes.io/secret/57007fdd-a5ca-4597-a8c4-901310d35117-cacerts\") pod \"57007fdd-a5ca-4597-a8c4-901310d35117\" (UID: \"57007fdd-a5ca-4597-a8c4-901310d35117\") " Oct 30 17:19:41.336962 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:19:41.336927 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57007fdd-a5ca-4597-a8c4-901310d35117-kube-api-access-kg77s\" (UniqueName: \"kubernetes.io/projected/57007fdd-a5ca-4597-a8c4-901310d35117-kube-api-access-kg77s\") pod \"57007fdd-a5ca-4597-a8c4-901310d35117\" (UID: \"57007fdd-a5ca-4597-a8c4-901310d35117\") " Oct 30 17:19:41.337194 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:19:41.336989 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57007fdd-a5ca-4597-a8c4-901310d35117-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/57007fdd-a5ca-4597-a8c4-901310d35117-istio-csr-dns-cert\") pod \"57007fdd-a5ca-4597-a8c4-901310d35117\" (UID: \"57007fdd-a5ca-4597-a8c4-901310d35117\") " Oct 30 17:19:41.337194 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:19:41.337019 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57007fdd-a5ca-4597-a8c4-901310d35117-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/57007fdd-a5ca-4597-a8c4-901310d35117-istio-csr-ca-configmap\") pod \"57007fdd-a5ca-4597-a8c4-901310d35117\" (UID: \"57007fdd-a5ca-4597-a8c4-901310d35117\") " Oct 30 17:19:41.337835 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:19:41.337770 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57007fdd-a5ca-4597-a8c4-901310d35117-istio-csr-ca-configmap" pod "57007fdd-a5ca-4597-a8c4-901310d35117" (UID: "57007fdd-a5ca-4597-a8c4-901310d35117"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:19:41.339716 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:19:41.339685 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57007fdd-a5ca-4597-a8c4-901310d35117-istio-kubeconfig" pod "57007fdd-a5ca-4597-a8c4-901310d35117" (UID: "57007fdd-a5ca-4597-a8c4-901310d35117"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:19:41.339848 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:19:41.339746 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57007fdd-a5ca-4597-a8c4-901310d35117-kube-api-access-kg77s" pod "57007fdd-a5ca-4597-a8c4-901310d35117" (UID: "57007fdd-a5ca-4597-a8c4-901310d35117"). InnerVolumeSpecName "kube-api-access-kg77s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:19:41.339955 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:19:41.339930 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57007fdd-a5ca-4597-a8c4-901310d35117-istio-token" pod "57007fdd-a5ca-4597-a8c4-901310d35117" (UID: "57007fdd-a5ca-4597-a8c4-901310d35117"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:19:41.340164 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:19:41.340141 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57007fdd-a5ca-4597-a8c4-901310d35117-istio-csr-dns-cert" pod "57007fdd-a5ca-4597-a8c4-901310d35117" (UID: "57007fdd-a5ca-4597-a8c4-901310d35117"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:19:41.340303 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:19:41.340279 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57007fdd-a5ca-4597-a8c4-901310d35117-local-certs" pod "57007fdd-a5ca-4597-a8c4-901310d35117" (UID: "57007fdd-a5ca-4597-a8c4-901310d35117"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:19:41.340409 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:19:41.340396 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57007fdd-a5ca-4597-a8c4-901310d35117-cacerts" pod "57007fdd-a5ca-4597-a8c4-901310d35117" (UID: "57007fdd-a5ca-4597-a8c4-901310d35117"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:19:41.438534 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:19:41.438498 5052 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57007fdd-a5ca-4597-a8c4-901310d35117-istio-token\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:19:41.438534 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:19:41.438528 5052 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/57007fdd-a5ca-4597-a8c4-901310d35117-istio-kubeconfig\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:19:41.438534 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:19:41.438537 5052 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/57007fdd-a5ca-4597-a8c4-901310d35117-local-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:19:41.438782 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:19:41.438546 5052 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/57007fdd-a5ca-4597-a8c4-901310d35117-cacerts\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:19:41.438782 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:19:41.438555 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kg77s\" (UniqueName: \"kubernetes.io/projected/57007fdd-a5ca-4597-a8c4-901310d35117-kube-api-access-kg77s\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:19:41.438782 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:19:41.438563 5052 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/57007fdd-a5ca-4597-a8c4-901310d35117-istio-csr-dns-cert\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:19:41.438782 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:19:41.438572 5052 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/57007fdd-a5ca-4597-a8c4-901310d35117-istio-csr-ca-configmap\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:19:41.847015 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:19:41.846932 5052 generic.go:356] "Generic (PLEG): container finished" podID="57007fdd-a5ca-4597-a8c4-901310d35117" containerID="622a8d32838021cb176634adf4ec426bef77bb5f2d2232a9f9e420da26d0a0f3" exitCode=0 Oct 30 17:19:41.847015 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:19:41.846979 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gd2bm" Oct 30 17:19:41.847234 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:19:41.847013 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gd2bm" event={"ID":"57007fdd-a5ca-4597-a8c4-901310d35117","Type":"ContainerDied","Data":"622a8d32838021cb176634adf4ec426bef77bb5f2d2232a9f9e420da26d0a0f3"} Oct 30 17:19:41.847234 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:19:41.847076 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gd2bm" event={"ID":"57007fdd-a5ca-4597-a8c4-901310d35117","Type":"ContainerDied","Data":"5c8bdd5e7e763a2f6df70135cd2a259b01ec9cd1ca48022c6a12e84719028d4a"} Oct 30 17:19:41.847234 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:19:41.847105 5052 scope.go:122] "RemoveContainer" containerID="622a8d32838021cb176634adf4ec426bef77bb5f2d2232a9f9e420da26d0a0f3" Oct 30 17:19:41.861971 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:19:41.861947 5052 scope.go:122] "RemoveContainer" containerID="622a8d32838021cb176634adf4ec426bef77bb5f2d2232a9f9e420da26d0a0f3" Oct 30 17:19:41.862269 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:19:41.862242 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"622a8d32838021cb176634adf4ec426bef77bb5f2d2232a9f9e420da26d0a0f3\": container with ID starting with 622a8d32838021cb176634adf4ec426bef77bb5f2d2232a9f9e420da26d0a0f3 not found: ID does not exist" containerID="622a8d32838021cb176634adf4ec426bef77bb5f2d2232a9f9e420da26d0a0f3" Oct 30 17:19:41.862339 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:19:41.862277 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"622a8d32838021cb176634adf4ec426bef77bb5f2d2232a9f9e420da26d0a0f3"} err="failed to get container status \"622a8d32838021cb176634adf4ec426bef77bb5f2d2232a9f9e420da26d0a0f3\": rpc error: code = NotFound desc = could not find container \"622a8d32838021cb176634adf4ec426bef77bb5f2d2232a9f9e420da26d0a0f3\": container with ID starting with 622a8d32838021cb176634adf4ec426bef77bb5f2d2232a9f9e420da26d0a0f3 not found: ID does not exist" Oct 30 17:19:41.915882 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:19:41.915851 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gd2bm"] Oct 30 17:19:41.929060 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:19:41.929040 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gd2bm"] Oct 30 17:19:42.596248 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:19:42.596213 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57007fdd-a5ca-4597-a8c4-901310d35117" path="/var/lib/kubelet/pods/57007fdd-a5ca-4597-a8c4-901310d35117/volumes" Oct 30 17:19:45.917412 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:19:45.917380 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cdwcj"] Oct 30 17:19:48.839482 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:19:48.839448 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 17:20:03.941791 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:03.941755 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cdwcj"] Oct 30 17:20:08.214888 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:08.214851 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cdwcj"] Oct 30 17:20:09.254358 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:09.254325 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9kbt9"] Oct 30 17:20:09.254859 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:09.254784 5052 cpu_manager.go:414] "containerMap: removing container" podUID="57007fdd-a5ca-4597-a8c4-901310d35117" containerName="discovery" Oct 30 17:20:09.254859 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:09.254799 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57007fdd-a5ca-4597-a8c4-901310d35117" containerName="discovery" Oct 30 17:20:09.254940 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:09.254870 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="57007fdd-a5ca-4597-a8c4-901310d35117" containerName="discovery" Oct 30 17:20:09.258044 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:09.258023 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9kbt9" Oct 30 17:20:09.290672 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:09.290640 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9kbt9"] Oct 30 17:20:09.293227 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:09.293195 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-kpphn"] Oct 30 17:20:09.297541 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:09.297517 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-kpphn" Oct 30 17:20:09.309328 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:09.309298 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/384a9c57-053d-477e-badb-355261588881-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-9kbt9\" (UID: \"384a9c57-053d-477e-badb-355261588881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9kbt9" Oct 30 17:20:09.309630 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:09.309564 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/384a9c57-053d-477e-badb-355261588881-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-9kbt9\" (UID: \"384a9c57-053d-477e-badb-355261588881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9kbt9" Oct 30 17:20:09.309864 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:09.309837 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwrzs\" (UniqueName: \"kubernetes.io/projected/384a9c57-053d-477e-badb-355261588881-kube-api-access-zwrzs\") pod \"maas-default-gateway-openshift-default-76f4876667-9kbt9\" (UID: \"384a9c57-053d-477e-badb-355261588881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9kbt9" Oct 30 17:20:09.310058 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:09.310025 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/384a9c57-053d-477e-badb-355261588881-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-9kbt9\" (UID: \"384a9c57-053d-477e-badb-355261588881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9kbt9" Oct 30 17:20:09.310182 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:09.310071 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/384a9c57-053d-477e-badb-355261588881-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-9kbt9\" (UID: \"384a9c57-053d-477e-badb-355261588881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9kbt9" Oct 30 17:20:09.310182 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:09.310160 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/384a9c57-053d-477e-badb-355261588881-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-9kbt9\" (UID: \"384a9c57-053d-477e-badb-355261588881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9kbt9" Oct 30 17:20:09.310319 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:09.310214 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/384a9c57-053d-477e-badb-355261588881-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-9kbt9\" (UID: \"384a9c57-053d-477e-badb-355261588881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9kbt9" Oct 30 17:20:09.310319 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:09.310252 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/384a9c57-053d-477e-badb-355261588881-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-9kbt9\" (UID: \"384a9c57-053d-477e-badb-355261588881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9kbt9" Oct 30 17:20:09.310319 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:09.310281 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/384a9c57-053d-477e-badb-355261588881-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-9kbt9\" (UID: \"384a9c57-053d-477e-badb-355261588881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9kbt9" Oct 30 17:20:09.326511 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:09.326484 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-kpphn"] Oct 30 17:20:09.410964 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:09.410926 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/384a9c57-053d-477e-badb-355261588881-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-9kbt9\" (UID: \"384a9c57-053d-477e-badb-355261588881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9kbt9" Oct 30 17:20:09.411147 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:09.410969 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4c977d2-7480-4d5c-87d6-743c519e2efc-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-kpphn\" (UID: \"c4c977d2-7480-4d5c-87d6-743c519e2efc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-kpphn" Oct 30 17:20:09.411147 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:09.410997 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/384a9c57-053d-477e-badb-355261588881-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-9kbt9\" (UID: \"384a9c57-053d-477e-badb-355261588881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9kbt9" Oct 30 17:20:09.411147 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:09.411078 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bg6wj\" (UniqueName: \"kubernetes.io/projected/c4c977d2-7480-4d5c-87d6-743c519e2efc-kube-api-access-bg6wj\") pod \"maas-default-gateway-openshift-default-76f4876667-kpphn\" (UID: \"c4c977d2-7480-4d5c-87d6-743c519e2efc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-kpphn" Oct 30 17:20:09.411147 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:09.411140 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwrzs\" (UniqueName: \"kubernetes.io/projected/384a9c57-053d-477e-badb-355261588881-kube-api-access-zwrzs\") pod \"maas-default-gateway-openshift-default-76f4876667-9kbt9\" (UID: \"384a9c57-053d-477e-badb-355261588881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9kbt9" Oct 30 17:20:09.411391 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:09.411174 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/384a9c57-053d-477e-badb-355261588881-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-9kbt9\" (UID: \"384a9c57-053d-477e-badb-355261588881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9kbt9" Oct 30 17:20:09.411391 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:09.411235 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/384a9c57-053d-477e-badb-355261588881-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-9kbt9\" (UID: \"384a9c57-053d-477e-badb-355261588881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9kbt9" Oct 30 17:20:09.411391 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:09.411285 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4c977d2-7480-4d5c-87d6-743c519e2efc-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-kpphn\" (UID: \"c4c977d2-7480-4d5c-87d6-743c519e2efc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-kpphn" Oct 30 17:20:09.411391 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:09.411319 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4c977d2-7480-4d5c-87d6-743c519e2efc-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-kpphn\" (UID: \"c4c977d2-7480-4d5c-87d6-743c519e2efc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-kpphn" Oct 30 17:20:09.411391 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:09.411363 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4c977d2-7480-4d5c-87d6-743c519e2efc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-kpphn\" (UID: \"c4c977d2-7480-4d5c-87d6-743c519e2efc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-kpphn" Oct 30 17:20:09.411391 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:09.411366 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/384a9c57-053d-477e-badb-355261588881-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-9kbt9\" (UID: \"384a9c57-053d-477e-badb-355261588881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9kbt9" Oct 30 17:20:09.411391 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:09.411389 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4c977d2-7480-4d5c-87d6-743c519e2efc-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-kpphn\" (UID: \"c4c977d2-7480-4d5c-87d6-743c519e2efc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-kpphn" Oct 30 17:20:09.411813 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:09.411427 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/384a9c57-053d-477e-badb-355261588881-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-9kbt9\" (UID: \"384a9c57-053d-477e-badb-355261588881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9kbt9" Oct 30 17:20:09.411813 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:09.411542 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/384a9c57-053d-477e-badb-355261588881-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-9kbt9\" (UID: \"384a9c57-053d-477e-badb-355261588881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9kbt9" Oct 30 17:20:09.411813 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:09.411634 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/384a9c57-053d-477e-badb-355261588881-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-9kbt9\" (UID: \"384a9c57-053d-477e-badb-355261588881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9kbt9" Oct 30 17:20:09.411813 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:09.411664 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4c977d2-7480-4d5c-87d6-743c519e2efc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-kpphn\" (UID: \"c4c977d2-7480-4d5c-87d6-743c519e2efc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-kpphn" Oct 30 17:20:09.411813 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:09.411698 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4c977d2-7480-4d5c-87d6-743c519e2efc-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-kpphn\" (UID: \"c4c977d2-7480-4d5c-87d6-743c519e2efc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-kpphn" Oct 30 17:20:09.411813 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:09.411734 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/384a9c57-053d-477e-badb-355261588881-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-9kbt9\" (UID: \"384a9c57-053d-477e-badb-355261588881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9kbt9" Oct 30 17:20:09.411813 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:09.411783 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4c977d2-7480-4d5c-87d6-743c519e2efc-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-kpphn\" (UID: \"c4c977d2-7480-4d5c-87d6-743c519e2efc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-kpphn" Oct 30 17:20:09.412259 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:09.411820 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/384a9c57-053d-477e-badb-355261588881-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-9kbt9\" (UID: \"384a9c57-053d-477e-badb-355261588881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9kbt9" Oct 30 17:20:09.412259 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:09.411984 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/384a9c57-053d-477e-badb-355261588881-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-9kbt9\" (UID: \"384a9c57-053d-477e-badb-355261588881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9kbt9" Oct 30 17:20:09.412259 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:09.411999 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/384a9c57-053d-477e-badb-355261588881-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-9kbt9\" (UID: \"384a9c57-053d-477e-badb-355261588881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9kbt9" Oct 30 17:20:09.412259 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:09.412125 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/384a9c57-053d-477e-badb-355261588881-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-9kbt9\" (UID: \"384a9c57-053d-477e-badb-355261588881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9kbt9" Oct 30 17:20:09.413922 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:09.413892 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/384a9c57-053d-477e-badb-355261588881-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-9kbt9\" (UID: \"384a9c57-053d-477e-badb-355261588881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9kbt9" Oct 30 17:20:09.414236 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:09.414215 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/384a9c57-053d-477e-badb-355261588881-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-9kbt9\" (UID: \"384a9c57-053d-477e-badb-355261588881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9kbt9" Oct 30 17:20:09.435152 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:09.435125 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/384a9c57-053d-477e-badb-355261588881-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-9kbt9\" (UID: \"384a9c57-053d-477e-badb-355261588881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9kbt9" Oct 30 17:20:09.435395 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:09.435372 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwrzs\" (UniqueName: \"kubernetes.io/projected/384a9c57-053d-477e-badb-355261588881-kube-api-access-zwrzs\") pod \"maas-default-gateway-openshift-default-76f4876667-9kbt9\" (UID: \"384a9c57-053d-477e-badb-355261588881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9kbt9" Oct 30 17:20:09.513263 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:09.513164 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bg6wj\" (UniqueName: \"kubernetes.io/projected/c4c977d2-7480-4d5c-87d6-743c519e2efc-kube-api-access-bg6wj\") pod \"maas-default-gateway-openshift-default-76f4876667-kpphn\" (UID: \"c4c977d2-7480-4d5c-87d6-743c519e2efc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-kpphn" Oct 30 17:20:09.513263 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:09.513258 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4c977d2-7480-4d5c-87d6-743c519e2efc-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-kpphn\" (UID: \"c4c977d2-7480-4d5c-87d6-743c519e2efc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-kpphn" Oct 30 17:20:09.513456 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:09.513292 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4c977d2-7480-4d5c-87d6-743c519e2efc-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-kpphn\" (UID: \"c4c977d2-7480-4d5c-87d6-743c519e2efc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-kpphn" Oct 30 17:20:09.513456 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:09.513337 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4c977d2-7480-4d5c-87d6-743c519e2efc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-kpphn\" (UID: \"c4c977d2-7480-4d5c-87d6-743c519e2efc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-kpphn" Oct 30 17:20:09.513456 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:09.513365 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4c977d2-7480-4d5c-87d6-743c519e2efc-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-kpphn\" (UID: \"c4c977d2-7480-4d5c-87d6-743c519e2efc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-kpphn" Oct 30 17:20:09.513456 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:09.513429 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4c977d2-7480-4d5c-87d6-743c519e2efc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-kpphn\" (UID: \"c4c977d2-7480-4d5c-87d6-743c519e2efc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-kpphn" Oct 30 17:20:09.513611 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:09.513461 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4c977d2-7480-4d5c-87d6-743c519e2efc-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-kpphn\" (UID: \"c4c977d2-7480-4d5c-87d6-743c519e2efc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-kpphn" Oct 30 17:20:09.513611 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:09.513493 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4c977d2-7480-4d5c-87d6-743c519e2efc-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-kpphn\" (UID: \"c4c977d2-7480-4d5c-87d6-743c519e2efc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-kpphn" Oct 30 17:20:09.513611 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:09.513549 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4c977d2-7480-4d5c-87d6-743c519e2efc-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-kpphn\" (UID: \"c4c977d2-7480-4d5c-87d6-743c519e2efc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-kpphn" Oct 30 17:20:09.514086 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:09.514058 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4c977d2-7480-4d5c-87d6-743c519e2efc-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-kpphn\" (UID: \"c4c977d2-7480-4d5c-87d6-743c519e2efc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-kpphn" Oct 30 17:20:09.514216 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:09.514099 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4c977d2-7480-4d5c-87d6-743c519e2efc-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-kpphn\" (UID: \"c4c977d2-7480-4d5c-87d6-743c519e2efc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-kpphn" Oct 30 17:20:09.514216 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:09.514152 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4c977d2-7480-4d5c-87d6-743c519e2efc-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-kpphn\" (UID: \"c4c977d2-7480-4d5c-87d6-743c519e2efc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-kpphn" Oct 30 17:20:09.514540 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:09.514517 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4c977d2-7480-4d5c-87d6-743c519e2efc-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-kpphn\" (UID: \"c4c977d2-7480-4d5c-87d6-743c519e2efc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-kpphn" Oct 30 17:20:09.514616 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:09.514557 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4c977d2-7480-4d5c-87d6-743c519e2efc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-kpphn\" (UID: \"c4c977d2-7480-4d5c-87d6-743c519e2efc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-kpphn" Oct 30 17:20:09.515981 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:09.515961 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4c977d2-7480-4d5c-87d6-743c519e2efc-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-kpphn\" (UID: \"c4c977d2-7480-4d5c-87d6-743c519e2efc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-kpphn" Oct 30 17:20:09.516468 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:09.516449 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4c977d2-7480-4d5c-87d6-743c519e2efc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-kpphn\" (UID: \"c4c977d2-7480-4d5c-87d6-743c519e2efc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-kpphn" Oct 30 17:20:09.538670 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:09.538643 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4c977d2-7480-4d5c-87d6-743c519e2efc-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-kpphn\" (UID: \"c4c977d2-7480-4d5c-87d6-743c519e2efc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-kpphn" Oct 30 17:20:09.540184 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:09.540154 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bg6wj\" (UniqueName: \"kubernetes.io/projected/c4c977d2-7480-4d5c-87d6-743c519e2efc-kube-api-access-bg6wj\") pod \"maas-default-gateway-openshift-default-76f4876667-kpphn\" (UID: \"c4c977d2-7480-4d5c-87d6-743c519e2efc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-kpphn" Oct 30 17:20:09.574877 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:09.574832 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9kbt9" Oct 30 17:20:09.617284 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:09.616994 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-kpphn" Oct 30 17:20:09.913386 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:09.913361 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9kbt9"] Oct 30 17:20:09.915989 ip-10-0-131-136 kubenswrapper[5052]: W1030 17:20:09.915957 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod384a9c57_053d_477e_badb_355261588881.slice/crio-002300998ffbb2b9adac292c5f8f8bafd04b4ea8932b00432630e20ed22c3afc WatchSource:0}: Error finding container 002300998ffbb2b9adac292c5f8f8bafd04b4ea8932b00432630e20ed22c3afc: Status 404 returned error can't find the container with id 002300998ffbb2b9adac292c5f8f8bafd04b4ea8932b00432630e20ed22c3afc Oct 30 17:20:09.918947 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:09.918903 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:20:09.919120 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:09.919099 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:20:09.919470 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:09.919449 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:20:09.973848 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:09.972869 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-kpphn"] Oct 30 17:20:09.997051 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:09.996930 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9kbt9" event={"ID":"384a9c57-053d-477e-badb-355261588881","Type":"ContainerStarted","Data":"002300998ffbb2b9adac292c5f8f8bafd04b4ea8932b00432630e20ed22c3afc"} Oct 30 17:20:09.999570 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:09.999528 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:20:09.999809 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:09.999776 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:20:09.999987 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:09.999965 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:20:11.060181 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:11.060130 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9kbt9" event={"ID":"384a9c57-053d-477e-badb-355261588881","Type":"ContainerStarted","Data":"412866de6c1f173ecd61049e96aab8dd01501c4baf9a0d3b2cc9a4d11e88ab49"} Oct 30 17:20:11.063630 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:11.063598 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-kpphn" event={"ID":"c4c977d2-7480-4d5c-87d6-743c519e2efc","Type":"ContainerStarted","Data":"9105500bee23a6f4283835e83c6a352c25fd5a56b8e185c02dad5ca0abd07cb5"} Oct 30 17:20:11.063889 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:11.063868 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-kpphn" event={"ID":"c4c977d2-7480-4d5c-87d6-743c519e2efc","Type":"ContainerStarted","Data":"fd8deaf4757c6de6ed7cf6a9bfe249a48466041bd4be8034be05da7a0e240e04"} Oct 30 17:20:11.182703 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:11.182634 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-kpphn" podStartSLOduration=2.182610921 podStartE2EDuration="2.182610921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:20:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:20:11.122438888 +0000 UTC m=+3566.649774118" watchObservedRunningTime="2026-10-30 17:20:11.182610921 +0000 UTC m=+3566.709946157" Oct 30 17:20:11.577701 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:11.577655 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9kbt9" Oct 30 17:20:11.579743 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:11.579703 5052 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-9kbt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" start-of-body= Oct 30 17:20:11.579935 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:11.579916 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9kbt9" podUID="384a9c57-053d-477e-badb-355261588881" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" Oct 30 17:20:11.617802 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:11.617774 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-kpphn" Oct 30 17:20:11.618904 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:11.618879 5052 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-kpphn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body= Oct 30 17:20:11.618997 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:11.618925 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-kpphn" podUID="c4c977d2-7480-4d5c-87d6-743c519e2efc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" Oct 30 17:20:12.576022 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:12.575982 5052 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-9kbt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" start-of-body= Oct 30 17:20:12.576668 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:12.576632 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9kbt9" podUID="384a9c57-053d-477e-badb-355261588881" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" Oct 30 17:20:12.617433 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:12.617399 5052 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-kpphn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body= Oct 30 17:20:12.617614 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:12.617456 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-kpphn" podUID="c4c977d2-7480-4d5c-87d6-743c519e2efc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" Oct 30 17:20:13.577993 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:13.577950 5052 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-9kbt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" start-of-body= Oct 30 17:20:13.578500 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:13.578013 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9kbt9" podUID="384a9c57-053d-477e-badb-355261588881" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" Oct 30 17:20:13.617432 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:13.617397 5052 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-kpphn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body= Oct 30 17:20:13.617616 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:13.617455 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-kpphn" podUID="c4c977d2-7480-4d5c-87d6-743c519e2efc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" Oct 30 17:20:14.577992 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:14.577955 5052 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-9kbt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" start-of-body= Oct 30 17:20:14.578533 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:14.578018 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9kbt9" podUID="384a9c57-053d-477e-badb-355261588881" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" Oct 30 17:20:14.618110 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:14.618077 5052 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-kpphn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body= Oct 30 17:20:14.618277 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:14.618137 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-kpphn" podUID="c4c977d2-7480-4d5c-87d6-743c519e2efc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" Oct 30 17:20:15.577062 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:15.577023 5052 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-9kbt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" start-of-body= Oct 30 17:20:15.577373 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:15.577322 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9kbt9" podUID="384a9c57-053d-477e-badb-355261588881" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" Oct 30 17:20:15.618385 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:15.618355 5052 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-kpphn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body= Oct 30 17:20:15.618808 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:15.618404 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-kpphn" podUID="c4c977d2-7480-4d5c-87d6-743c519e2efc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" Oct 30 17:20:16.576279 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:16.576235 5052 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-9kbt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" start-of-body= Oct 30 17:20:16.576469 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:16.576308 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9kbt9" podUID="384a9c57-053d-477e-badb-355261588881" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" Oct 30 17:20:16.617319 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:16.617290 5052 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-kpphn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body= Oct 30 17:20:16.617500 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:16.617341 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-kpphn" podUID="c4c977d2-7480-4d5c-87d6-743c519e2efc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" Oct 30 17:20:17.576102 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:17.576065 5052 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-9kbt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" start-of-body= Oct 30 17:20:17.576532 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:17.576118 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9kbt9" podUID="384a9c57-053d-477e-badb-355261588881" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" Oct 30 17:20:17.617482 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:17.617449 5052 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-kpphn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body= Oct 30 17:20:17.617661 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:17.617495 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-kpphn" podUID="c4c977d2-7480-4d5c-87d6-743c519e2efc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" Oct 30 17:20:18.580010 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:18.579974 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9kbt9" Oct 30 17:20:18.580457 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:18.580278 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9kbt9" Oct 30 17:20:18.581383 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:18.581360 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9kbt9" Oct 30 17:20:18.620994 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:18.620972 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-kpphn" Oct 30 17:20:18.621259 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:18.621237 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-kpphn" Oct 30 17:20:18.621844 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:18.621828 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-kpphn" Oct 30 17:20:18.642780 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:18.642739 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9kbt9" podStartSLOduration=9.642727514 podStartE2EDuration="9.642727514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:20:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:20:11.185836766 +0000 UTC m=+3566.713171986" watchObservedRunningTime="2026-10-30 17:20:18.642727514 +0000 UTC m=+3574.170062734" Oct 30 17:20:18.839813 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:20:18.839723 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 17:20:24.302000 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:24.301962 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cdwcj"] Oct 30 17:20:45.927278 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:45.927252 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4wrf8_4f0f46f4-5cda-4d67-b6f3-47df06eb844c/console-operator/2.log" Oct 30 17:20:45.932660 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:45.932640 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4wrf8_4f0f46f4-5cda-4d67-b6f3-47df06eb844c/console-operator/2.log" Oct 30 17:20:48.840822 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:20:48.840783 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 17:20:49.806935 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:20:49.806900 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cdwcj"] Oct 30 17:21:18.839621 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:21:18.839567 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 17:21:23.401008 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:21:23.400969 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cdwcj"] Oct 30 17:21:33.686998 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:21:33.686962 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cdwcj"] Oct 30 17:21:37.825474 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:21:37.825396 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cdwcj"] Oct 30 17:21:48.839666 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:21:48.839626 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 17:21:50.352524 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:21:50.352491 5052 scope.go:122] "RemoveContainer" containerID="02ff36057c33cfa54424bd9203d4db063cde5cb59d199c726a5691d1b31b430a" Oct 30 17:21:56.149901 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:21:56.149862 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cdwcj"] Oct 30 17:22:00.952603 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:22:00.952093 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cdwcj"] Oct 30 17:22:17.624347 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:22:17.624311 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cdwcj"] Oct 30 17:22:18.840676 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:22:18.840641 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 17:22:22.611353 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:22:22.611302 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cdwcj"] Oct 30 17:22:37.919325 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:22:37.919291 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cdwcj"] Oct 30 17:22:42.643654 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:22:42.642900 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cdwcj"] Oct 30 17:22:48.838932 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:22:48.838898 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 17:22:59.023282 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:22:59.023246 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cdwcj"] Oct 30 17:23:09.888075 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:23:09.888040 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cdwcj"] Oct 30 17:23:18.842210 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:23:18.842167 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 17:23:20.724043 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:23:20.723987 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cdwcj"] Oct 30 17:23:48.839247 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:23:48.839209 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 17:24:10.136316 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:10.136280 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-57f1de-openshift-default-77c5d6878fqjd27"] Oct 30 17:24:10.140668 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:10.140646 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-57f1de-openshift-default-77c5d6878fqjd27" Oct 30 17:24:10.154855 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:10.154827 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-57f1de-openshift-default-dockercfg-h7n8n\"" Oct 30 17:24:10.169110 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:10.169084 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-57f1de-openshift-default-77c5d6878fqjd27"] Oct 30 17:24:10.234512 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:10.234475 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9mp6\" (UniqueName: \"kubernetes.io/projected/1ed76266-22d9-4ec7-bc9e-8f84355b3485-kube-api-access-q9mp6\") pod \"e2e-worker-single-main-57f1de-openshift-default-77c5d6878fqjd27\" (UID: \"1ed76266-22d9-4ec7-bc9e-8f84355b3485\") " pod="openshift-ingress/e2e-worker-single-main-57f1de-openshift-default-77c5d6878fqjd27" Oct 30 17:24:10.234724 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:10.234543 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ed76266-22d9-4ec7-bc9e-8f84355b3485-istiod-ca-cert\") pod \"e2e-worker-single-main-57f1de-openshift-default-77c5d6878fqjd27\" (UID: \"1ed76266-22d9-4ec7-bc9e-8f84355b3485\") " pod="openshift-ingress/e2e-worker-single-main-57f1de-openshift-default-77c5d6878fqjd27" Oct 30 17:24:10.234724 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:10.234617 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ed76266-22d9-4ec7-bc9e-8f84355b3485-istio-envoy\") pod \"e2e-worker-single-main-57f1de-openshift-default-77c5d6878fqjd27\" (UID: \"1ed76266-22d9-4ec7-bc9e-8f84355b3485\") " pod="openshift-ingress/e2e-worker-single-main-57f1de-openshift-default-77c5d6878fqjd27" Oct 30 17:24:10.234724 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:10.234686 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ed76266-22d9-4ec7-bc9e-8f84355b3485-istio-token\") pod \"e2e-worker-single-main-57f1de-openshift-default-77c5d6878fqjd27\" (UID: \"1ed76266-22d9-4ec7-bc9e-8f84355b3485\") " pod="openshift-ingress/e2e-worker-single-main-57f1de-openshift-default-77c5d6878fqjd27" Oct 30 17:24:10.234933 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:10.234759 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ed76266-22d9-4ec7-bc9e-8f84355b3485-istio-data\") pod \"e2e-worker-single-main-57f1de-openshift-default-77c5d6878fqjd27\" (UID: \"1ed76266-22d9-4ec7-bc9e-8f84355b3485\") " pod="openshift-ingress/e2e-worker-single-main-57f1de-openshift-default-77c5d6878fqjd27" Oct 30 17:24:10.234933 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:10.234796 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ed76266-22d9-4ec7-bc9e-8f84355b3485-istio-podinfo\") pod \"e2e-worker-single-main-57f1de-openshift-default-77c5d6878fqjd27\" (UID: \"1ed76266-22d9-4ec7-bc9e-8f84355b3485\") " pod="openshift-ingress/e2e-worker-single-main-57f1de-openshift-default-77c5d6878fqjd27" Oct 30 17:24:10.234933 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:10.234903 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ed76266-22d9-4ec7-bc9e-8f84355b3485-workload-socket\") pod \"e2e-worker-single-main-57f1de-openshift-default-77c5d6878fqjd27\" (UID: \"1ed76266-22d9-4ec7-bc9e-8f84355b3485\") " pod="openshift-ingress/e2e-worker-single-main-57f1de-openshift-default-77c5d6878fqjd27" Oct 30 17:24:10.235109 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:10.234962 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ed76266-22d9-4ec7-bc9e-8f84355b3485-workload-certs\") pod \"e2e-worker-single-main-57f1de-openshift-default-77c5d6878fqjd27\" (UID: \"1ed76266-22d9-4ec7-bc9e-8f84355b3485\") " pod="openshift-ingress/e2e-worker-single-main-57f1de-openshift-default-77c5d6878fqjd27" Oct 30 17:24:10.235109 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:10.235023 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ed76266-22d9-4ec7-bc9e-8f84355b3485-credential-socket\") pod \"e2e-worker-single-main-57f1de-openshift-default-77c5d6878fqjd27\" (UID: \"1ed76266-22d9-4ec7-bc9e-8f84355b3485\") " pod="openshift-ingress/e2e-worker-single-main-57f1de-openshift-default-77c5d6878fqjd27" Oct 30 17:24:10.336452 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:10.336421 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ed76266-22d9-4ec7-bc9e-8f84355b3485-istio-envoy\") pod \"e2e-worker-single-main-57f1de-openshift-default-77c5d6878fqjd27\" (UID: \"1ed76266-22d9-4ec7-bc9e-8f84355b3485\") " pod="openshift-ingress/e2e-worker-single-main-57f1de-openshift-default-77c5d6878fqjd27" Oct 30 17:24:10.336452 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:10.336452 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ed76266-22d9-4ec7-bc9e-8f84355b3485-istio-token\") pod \"e2e-worker-single-main-57f1de-openshift-default-77c5d6878fqjd27\" (UID: \"1ed76266-22d9-4ec7-bc9e-8f84355b3485\") " pod="openshift-ingress/e2e-worker-single-main-57f1de-openshift-default-77c5d6878fqjd27" Oct 30 17:24:10.336695 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:10.336472 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ed76266-22d9-4ec7-bc9e-8f84355b3485-istio-data\") pod \"e2e-worker-single-main-57f1de-openshift-default-77c5d6878fqjd27\" (UID: \"1ed76266-22d9-4ec7-bc9e-8f84355b3485\") " pod="openshift-ingress/e2e-worker-single-main-57f1de-openshift-default-77c5d6878fqjd27" Oct 30 17:24:10.336695 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:10.336493 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ed76266-22d9-4ec7-bc9e-8f84355b3485-istio-podinfo\") pod \"e2e-worker-single-main-57f1de-openshift-default-77c5d6878fqjd27\" (UID: \"1ed76266-22d9-4ec7-bc9e-8f84355b3485\") " pod="openshift-ingress/e2e-worker-single-main-57f1de-openshift-default-77c5d6878fqjd27" Oct 30 17:24:10.336695 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:10.336537 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ed76266-22d9-4ec7-bc9e-8f84355b3485-workload-socket\") pod \"e2e-worker-single-main-57f1de-openshift-default-77c5d6878fqjd27\" (UID: \"1ed76266-22d9-4ec7-bc9e-8f84355b3485\") " pod="openshift-ingress/e2e-worker-single-main-57f1de-openshift-default-77c5d6878fqjd27" Oct 30 17:24:10.336695 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:10.336615 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ed76266-22d9-4ec7-bc9e-8f84355b3485-workload-certs\") pod \"e2e-worker-single-main-57f1de-openshift-default-77c5d6878fqjd27\" (UID: \"1ed76266-22d9-4ec7-bc9e-8f84355b3485\") " pod="openshift-ingress/e2e-worker-single-main-57f1de-openshift-default-77c5d6878fqjd27" Oct 30 17:24:10.336695 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:10.336679 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ed76266-22d9-4ec7-bc9e-8f84355b3485-credential-socket\") pod \"e2e-worker-single-main-57f1de-openshift-default-77c5d6878fqjd27\" (UID: \"1ed76266-22d9-4ec7-bc9e-8f84355b3485\") " pod="openshift-ingress/e2e-worker-single-main-57f1de-openshift-default-77c5d6878fqjd27" Oct 30 17:24:10.336997 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:10.336760 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9mp6\" (UniqueName: \"kubernetes.io/projected/1ed76266-22d9-4ec7-bc9e-8f84355b3485-kube-api-access-q9mp6\") pod \"e2e-worker-single-main-57f1de-openshift-default-77c5d6878fqjd27\" (UID: \"1ed76266-22d9-4ec7-bc9e-8f84355b3485\") " pod="openshift-ingress/e2e-worker-single-main-57f1de-openshift-default-77c5d6878fqjd27" Oct 30 17:24:10.336997 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:10.336808 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ed76266-22d9-4ec7-bc9e-8f84355b3485-istiod-ca-cert\") pod \"e2e-worker-single-main-57f1de-openshift-default-77c5d6878fqjd27\" (UID: \"1ed76266-22d9-4ec7-bc9e-8f84355b3485\") " pod="openshift-ingress/e2e-worker-single-main-57f1de-openshift-default-77c5d6878fqjd27" Oct 30 17:24:10.337124 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:10.337006 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ed76266-22d9-4ec7-bc9e-8f84355b3485-workload-socket\") pod \"e2e-worker-single-main-57f1de-openshift-default-77c5d6878fqjd27\" (UID: \"1ed76266-22d9-4ec7-bc9e-8f84355b3485\") " pod="openshift-ingress/e2e-worker-single-main-57f1de-openshift-default-77c5d6878fqjd27" Oct 30 17:24:10.337124 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:10.337076 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ed76266-22d9-4ec7-bc9e-8f84355b3485-workload-certs\") pod \"e2e-worker-single-main-57f1de-openshift-default-77c5d6878fqjd27\" (UID: \"1ed76266-22d9-4ec7-bc9e-8f84355b3485\") " pod="openshift-ingress/e2e-worker-single-main-57f1de-openshift-default-77c5d6878fqjd27" Oct 30 17:24:10.337245 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:10.337123 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ed76266-22d9-4ec7-bc9e-8f84355b3485-credential-socket\") pod \"e2e-worker-single-main-57f1de-openshift-default-77c5d6878fqjd27\" (UID: \"1ed76266-22d9-4ec7-bc9e-8f84355b3485\") " pod="openshift-ingress/e2e-worker-single-main-57f1de-openshift-default-77c5d6878fqjd27" Oct 30 17:24:10.337245 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:10.337129 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ed76266-22d9-4ec7-bc9e-8f84355b3485-istio-data\") pod \"e2e-worker-single-main-57f1de-openshift-default-77c5d6878fqjd27\" (UID: \"1ed76266-22d9-4ec7-bc9e-8f84355b3485\") " pod="openshift-ingress/e2e-worker-single-main-57f1de-openshift-default-77c5d6878fqjd27" Oct 30 17:24:10.337523 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:10.337505 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ed76266-22d9-4ec7-bc9e-8f84355b3485-istiod-ca-cert\") pod \"e2e-worker-single-main-57f1de-openshift-default-77c5d6878fqjd27\" (UID: \"1ed76266-22d9-4ec7-bc9e-8f84355b3485\") " pod="openshift-ingress/e2e-worker-single-main-57f1de-openshift-default-77c5d6878fqjd27" Oct 30 17:24:10.339440 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:10.339417 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ed76266-22d9-4ec7-bc9e-8f84355b3485-istio-envoy\") pod \"e2e-worker-single-main-57f1de-openshift-default-77c5d6878fqjd27\" (UID: \"1ed76266-22d9-4ec7-bc9e-8f84355b3485\") " pod="openshift-ingress/e2e-worker-single-main-57f1de-openshift-default-77c5d6878fqjd27" Oct 30 17:24:10.339563 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:10.339526 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ed76266-22d9-4ec7-bc9e-8f84355b3485-istio-podinfo\") pod \"e2e-worker-single-main-57f1de-openshift-default-77c5d6878fqjd27\" (UID: \"1ed76266-22d9-4ec7-bc9e-8f84355b3485\") " pod="openshift-ingress/e2e-worker-single-main-57f1de-openshift-default-77c5d6878fqjd27" Oct 30 17:24:10.358806 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:10.358780 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ed76266-22d9-4ec7-bc9e-8f84355b3485-istio-token\") pod \"e2e-worker-single-main-57f1de-openshift-default-77c5d6878fqjd27\" (UID: \"1ed76266-22d9-4ec7-bc9e-8f84355b3485\") " pod="openshift-ingress/e2e-worker-single-main-57f1de-openshift-default-77c5d6878fqjd27" Oct 30 17:24:10.358980 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:10.358963 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9mp6\" (UniqueName: \"kubernetes.io/projected/1ed76266-22d9-4ec7-bc9e-8f84355b3485-kube-api-access-q9mp6\") pod \"e2e-worker-single-main-57f1de-openshift-default-77c5d6878fqjd27\" (UID: \"1ed76266-22d9-4ec7-bc9e-8f84355b3485\") " pod="openshift-ingress/e2e-worker-single-main-57f1de-openshift-default-77c5d6878fqjd27" Oct 30 17:24:10.454052 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:10.454008 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-57f1de-openshift-default-77c5d6878fqjd27" Oct 30 17:24:10.770680 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:10.770590 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-57f1de-openshift-default-77c5d6878fqjd27"] Oct 30 17:24:10.775081 ip-10-0-131-136 kubenswrapper[5052]: W1030 17:24:10.775052 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ed76266_22d9_4ec7_bc9e_8f84355b3485.slice/crio-4879337a7b024adc9c45466210b2425ce5175d1c56caaa16308ecea094d5bf83 WatchSource:0}: Error finding container 4879337a7b024adc9c45466210b2425ce5175d1c56caaa16308ecea094d5bf83: Status 404 returned error can't find the container with id 4879337a7b024adc9c45466210b2425ce5175d1c56caaa16308ecea094d5bf83 Oct 30 17:24:10.777170 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:10.777132 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:24:10.777259 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:10.777222 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:24:10.777259 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:10.777252 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:24:11.453121 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:11.453083 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-57f1de-openshift-default-77c5d6878fqjd27" event={"ID":"1ed76266-22d9-4ec7-bc9e-8f84355b3485","Type":"ContainerStarted","Data":"813286bc891d79dfba4eace2f817569d076169866ab98b4543bfc0f57e938d75"} Oct 30 17:24:11.453704 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:11.453149 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-57f1de-openshift-default-77c5d6878fqjd27" event={"ID":"1ed76266-22d9-4ec7-bc9e-8f84355b3485","Type":"ContainerStarted","Data":"4879337a7b024adc9c45466210b2425ce5175d1c56caaa16308ecea094d5bf83"} Oct 30 17:24:11.454477 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:11.454445 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-57f1de-openshift-default-77c5d6878fqjd27" Oct 30 17:24:11.459985 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:11.459957 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-57f1de-openshift-default-77c5d6878fqjd27" Oct 30 17:24:11.509380 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:11.509334 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-57f1de-openshift-default-77c5d6878fqjd27" podStartSLOduration=1.509321334 podStartE2EDuration="1.509321334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:24:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:24:11.507318886 +0000 UTC m=+3807.034654124" watchObservedRunningTime="2026-10-30 17:24:11.509321334 +0000 UTC m=+3807.036656623" Oct 30 17:24:12.457821 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:12.457788 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-57f1de-openshift-default-77c5d6878fqjd27" Oct 30 17:24:12.458706 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:12.458688 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-57f1de-openshift-default-77c5d6878fqjd27" Oct 30 17:24:15.266063 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:15.266025 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-57f1de-67899596dc-7h5v4"] Oct 30 17:24:15.270218 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:15.270194 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-57f1de-67899596dc-7h5v4" Oct 30 17:24:15.285562 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:15.285535 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d81b687a-7327-4433-975a-a28e5f9c2780-metrics-certs\") pod \"maas-api-e2e-worker-single-main-57f1de-67899596dc-7h5v4\" (UID: \"d81b687a-7327-4433-975a-a28e5f9c2780\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-57f1de-67899596dc-7h5v4" Oct 30 17:24:15.285735 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:15.285676 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qn6g6\" (UniqueName: \"kubernetes.io/projected/d81b687a-7327-4433-975a-a28e5f9c2780-kube-api-access-qn6g6\") pod \"maas-api-e2e-worker-single-main-57f1de-67899596dc-7h5v4\" (UID: \"d81b687a-7327-4433-975a-a28e5f9c2780\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-57f1de-67899596dc-7h5v4" Oct 30 17:24:15.285735 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:15.285708 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d81b687a-7327-4433-975a-a28e5f9c2780-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-57f1de-67899596dc-7h5v4\" (UID: \"d81b687a-7327-4433-975a-a28e5f9c2780\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-57f1de-67899596dc-7h5v4" Oct 30 17:24:15.285735 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:15.285731 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d81b687a-7327-4433-975a-a28e5f9c2780-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-57f1de-67899596dc-7h5v4\" (UID: \"d81b687a-7327-4433-975a-a28e5f9c2780\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-57f1de-67899596dc-7h5v4" Oct 30 17:24:15.299881 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:15.299848 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-57f1de-serving-cert\"" Oct 30 17:24:15.302370 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:15.302338 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-57f1de-67899596dc-7h5v4"] Oct 30 17:24:15.386955 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:15.386918 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d81b687a-7327-4433-975a-a28e5f9c2780-metrics-certs\") pod \"maas-api-e2e-worker-single-main-57f1de-67899596dc-7h5v4\" (UID: \"d81b687a-7327-4433-975a-a28e5f9c2780\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-57f1de-67899596dc-7h5v4" Oct 30 17:24:15.387148 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:15.387005 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn6g6\" (UniqueName: \"kubernetes.io/projected/d81b687a-7327-4433-975a-a28e5f9c2780-kube-api-access-qn6g6\") pod \"maas-api-e2e-worker-single-main-57f1de-67899596dc-7h5v4\" (UID: \"d81b687a-7327-4433-975a-a28e5f9c2780\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-57f1de-67899596dc-7h5v4" Oct 30 17:24:15.387148 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:15.387028 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d81b687a-7327-4433-975a-a28e5f9c2780-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-57f1de-67899596dc-7h5v4\" (UID: \"d81b687a-7327-4433-975a-a28e5f9c2780\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-57f1de-67899596dc-7h5v4" Oct 30 17:24:15.387148 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:15.387053 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d81b687a-7327-4433-975a-a28e5f9c2780-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-57f1de-67899596dc-7h5v4\" (UID: \"d81b687a-7327-4433-975a-a28e5f9c2780\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-57f1de-67899596dc-7h5v4" Oct 30 17:24:15.387680 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:15.387655 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d81b687a-7327-4433-975a-a28e5f9c2780-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-57f1de-67899596dc-7h5v4\" (UID: \"d81b687a-7327-4433-975a-a28e5f9c2780\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-57f1de-67899596dc-7h5v4" Oct 30 17:24:15.389386 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:15.389361 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d81b687a-7327-4433-975a-a28e5f9c2780-metrics-certs\") pod \"maas-api-e2e-worker-single-main-57f1de-67899596dc-7h5v4\" (UID: \"d81b687a-7327-4433-975a-a28e5f9c2780\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-57f1de-67899596dc-7h5v4" Oct 30 17:24:15.389497 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:15.389399 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d81b687a-7327-4433-975a-a28e5f9c2780-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-57f1de-67899596dc-7h5v4\" (UID: \"d81b687a-7327-4433-975a-a28e5f9c2780\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-57f1de-67899596dc-7h5v4" Oct 30 17:24:15.410514 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:15.410486 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn6g6\" (UniqueName: \"kubernetes.io/projected/d81b687a-7327-4433-975a-a28e5f9c2780-kube-api-access-qn6g6\") pod \"maas-api-e2e-worker-single-main-57f1de-67899596dc-7h5v4\" (UID: \"d81b687a-7327-4433-975a-a28e5f9c2780\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-57f1de-67899596dc-7h5v4" Oct 30 17:24:15.584035 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:15.583953 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-57f1de-67899596dc-7h5v4" Oct 30 17:24:15.689776 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:15.689740 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-57f1de-66c5bbc45fd7cb"] Oct 30 17:24:15.693956 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:15.693932 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-57f1de-66c5bbc45fd7cb" Oct 30 17:24:15.710084 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:15.710019 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-57f1de\"" Oct 30 17:24:15.710297 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:15.710274 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-6a8e3743-dockercfg-vrwhd\"" Oct 30 17:24:15.713383 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:15.713358 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-57f1de-66c5bbc45fd7cb"] Oct 30 17:24:15.747892 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:15.747861 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-57f1de-dc7474p26p"] Oct 30 17:24:15.751480 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:15.751462 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-57f1de-dc7474p26p" Oct 30 17:24:15.772115 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:15.772089 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-57f1de-dc7474p26p"] Oct 30 17:24:15.791955 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:15.791779 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57cbccbe-ea6d-4699-a560-5d44ee5a9915-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-57f1de-66c5bbc45fd7cb\" (UID: \"57cbccbe-ea6d-4699-a560-5d44ee5a9915\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-57f1de-66c5bbc45fd7cb" Oct 30 17:24:15.791955 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:15.791855 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6k594\" (UniqueName: \"kubernetes.io/projected/f9770be6-158c-4dc1-ab63-d06993212c6c-kube-api-access-6k594\") pod \"payload-pre-processing-e2e-worker-single-main-57f1de-dc7474p26p\" (UID: \"f9770be6-158c-4dc1-ab63-d06993212c6c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-57f1de-dc7474p26p" Oct 30 17:24:15.791955 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:15.791943 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pws5\" (UniqueName: \"kubernetes.io/projected/57cbccbe-ea6d-4699-a560-5d44ee5a9915-kube-api-access-7pws5\") pod \"payload-processing-e2e-worker-single-main-57f1de-66c5bbc45fd7cb\" (UID: \"57cbccbe-ea6d-4699-a560-5d44ee5a9915\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-57f1de-66c5bbc45fd7cb" Oct 30 17:24:15.796203 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:15.791998 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9770be6-158c-4dc1-ab63-d06993212c6c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-57f1de-dc7474p26p\" (UID: \"f9770be6-158c-4dc1-ab63-d06993212c6c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-57f1de-dc7474p26p" Oct 30 17:24:15.893053 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:15.892980 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57cbccbe-ea6d-4699-a560-5d44ee5a9915-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-57f1de-66c5bbc45fd7cb\" (UID: \"57cbccbe-ea6d-4699-a560-5d44ee5a9915\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-57f1de-66c5bbc45fd7cb" Oct 30 17:24:15.893053 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:15.893034 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6k594\" (UniqueName: \"kubernetes.io/projected/f9770be6-158c-4dc1-ab63-d06993212c6c-kube-api-access-6k594\") pod \"payload-pre-processing-e2e-worker-single-main-57f1de-dc7474p26p\" (UID: \"f9770be6-158c-4dc1-ab63-d06993212c6c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-57f1de-dc7474p26p" Oct 30 17:24:15.893272 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:15.893095 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pws5\" (UniqueName: \"kubernetes.io/projected/57cbccbe-ea6d-4699-a560-5d44ee5a9915-kube-api-access-7pws5\") pod \"payload-processing-e2e-worker-single-main-57f1de-66c5bbc45fd7cb\" (UID: \"57cbccbe-ea6d-4699-a560-5d44ee5a9915\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-57f1de-66c5bbc45fd7cb" Oct 30 17:24:15.893335 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:15.893282 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9770be6-158c-4dc1-ab63-d06993212c6c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-57f1de-dc7474p26p\" (UID: \"f9770be6-158c-4dc1-ab63-d06993212c6c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-57f1de-dc7474p26p" Oct 30 17:24:15.893940 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:15.893914 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57cbccbe-ea6d-4699-a560-5d44ee5a9915-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-57f1de-66c5bbc45fd7cb\" (UID: \"57cbccbe-ea6d-4699-a560-5d44ee5a9915\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-57f1de-66c5bbc45fd7cb" Oct 30 17:24:15.894082 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:15.894066 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9770be6-158c-4dc1-ab63-d06993212c6c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-57f1de-dc7474p26p\" (UID: \"f9770be6-158c-4dc1-ab63-d06993212c6c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-57f1de-dc7474p26p" Oct 30 17:24:15.897269 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:15.897243 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-57f1de-67899596dc-7h5v4"] Oct 30 17:24:15.901526 ip-10-0-131-136 kubenswrapper[5052]: W1030 17:24:15.901503 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd81b687a_7327_4433_975a_a28e5f9c2780.slice/crio-6a6bf90de7d797d53887690e84a7bee4b2ed22b29b01de4d61a9bb07878cbac8 WatchSource:0}: Error finding container 6a6bf90de7d797d53887690e84a7bee4b2ed22b29b01de4d61a9bb07878cbac8: Status 404 returned error can't find the container with id 6a6bf90de7d797d53887690e84a7bee4b2ed22b29b01de4d61a9bb07878cbac8 Oct 30 17:24:15.902841 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:15.902825 5052 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 17:24:15.916915 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:15.916891 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6k594\" (UniqueName: \"kubernetes.io/projected/f9770be6-158c-4dc1-ab63-d06993212c6c-kube-api-access-6k594\") pod \"payload-pre-processing-e2e-worker-single-main-57f1de-dc7474p26p\" (UID: \"f9770be6-158c-4dc1-ab63-d06993212c6c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-57f1de-dc7474p26p" Oct 30 17:24:15.917062 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:15.917046 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pws5\" (UniqueName: \"kubernetes.io/projected/57cbccbe-ea6d-4699-a560-5d44ee5a9915-kube-api-access-7pws5\") pod \"payload-processing-e2e-worker-single-main-57f1de-66c5bbc45fd7cb\" (UID: \"57cbccbe-ea6d-4699-a560-5d44ee5a9915\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-57f1de-66c5bbc45fd7cb" Oct 30 17:24:16.007910 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:16.007881 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-57f1de-66c5bbc45fd7cb" Oct 30 17:24:16.068950 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:16.068920 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-57f1de-dc7474p26p" Oct 30 17:24:16.308292 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:16.308217 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-57f1de-66c5bbc45fd7cb"] Oct 30 17:24:16.312101 ip-10-0-131-136 kubenswrapper[5052]: W1030 17:24:16.312077 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57cbccbe_ea6d_4699_a560_5d44ee5a9915.slice/crio-a04be8785196013052a878932aed721fb0d6ecbeb5a3c33b853608dbc9802452 WatchSource:0}: Error finding container a04be8785196013052a878932aed721fb0d6ecbeb5a3c33b853608dbc9802452: Status 404 returned error can't find the container with id a04be8785196013052a878932aed721fb0d6ecbeb5a3c33b853608dbc9802452 Oct 30 17:24:16.373633 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:16.373607 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-57f1de-dc7474p26p"] Oct 30 17:24:16.377492 ip-10-0-131-136 kubenswrapper[5052]: W1030 17:24:16.377467 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9770be6_158c_4dc1_ab63_d06993212c6c.slice/crio-797a19d145723ac4f3bd167c600025a6e0c1d99b5f57f23d3b3080e690114608 WatchSource:0}: Error finding container 797a19d145723ac4f3bd167c600025a6e0c1d99b5f57f23d3b3080e690114608: Status 404 returned error can't find the container with id 797a19d145723ac4f3bd167c600025a6e0c1d99b5f57f23d3b3080e690114608 Oct 30 17:24:16.478728 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:16.478698 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-57f1de-66c5bbc45fd7cb" event={"ID":"57cbccbe-ea6d-4699-a560-5d44ee5a9915","Type":"ContainerStarted","Data":"a6bbfbfe1f2704f466a530199ae96cf5761943e2384e59d01830a72dc9f5f6f2"} Oct 30 17:24:16.478879 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:16.478759 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-57f1de-66c5bbc45fd7cb" event={"ID":"57cbccbe-ea6d-4699-a560-5d44ee5a9915","Type":"ContainerStarted","Data":"a04be8785196013052a878932aed721fb0d6ecbeb5a3c33b853608dbc9802452"} Oct 30 17:24:16.478964 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:16.478907 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-57f1de-66c5bbc45fd7cb" Oct 30 17:24:16.480208 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:16.480183 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-57f1de-dc7474p26p" event={"ID":"f9770be6-158c-4dc1-ab63-d06993212c6c","Type":"ContainerStarted","Data":"9d2df79ae5eab609a3241ce0ae7c24ba5d4676e4a6f6ec8a9aae989d882b9282"} Oct 30 17:24:16.480323 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:16.480230 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-57f1de-dc7474p26p" event={"ID":"f9770be6-158c-4dc1-ab63-d06993212c6c","Type":"ContainerStarted","Data":"797a19d145723ac4f3bd167c600025a6e0c1d99b5f57f23d3b3080e690114608"} Oct 30 17:24:16.480323 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:16.480260 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-57f1de-dc7474p26p" Oct 30 17:24:16.481756 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:16.481733 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-57f1de-67899596dc-7h5v4" event={"ID":"d81b687a-7327-4433-975a-a28e5f9c2780","Type":"ContainerStarted","Data":"9deec8afe44ceb10bf1b04f9f15cd2c34ac1ff632e3a21156dff854c733c5c62"} Oct 30 17:24:16.481841 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:16.481769 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-57f1de-67899596dc-7h5v4" event={"ID":"d81b687a-7327-4433-975a-a28e5f9c2780","Type":"ContainerStarted","Data":"6a6bf90de7d797d53887690e84a7bee4b2ed22b29b01de4d61a9bb07878cbac8"} Oct 30 17:24:16.481841 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:16.481789 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-57f1de-67899596dc-7h5v4" Oct 30 17:24:16.519221 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:16.519095 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-57f1de-66c5bbc45fd7cb" podStartSLOduration=1.5190757559999999 podStartE2EDuration="1.519075756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:24:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:24:16.516515973 +0000 UTC m=+3812.043851192" watchObservedRunningTime="2026-10-30 17:24:16.519075756 +0000 UTC m=+3812.046410977" Oct 30 17:24:16.558124 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:16.558071 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-57f1de-67899596dc-7h5v4" podStartSLOduration=1.212416307 podStartE2EDuration="1.558022265s" totalImagesPullingTime="345.605958ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:24:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:24:16.55609861 +0000 UTC m=+3812.083433842" watchObservedRunningTime="2026-10-30 17:24:16.558022265 +0000 UTC m=+3812.085357483" Oct 30 17:24:16.603964 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:16.603909 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-57f1de-dc7474p26p" podStartSLOduration=1.603896132 podStartE2EDuration="1.603896132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:24:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:24:16.601659561 +0000 UTC m=+3812.128994777" watchObservedRunningTime="2026-10-30 17:24:16.603896132 +0000 UTC m=+3812.131231351" Oct 30 17:24:18.702346 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:18.702298 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cdwcj"] Oct 30 17:24:18.847935 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:24:18.847883 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 17:24:22.492555 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:22.492521 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-57f1de-67899596dc-7h5v4" Oct 30 17:24:27.488961 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:27.488926 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-57f1de-dc7474p26p" Oct 30 17:24:27.489450 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:27.489020 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-57f1de-66c5bbc45fd7cb" Oct 30 17:24:28.321383 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:28.320328 5052 patch_prober.go:29] interesting pod/e2e-worker-single-main-57f1de-openshift-default-77c5d6878fqjd27 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.238:15021/healthz/ready\": read tcp 10.132.0.2:48596->10.132.0.238:15021: read: connection reset by peer" start-of-body= Oct 30 17:24:28.321383 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:28.320405 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-57f1de-openshift-default-77c5d6878fqjd27" podUID="1ed76266-22d9-4ec7-bc9e-8f84355b3485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.238:15021/healthz/ready\": read tcp 10.132.0.2:48596->10.132.0.238:15021: read: connection reset by peer" Oct 30 17:24:28.547359 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:28.547281 5052 generic.go:356] "Generic (PLEG): container finished" podID="1ed76266-22d9-4ec7-bc9e-8f84355b3485" containerID="813286bc891d79dfba4eace2f817569d076169866ab98b4543bfc0f57e938d75" exitCode=0 Oct 30 17:24:28.547777 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:28.547356 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-57f1de-openshift-default-77c5d6878fqjd27" event={"ID":"1ed76266-22d9-4ec7-bc9e-8f84355b3485","Type":"ContainerDied","Data":"813286bc891d79dfba4eace2f817569d076169866ab98b4543bfc0f57e938d75"} Oct 30 17:24:28.547777 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:28.547740 5052 scope.go:122] "RemoveContainer" containerID="813286bc891d79dfba4eace2f817569d076169866ab98b4543bfc0f57e938d75" Oct 30 17:24:28.548939 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:28.548911 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:24:28.549012 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:28.548982 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:24:28.549056 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:28.549014 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:24:29.561104 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:29.561066 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-57f1de-openshift-default-77c5d6878fqjd27" event={"ID":"1ed76266-22d9-4ec7-bc9e-8f84355b3485","Type":"ContainerStarted","Data":"1b0a95df9a6f1e0aba896205e8a077f6642dae03348f0c0024265b98badf96ef"} Oct 30 17:24:30.455232 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:30.455193 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-57f1de-openshift-default-77c5d6878fqjd27" Oct 30 17:24:30.455522 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:30.455491 5052 patch_prober.go:29] interesting pod/e2e-worker-single-main-57f1de-openshift-default-77c5d6878fqjd27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" start-of-body= Oct 30 17:24:30.455605 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:30.455556 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-57f1de-openshift-default-77c5d6878fqjd27" podUID="1ed76266-22d9-4ec7-bc9e-8f84355b3485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" Oct 30 17:24:31.159333 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:31.159286 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-57f1de/simulator-single-main-57f1de-kserve-859d6c4b4d-k8x5n"] Oct 30 17:24:31.163513 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:31.163491 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-57f1de/simulator-single-main-57f1de-kserve-859d6c4b4d-k8x5n" Oct 30 17:24:31.183722 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:31.183691 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-57f1de/simulator-single-main-57f1de-kserve-859d6c4b4d-k8x5n"] Oct 30 17:24:31.192072 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:31.192048 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-57f1de\"/\"openshift-service-ca.crt\"" Oct 30 17:24:31.192204 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:31.192093 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-57f1de\"/\"kube-root-ca.crt\"" Oct 30 17:24:31.195436 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:31.195418 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-57f1de\"/\"simulator-single-main-57f1de-kserve-self-signed-certs\"" Oct 30 17:24:31.195545 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:31.195480 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-57f1de\"/\"default-dockercfg-d4m76\"" Oct 30 17:24:31.279778 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:31.279748 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b27fdf08-0633-4e1b-8e8b-102e27c7cb20-model-cache\") pod \"simulator-single-main-57f1de-kserve-859d6c4b4d-k8x5n\" (UID: \"b27fdf08-0633-4e1b-8e8b-102e27c7cb20\") " pod="e2e-models-e2e-worker-single-main-57f1de/simulator-single-main-57f1de-kserve-859d6c4b4d-k8x5n" Oct 30 17:24:31.279965 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:31.279788 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b27fdf08-0633-4e1b-8e8b-102e27c7cb20-dshm\") pod \"simulator-single-main-57f1de-kserve-859d6c4b4d-k8x5n\" (UID: \"b27fdf08-0633-4e1b-8e8b-102e27c7cb20\") " pod="e2e-models-e2e-worker-single-main-57f1de/simulator-single-main-57f1de-kserve-859d6c4b4d-k8x5n" Oct 30 17:24:31.279965 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:31.279908 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4t49l\" (UniqueName: \"kubernetes.io/projected/b27fdf08-0633-4e1b-8e8b-102e27c7cb20-kube-api-access-4t49l\") pod \"simulator-single-main-57f1de-kserve-859d6c4b4d-k8x5n\" (UID: \"b27fdf08-0633-4e1b-8e8b-102e27c7cb20\") " pod="e2e-models-e2e-worker-single-main-57f1de/simulator-single-main-57f1de-kserve-859d6c4b4d-k8x5n" Oct 30 17:24:31.279965 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:31.279958 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b27fdf08-0633-4e1b-8e8b-102e27c7cb20-tmp-dir\") pod \"simulator-single-main-57f1de-kserve-859d6c4b4d-k8x5n\" (UID: \"b27fdf08-0633-4e1b-8e8b-102e27c7cb20\") " pod="e2e-models-e2e-worker-single-main-57f1de/simulator-single-main-57f1de-kserve-859d6c4b4d-k8x5n" Oct 30 17:24:31.280152 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:31.280069 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b27fdf08-0633-4e1b-8e8b-102e27c7cb20-home\") pod \"simulator-single-main-57f1de-kserve-859d6c4b4d-k8x5n\" (UID: \"b27fdf08-0633-4e1b-8e8b-102e27c7cb20\") " pod="e2e-models-e2e-worker-single-main-57f1de/simulator-single-main-57f1de-kserve-859d6c4b4d-k8x5n" Oct 30 17:24:31.280152 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:31.280132 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b27fdf08-0633-4e1b-8e8b-102e27c7cb20-tls-certs\") pod \"simulator-single-main-57f1de-kserve-859d6c4b4d-k8x5n\" (UID: \"b27fdf08-0633-4e1b-8e8b-102e27c7cb20\") " pod="e2e-models-e2e-worker-single-main-57f1de/simulator-single-main-57f1de-kserve-859d6c4b4d-k8x5n" Oct 30 17:24:31.381703 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:31.381666 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b27fdf08-0633-4e1b-8e8b-102e27c7cb20-tls-certs\") pod \"simulator-single-main-57f1de-kserve-859d6c4b4d-k8x5n\" (UID: \"b27fdf08-0633-4e1b-8e8b-102e27c7cb20\") " pod="e2e-models-e2e-worker-single-main-57f1de/simulator-single-main-57f1de-kserve-859d6c4b4d-k8x5n" Oct 30 17:24:31.381919 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:31.381741 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b27fdf08-0633-4e1b-8e8b-102e27c7cb20-model-cache\") pod \"simulator-single-main-57f1de-kserve-859d6c4b4d-k8x5n\" (UID: \"b27fdf08-0633-4e1b-8e8b-102e27c7cb20\") " pod="e2e-models-e2e-worker-single-main-57f1de/simulator-single-main-57f1de-kserve-859d6c4b4d-k8x5n" Oct 30 17:24:31.381919 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:31.381775 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b27fdf08-0633-4e1b-8e8b-102e27c7cb20-dshm\") pod \"simulator-single-main-57f1de-kserve-859d6c4b4d-k8x5n\" (UID: \"b27fdf08-0633-4e1b-8e8b-102e27c7cb20\") " pod="e2e-models-e2e-worker-single-main-57f1de/simulator-single-main-57f1de-kserve-859d6c4b4d-k8x5n" Oct 30 17:24:31.381919 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:31.381845 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4t49l\" (UniqueName: \"kubernetes.io/projected/b27fdf08-0633-4e1b-8e8b-102e27c7cb20-kube-api-access-4t49l\") pod \"simulator-single-main-57f1de-kserve-859d6c4b4d-k8x5n\" (UID: \"b27fdf08-0633-4e1b-8e8b-102e27c7cb20\") " pod="e2e-models-e2e-worker-single-main-57f1de/simulator-single-main-57f1de-kserve-859d6c4b4d-k8x5n" Oct 30 17:24:31.381919 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:31.381890 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b27fdf08-0633-4e1b-8e8b-102e27c7cb20-tmp-dir\") pod \"simulator-single-main-57f1de-kserve-859d6c4b4d-k8x5n\" (UID: \"b27fdf08-0633-4e1b-8e8b-102e27c7cb20\") " pod="e2e-models-e2e-worker-single-main-57f1de/simulator-single-main-57f1de-kserve-859d6c4b4d-k8x5n" Oct 30 17:24:31.382190 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:31.381923 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b27fdf08-0633-4e1b-8e8b-102e27c7cb20-home\") pod \"simulator-single-main-57f1de-kserve-859d6c4b4d-k8x5n\" (UID: \"b27fdf08-0633-4e1b-8e8b-102e27c7cb20\") " pod="e2e-models-e2e-worker-single-main-57f1de/simulator-single-main-57f1de-kserve-859d6c4b4d-k8x5n" Oct 30 17:24:31.382258 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:31.382237 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b27fdf08-0633-4e1b-8e8b-102e27c7cb20-model-cache\") pod \"simulator-single-main-57f1de-kserve-859d6c4b4d-k8x5n\" (UID: \"b27fdf08-0633-4e1b-8e8b-102e27c7cb20\") " pod="e2e-models-e2e-worker-single-main-57f1de/simulator-single-main-57f1de-kserve-859d6c4b4d-k8x5n" Oct 30 17:24:31.382337 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:31.382285 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b27fdf08-0633-4e1b-8e8b-102e27c7cb20-home\") pod \"simulator-single-main-57f1de-kserve-859d6c4b4d-k8x5n\" (UID: \"b27fdf08-0633-4e1b-8e8b-102e27c7cb20\") " pod="e2e-models-e2e-worker-single-main-57f1de/simulator-single-main-57f1de-kserve-859d6c4b4d-k8x5n" Oct 30 17:24:31.383023 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:31.382995 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b27fdf08-0633-4e1b-8e8b-102e27c7cb20-tmp-dir\") pod \"simulator-single-main-57f1de-kserve-859d6c4b4d-k8x5n\" (UID: \"b27fdf08-0633-4e1b-8e8b-102e27c7cb20\") " pod="e2e-models-e2e-worker-single-main-57f1de/simulator-single-main-57f1de-kserve-859d6c4b4d-k8x5n" Oct 30 17:24:31.385043 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:31.385020 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b27fdf08-0633-4e1b-8e8b-102e27c7cb20-dshm\") pod \"simulator-single-main-57f1de-kserve-859d6c4b4d-k8x5n\" (UID: \"b27fdf08-0633-4e1b-8e8b-102e27c7cb20\") " pod="e2e-models-e2e-worker-single-main-57f1de/simulator-single-main-57f1de-kserve-859d6c4b4d-k8x5n" Oct 30 17:24:31.386698 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:31.386673 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b27fdf08-0633-4e1b-8e8b-102e27c7cb20-tls-certs\") pod \"simulator-single-main-57f1de-kserve-859d6c4b4d-k8x5n\" (UID: \"b27fdf08-0633-4e1b-8e8b-102e27c7cb20\") " pod="e2e-models-e2e-worker-single-main-57f1de/simulator-single-main-57f1de-kserve-859d6c4b4d-k8x5n" Oct 30 17:24:31.406775 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:31.406747 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4t49l\" (UniqueName: \"kubernetes.io/projected/b27fdf08-0633-4e1b-8e8b-102e27c7cb20-kube-api-access-4t49l\") pod \"simulator-single-main-57f1de-kserve-859d6c4b4d-k8x5n\" (UID: \"b27fdf08-0633-4e1b-8e8b-102e27c7cb20\") " pod="e2e-models-e2e-worker-single-main-57f1de/simulator-single-main-57f1de-kserve-859d6c4b4d-k8x5n" Oct 30 17:24:31.454411 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:31.454322 5052 patch_prober.go:29] interesting pod/e2e-worker-single-main-57f1de-openshift-default-77c5d6878fqjd27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" start-of-body= Oct 30 17:24:31.454411 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:31.454372 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-57f1de-openshift-default-77c5d6878fqjd27" podUID="1ed76266-22d9-4ec7-bc9e-8f84355b3485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" Oct 30 17:24:31.476106 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:31.476073 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-57f1de/simulator-single-main-57f1de-kserve-859d6c4b4d-k8x5n" Oct 30 17:24:31.795121 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:31.795035 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-57f1de/simulator-single-main-57f1de-kserve-859d6c4b4d-k8x5n"] Oct 30 17:24:31.798767 ip-10-0-131-136 kubenswrapper[5052]: W1030 17:24:31.798741 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb27fdf08_0633_4e1b_8e8b_102e27c7cb20.slice/crio-8d29deed86eddb953b6fcd0c5396b811c818b0476498d5d4225b842447a6fa71 WatchSource:0}: Error finding container 8d29deed86eddb953b6fcd0c5396b811c818b0476498d5d4225b842447a6fa71: Status 404 returned error can't find the container with id 8d29deed86eddb953b6fcd0c5396b811c818b0476498d5d4225b842447a6fa71 Oct 30 17:24:32.454931 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:32.454894 5052 patch_prober.go:29] interesting pod/e2e-worker-single-main-57f1de-openshift-default-77c5d6878fqjd27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" start-of-body= Oct 30 17:24:32.455362 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:32.454950 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-57f1de-openshift-default-77c5d6878fqjd27" podUID="1ed76266-22d9-4ec7-bc9e-8f84355b3485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" Oct 30 17:24:32.580894 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:32.580856 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-57f1de/simulator-single-main-57f1de-kserve-859d6c4b4d-k8x5n" event={"ID":"b27fdf08-0633-4e1b-8e8b-102e27c7cb20","Type":"ContainerStarted","Data":"955fe7da56ce13cb26cfd0da5e8a421b101bf96236bd447e213b6eaeb5df0303"} Oct 30 17:24:32.580894 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:32.580900 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-57f1de/simulator-single-main-57f1de-kserve-859d6c4b4d-k8x5n" event={"ID":"b27fdf08-0633-4e1b-8e8b-102e27c7cb20","Type":"ContainerStarted","Data":"8d29deed86eddb953b6fcd0c5396b811c818b0476498d5d4225b842447a6fa71"} Oct 30 17:24:32.622616 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:32.622531 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-57f1de/simulator-single-main-57f1de-kserve-859d6c4b4d-k8x5n" podStartSLOduration=1.622509656 podStartE2EDuration="1.622509656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:24:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:24:32.620476636 +0000 UTC m=+3828.147811859" watchObservedRunningTime="2026-10-30 17:24:32.622509656 +0000 UTC m=+3828.149844877" Oct 30 17:24:33.339605 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:33.338383 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cdwcj"] Oct 30 17:24:33.455316 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:33.455277 5052 patch_prober.go:29] interesting pod/e2e-worker-single-main-57f1de-openshift-default-77c5d6878fqjd27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" start-of-body= Oct 30 17:24:33.455822 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:33.455341 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-57f1de-openshift-default-77c5d6878fqjd27" podUID="1ed76266-22d9-4ec7-bc9e-8f84355b3485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" Oct 30 17:24:33.476317 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:33.476292 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-57f1de/simulator-single-main-57f1de-kserve-859d6c4b4d-k8x5n" Oct 30 17:24:34.455358 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:34.455316 5052 patch_prober.go:29] interesting pod/e2e-worker-single-main-57f1de-openshift-default-77c5d6878fqjd27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" start-of-body= Oct 30 17:24:34.455808 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:34.455378 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-57f1de-openshift-default-77c5d6878fqjd27" podUID="1ed76266-22d9-4ec7-bc9e-8f84355b3485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" Oct 30 17:24:35.459086 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:35.459048 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-57f1de-openshift-default-77c5d6878fqjd27" Oct 30 17:24:35.459538 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:35.459343 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-57f1de-openshift-default-77c5d6878fqjd27" Oct 30 17:24:35.460059 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:35.460038 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-57f1de-openshift-default-77c5d6878fqjd27" Oct 30 17:24:41.476976 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:41.476941 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-57f1de/simulator-single-main-57f1de-kserve-859d6c4b4d-k8x5n" Oct 30 17:24:41.489223 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:41.489198 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-57f1de/simulator-single-main-57f1de-kserve-859d6c4b4d-k8x5n" Oct 30 17:24:41.649425 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:41.649395 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-57f1de/simulator-single-main-57f1de-kserve-859d6c4b4d-k8x5n" Oct 30 17:24:48.778805 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:48.778769 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cdwcj"] Oct 30 17:24:48.842496 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:24:48.842460 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 17:24:50.388628 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:24:50.388573 5052 scope.go:122] "RemoveContainer" containerID="06956a75e919ebef5178713769eb881776407ec7cbcacc858346cbac2061a860" Oct 30 17:25:04.314351 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:04.314313 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cdwcj"] Oct 30 17:25:08.983998 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:08.983960 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-57f1de/simulator-single-main-57f1de-kserve-859d6c4b4d-k8x5n"] Oct 30 17:25:14.582193 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:14.582154 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-57f1de-66c5bbc45fd7cb"] Oct 30 17:25:14.610340 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:14.610307 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-57f1de-dc7474p26p"] Oct 30 17:25:17.532456 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:17.532419 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-57f1de-67899596dc-7h5v4"] Oct 30 17:25:17.533046 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:17.532710 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-57f1de-67899596dc-7h5v4" podUID="d81b687a-7327-4433-975a-a28e5f9c2780" containerName="maas-api" containerID="cri-o://9deec8afe44ceb10bf1b04f9f15cd2c34ac1ff632e3a21156dff854c733c5c62" gracePeriod=30 Oct 30 17:25:18.017290 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:18.017262 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-57f1de-67899596dc-7h5v4" Oct 30 17:25:18.069276 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:18.069196 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d81b687a-7327-4433-975a-a28e5f9c2780-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d81b687a-7327-4433-975a-a28e5f9c2780-maas-api-tls\") pod \"d81b687a-7327-4433-975a-a28e5f9c2780\" (UID: \"d81b687a-7327-4433-975a-a28e5f9c2780\") " Oct 30 17:25:18.069276 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:18.069252 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d81b687a-7327-4433-975a-a28e5f9c2780-kube-api-access-qn6g6\" (UniqueName: \"kubernetes.io/projected/d81b687a-7327-4433-975a-a28e5f9c2780-kube-api-access-qn6g6\") pod \"d81b687a-7327-4433-975a-a28e5f9c2780\" (UID: \"d81b687a-7327-4433-975a-a28e5f9c2780\") " Oct 30 17:25:18.069513 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:18.069323 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d81b687a-7327-4433-975a-a28e5f9c2780-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d81b687a-7327-4433-975a-a28e5f9c2780-openshift-service-ca\") pod \"d81b687a-7327-4433-975a-a28e5f9c2780\" (UID: \"d81b687a-7327-4433-975a-a28e5f9c2780\") " Oct 30 17:25:18.069513 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:18.069411 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d81b687a-7327-4433-975a-a28e5f9c2780-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d81b687a-7327-4433-975a-a28e5f9c2780-metrics-certs\") pod \"d81b687a-7327-4433-975a-a28e5f9c2780\" (UID: \"d81b687a-7327-4433-975a-a28e5f9c2780\") " Oct 30 17:25:18.069968 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:18.069939 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d81b687a-7327-4433-975a-a28e5f9c2780-openshift-service-ca" pod "d81b687a-7327-4433-975a-a28e5f9c2780" (UID: "d81b687a-7327-4433-975a-a28e5f9c2780"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:25:18.071380 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:18.071358 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d81b687a-7327-4433-975a-a28e5f9c2780-maas-api-tls" pod "d81b687a-7327-4433-975a-a28e5f9c2780" (UID: "d81b687a-7327-4433-975a-a28e5f9c2780"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:25:18.071919 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:18.071892 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d81b687a-7327-4433-975a-a28e5f9c2780-metrics-certs" pod "d81b687a-7327-4433-975a-a28e5f9c2780" (UID: "d81b687a-7327-4433-975a-a28e5f9c2780"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:25:18.072037 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:18.071997 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d81b687a-7327-4433-975a-a28e5f9c2780-kube-api-access-qn6g6" pod "d81b687a-7327-4433-975a-a28e5f9c2780" (UID: "d81b687a-7327-4433-975a-a28e5f9c2780"). InnerVolumeSpecName "kube-api-access-qn6g6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:25:18.170407 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:18.170370 5052 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d81b687a-7327-4433-975a-a28e5f9c2780-maas-api-tls\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:25:18.170407 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:18.170402 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qn6g6\" (UniqueName: \"kubernetes.io/projected/d81b687a-7327-4433-975a-a28e5f9c2780-kube-api-access-qn6g6\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:25:18.170407 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:18.170411 5052 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d81b687a-7327-4433-975a-a28e5f9c2780-openshift-service-ca\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:25:18.170706 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:18.170420 5052 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d81b687a-7327-4433-975a-a28e5f9c2780-metrics-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:25:18.833911 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:18.833878 5052 generic.go:356] "Generic (PLEG): container finished" podID="d81b687a-7327-4433-975a-a28e5f9c2780" containerID="9deec8afe44ceb10bf1b04f9f15cd2c34ac1ff632e3a21156dff854c733c5c62" exitCode=0 Oct 30 17:25:18.834402 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:18.833934 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-57f1de-67899596dc-7h5v4" Oct 30 17:25:18.834402 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:18.833964 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-57f1de-67899596dc-7h5v4" event={"ID":"d81b687a-7327-4433-975a-a28e5f9c2780","Type":"ContainerDied","Data":"9deec8afe44ceb10bf1b04f9f15cd2c34ac1ff632e3a21156dff854c733c5c62"} Oct 30 17:25:18.834402 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:18.834020 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-57f1de-67899596dc-7h5v4" event={"ID":"d81b687a-7327-4433-975a-a28e5f9c2780","Type":"ContainerDied","Data":"6a6bf90de7d797d53887690e84a7bee4b2ed22b29b01de4d61a9bb07878cbac8"} Oct 30 17:25:18.834402 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:18.834054 5052 scope.go:122] "RemoveContainer" containerID="9deec8afe44ceb10bf1b04f9f15cd2c34ac1ff632e3a21156dff854c733c5c62" Oct 30 17:25:18.844678 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:25:18.844639 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 17:25:18.846678 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:18.846657 5052 scope.go:122] "RemoveContainer" containerID="9deec8afe44ceb10bf1b04f9f15cd2c34ac1ff632e3a21156dff854c733c5c62" Oct 30 17:25:18.846987 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:25:18.846955 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9deec8afe44ceb10bf1b04f9f15cd2c34ac1ff632e3a21156dff854c733c5c62\": container with ID starting with 9deec8afe44ceb10bf1b04f9f15cd2c34ac1ff632e3a21156dff854c733c5c62 not found: ID does not exist" containerID="9deec8afe44ceb10bf1b04f9f15cd2c34ac1ff632e3a21156dff854c733c5c62" Oct 30 17:25:18.847069 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:18.846999 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9deec8afe44ceb10bf1b04f9f15cd2c34ac1ff632e3a21156dff854c733c5c62"} err="failed to get container status \"9deec8afe44ceb10bf1b04f9f15cd2c34ac1ff632e3a21156dff854c733c5c62\": rpc error: code = NotFound desc = could not find container \"9deec8afe44ceb10bf1b04f9f15cd2c34ac1ff632e3a21156dff854c733c5c62\": container with ID starting with 9deec8afe44ceb10bf1b04f9f15cd2c34ac1ff632e3a21156dff854c733c5c62 not found: ID does not exist" Oct 30 17:25:18.884048 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:18.883986 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-57f1de-67899596dc-7h5v4"] Oct 30 17:25:18.895393 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:18.895368 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-57f1de-67899596dc-7h5v4"] Oct 30 17:25:19.582960 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:19.582923 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-57f1de-66c5bbc45fd7cb" podUID="57cbccbe-ea6d-4699-a560-5d44ee5a9915" containerName="payload-processing" containerID="cri-o://a6bbfbfe1f2704f466a530199ae96cf5761943e2384e59d01830a72dc9f5f6f2" gracePeriod=25 Oct 30 17:25:19.610632 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:19.610553 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-57f1de-dc7474p26p" podUID="f9770be6-158c-4dc1-ab63-d06993212c6c" containerName="payload-pre-processing" containerID="cri-o://9d2df79ae5eab609a3241ce0ae7c24ba5d4676e4a6f6ec8a9aae989d882b9282" gracePeriod=25 Oct 30 17:25:19.921197 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:19.921150 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cdwcj"] Oct 30 17:25:20.597169 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:20.597135 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d81b687a-7327-4433-975a-a28e5f9c2780" path="/var/lib/kubelet/pods/d81b687a-7327-4433-975a-a28e5f9c2780/volumes" Oct 30 17:25:24.004995 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:24.004949 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-57f1de/simulator-single-main-57f1de-kserve-859d6c4b4d-k8x5n" podUID="b27fdf08-0633-4e1b-8e8b-102e27c7cb20" containerName="main" containerID="cri-o://955fe7da56ce13cb26cfd0da5e8a421b101bf96236bd447e213b6eaeb5df0303" gracePeriod=45 Oct 30 17:25:24.271976 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:24.271914 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-57f1de/simulator-single-main-57f1de-kserve-859d6c4b4d-k8x5n" Oct 30 17:25:24.437189 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:24.437155 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b27fdf08-0633-4e1b-8e8b-102e27c7cb20-home\" (UniqueName: \"kubernetes.io/empty-dir/b27fdf08-0633-4e1b-8e8b-102e27c7cb20-home\") pod \"b27fdf08-0633-4e1b-8e8b-102e27c7cb20\" (UID: \"b27fdf08-0633-4e1b-8e8b-102e27c7cb20\") " Oct 30 17:25:24.437402 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:24.437210 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b27fdf08-0633-4e1b-8e8b-102e27c7cb20-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b27fdf08-0633-4e1b-8e8b-102e27c7cb20-dshm\") pod \"b27fdf08-0633-4e1b-8e8b-102e27c7cb20\" (UID: \"b27fdf08-0633-4e1b-8e8b-102e27c7cb20\") " Oct 30 17:25:24.437402 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:24.437244 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b27fdf08-0633-4e1b-8e8b-102e27c7cb20-kube-api-access-4t49l\" (UniqueName: \"kubernetes.io/projected/b27fdf08-0633-4e1b-8e8b-102e27c7cb20-kube-api-access-4t49l\") pod \"b27fdf08-0633-4e1b-8e8b-102e27c7cb20\" (UID: \"b27fdf08-0633-4e1b-8e8b-102e27c7cb20\") " Oct 30 17:25:24.437402 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:24.437280 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b27fdf08-0633-4e1b-8e8b-102e27c7cb20-tls-certs\" (UniqueName: \"kubernetes.io/secret/b27fdf08-0633-4e1b-8e8b-102e27c7cb20-tls-certs\") pod \"b27fdf08-0633-4e1b-8e8b-102e27c7cb20\" (UID: \"b27fdf08-0633-4e1b-8e8b-102e27c7cb20\") " Oct 30 17:25:24.437402 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:24.437314 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b27fdf08-0633-4e1b-8e8b-102e27c7cb20-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b27fdf08-0633-4e1b-8e8b-102e27c7cb20-model-cache\") pod \"b27fdf08-0633-4e1b-8e8b-102e27c7cb20\" (UID: \"b27fdf08-0633-4e1b-8e8b-102e27c7cb20\") " Oct 30 17:25:24.437673 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:24.437437 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b27fdf08-0633-4e1b-8e8b-102e27c7cb20-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b27fdf08-0633-4e1b-8e8b-102e27c7cb20-tmp-dir\") pod \"b27fdf08-0633-4e1b-8e8b-102e27c7cb20\" (UID: \"b27fdf08-0633-4e1b-8e8b-102e27c7cb20\") " Oct 30 17:25:24.437673 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:24.437449 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b27fdf08-0633-4e1b-8e8b-102e27c7cb20-home" pod "b27fdf08-0633-4e1b-8e8b-102e27c7cb20" (UID: "b27fdf08-0633-4e1b-8e8b-102e27c7cb20"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:25:24.437787 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:24.437702 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b27fdf08-0633-4e1b-8e8b-102e27c7cb20-model-cache" pod "b27fdf08-0633-4e1b-8e8b-102e27c7cb20" (UID: "b27fdf08-0633-4e1b-8e8b-102e27c7cb20"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:25:24.437787 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:24.437753 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b27fdf08-0633-4e1b-8e8b-102e27c7cb20-tmp-dir" pod "b27fdf08-0633-4e1b-8e8b-102e27c7cb20" (UID: "b27fdf08-0633-4e1b-8e8b-102e27c7cb20"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:25:24.437880 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:24.437868 5052 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b27fdf08-0633-4e1b-8e8b-102e27c7cb20-home\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:25:24.437927 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:24.437887 5052 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b27fdf08-0633-4e1b-8e8b-102e27c7cb20-model-cache\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:25:24.437927 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:24.437909 5052 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b27fdf08-0633-4e1b-8e8b-102e27c7cb20-tmp-dir\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:25:24.439486 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:24.439465 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b27fdf08-0633-4e1b-8e8b-102e27c7cb20-tls-certs" pod "b27fdf08-0633-4e1b-8e8b-102e27c7cb20" (UID: "b27fdf08-0633-4e1b-8e8b-102e27c7cb20"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:25:24.439884 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:24.439867 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b27fdf08-0633-4e1b-8e8b-102e27c7cb20-dshm" pod "b27fdf08-0633-4e1b-8e8b-102e27c7cb20" (UID: "b27fdf08-0633-4e1b-8e8b-102e27c7cb20"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:25:24.439945 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:24.439925 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b27fdf08-0633-4e1b-8e8b-102e27c7cb20-kube-api-access-4t49l" pod "b27fdf08-0633-4e1b-8e8b-102e27c7cb20" (UID: "b27fdf08-0633-4e1b-8e8b-102e27c7cb20"). InnerVolumeSpecName "kube-api-access-4t49l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:25:24.538880 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:24.538815 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4t49l\" (UniqueName: \"kubernetes.io/projected/b27fdf08-0633-4e1b-8e8b-102e27c7cb20-kube-api-access-4t49l\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:25:24.538880 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:24.538840 5052 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b27fdf08-0633-4e1b-8e8b-102e27c7cb20-tls-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:25:24.538880 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:24.538849 5052 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b27fdf08-0633-4e1b-8e8b-102e27c7cb20-dshm\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:25:24.867935 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:24.867841 5052 generic.go:356] "Generic (PLEG): container finished" podID="b27fdf08-0633-4e1b-8e8b-102e27c7cb20" containerID="955fe7da56ce13cb26cfd0da5e8a421b101bf96236bd447e213b6eaeb5df0303" exitCode=0 Oct 30 17:25:24.867935 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:24.867899 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-57f1de/simulator-single-main-57f1de-kserve-859d6c4b4d-k8x5n" Oct 30 17:25:24.868152 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:24.867929 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-57f1de/simulator-single-main-57f1de-kserve-859d6c4b4d-k8x5n" event={"ID":"b27fdf08-0633-4e1b-8e8b-102e27c7cb20","Type":"ContainerDied","Data":"955fe7da56ce13cb26cfd0da5e8a421b101bf96236bd447e213b6eaeb5df0303"} Oct 30 17:25:24.868152 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:24.867978 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-57f1de/simulator-single-main-57f1de-kserve-859d6c4b4d-k8x5n" event={"ID":"b27fdf08-0633-4e1b-8e8b-102e27c7cb20","Type":"ContainerDied","Data":"8d29deed86eddb953b6fcd0c5396b811c818b0476498d5d4225b842447a6fa71"} Oct 30 17:25:24.868152 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:24.867999 5052 scope.go:122] "RemoveContainer" containerID="955fe7da56ce13cb26cfd0da5e8a421b101bf96236bd447e213b6eaeb5df0303" Oct 30 17:25:24.877484 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:24.877462 5052 scope.go:122] "RemoveContainer" containerID="955fe7da56ce13cb26cfd0da5e8a421b101bf96236bd447e213b6eaeb5df0303" Oct 30 17:25:24.877808 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:25:24.877779 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"955fe7da56ce13cb26cfd0da5e8a421b101bf96236bd447e213b6eaeb5df0303\": container with ID starting with 955fe7da56ce13cb26cfd0da5e8a421b101bf96236bd447e213b6eaeb5df0303 not found: ID does not exist" containerID="955fe7da56ce13cb26cfd0da5e8a421b101bf96236bd447e213b6eaeb5df0303" Oct 30 17:25:24.877919 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:24.877816 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"955fe7da56ce13cb26cfd0da5e8a421b101bf96236bd447e213b6eaeb5df0303"} err="failed to get container status \"955fe7da56ce13cb26cfd0da5e8a421b101bf96236bd447e213b6eaeb5df0303\": rpc error: code = NotFound desc = could not find container \"955fe7da56ce13cb26cfd0da5e8a421b101bf96236bd447e213b6eaeb5df0303\": container with ID starting with 955fe7da56ce13cb26cfd0da5e8a421b101bf96236bd447e213b6eaeb5df0303 not found: ID does not exist" Oct 30 17:25:24.919030 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:24.918991 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-57f1de/simulator-single-main-57f1de-kserve-859d6c4b4d-k8x5n"] Oct 30 17:25:24.925835 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:24.925808 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-57f1de/simulator-single-main-57f1de-kserve-859d6c4b4d-k8x5n"] Oct 30 17:25:26.597128 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:26.597092 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b27fdf08-0633-4e1b-8e8b-102e27c7cb20" path="/var/lib/kubelet/pods/b27fdf08-0633-4e1b-8e8b-102e27c7cb20/volumes" Oct 30 17:25:27.487561 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:27.487515 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-57f1de-dc7474p26p" podUID="f9770be6-158c-4dc1-ab63-d06993212c6c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:25:27.487793 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:27.487537 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-57f1de-66c5bbc45fd7cb" podUID="57cbccbe-ea6d-4699-a560-5d44ee5a9915" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:25:34.989001 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:34.988958 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-57f1de-openshift-default-77c5d6878fqjd27"] Oct 30 17:25:34.989395 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:34.989224 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-57f1de-openshift-default-77c5d6878fqjd27" podUID="1ed76266-22d9-4ec7-bc9e-8f84355b3485" containerName="istio-proxy" containerID="cri-o://1b0a95df9a6f1e0aba896205e8a077f6642dae03348f0c0024265b98badf96ef" gracePeriod=30 Oct 30 17:25:35.462200 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:35.462157 5052 patch_prober.go:29] interesting pod/e2e-worker-single-main-57f1de-openshift-default-77c5d6878fqjd27 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 17:25:35.462374 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:35.462227 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-57f1de-openshift-default-77c5d6878fqjd27" podUID="1ed76266-22d9-4ec7-bc9e-8f84355b3485" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 17:25:37.487977 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:37.487936 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-57f1de-dc7474p26p" podUID="f9770be6-158c-4dc1-ab63-d06993212c6c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:25:37.487977 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:37.487964 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-57f1de-66c5bbc45fd7cb" podUID="57cbccbe-ea6d-4699-a560-5d44ee5a9915" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:25:39.868431 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:39.868406 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-57f1de-66c5bbc45fd7cb" Oct 30 17:25:39.889644 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:39.889618 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57cbccbe-ea6d-4699-a560-5d44ee5a9915-kube-api-access-7pws5\" (UniqueName: \"kubernetes.io/projected/57cbccbe-ea6d-4699-a560-5d44ee5a9915-kube-api-access-7pws5\") pod \"57cbccbe-ea6d-4699-a560-5d44ee5a9915\" (UID: \"57cbccbe-ea6d-4699-a560-5d44ee5a9915\") " Oct 30 17:25:39.889748 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:39.889663 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57cbccbe-ea6d-4699-a560-5d44ee5a9915-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57cbccbe-ea6d-4699-a560-5d44ee5a9915-plugins-config-volume\") pod \"57cbccbe-ea6d-4699-a560-5d44ee5a9915\" (UID: \"57cbccbe-ea6d-4699-a560-5d44ee5a9915\") " Oct 30 17:25:39.890103 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:39.890082 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57cbccbe-ea6d-4699-a560-5d44ee5a9915-plugins-config-volume" pod "57cbccbe-ea6d-4699-a560-5d44ee5a9915" (UID: "57cbccbe-ea6d-4699-a560-5d44ee5a9915"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:25:39.891749 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:39.891726 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57cbccbe-ea6d-4699-a560-5d44ee5a9915-kube-api-access-7pws5" pod "57cbccbe-ea6d-4699-a560-5d44ee5a9915" (UID: "57cbccbe-ea6d-4699-a560-5d44ee5a9915"). InnerVolumeSpecName "kube-api-access-7pws5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:25:39.895787 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:39.895769 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-57f1de-dc7474p26p" Oct 30 17:25:39.953120 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:39.953088 5052 generic.go:356] "Generic (PLEG): container finished" podID="57cbccbe-ea6d-4699-a560-5d44ee5a9915" containerID="a6bbfbfe1f2704f466a530199ae96cf5761943e2384e59d01830a72dc9f5f6f2" exitCode=0 Oct 30 17:25:39.953299 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:39.953151 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-57f1de-66c5bbc45fd7cb" Oct 30 17:25:39.953299 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:39.953172 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-57f1de-66c5bbc45fd7cb" event={"ID":"57cbccbe-ea6d-4699-a560-5d44ee5a9915","Type":"ContainerDied","Data":"a6bbfbfe1f2704f466a530199ae96cf5761943e2384e59d01830a72dc9f5f6f2"} Oct 30 17:25:39.953299 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:39.953234 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-57f1de-66c5bbc45fd7cb" event={"ID":"57cbccbe-ea6d-4699-a560-5d44ee5a9915","Type":"ContainerDied","Data":"a04be8785196013052a878932aed721fb0d6ecbeb5a3c33b853608dbc9802452"} Oct 30 17:25:39.953299 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:39.953265 5052 scope.go:122] "RemoveContainer" containerID="a6bbfbfe1f2704f466a530199ae96cf5761943e2384e59d01830a72dc9f5f6f2" Oct 30 17:25:39.955164 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:39.955110 5052 generic.go:356] "Generic (PLEG): container finished" podID="f9770be6-158c-4dc1-ab63-d06993212c6c" containerID="9d2df79ae5eab609a3241ce0ae7c24ba5d4676e4a6f6ec8a9aae989d882b9282" exitCode=0 Oct 30 17:25:39.955164 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:39.955156 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-57f1de-dc7474p26p" event={"ID":"f9770be6-158c-4dc1-ab63-d06993212c6c","Type":"ContainerDied","Data":"9d2df79ae5eab609a3241ce0ae7c24ba5d4676e4a6f6ec8a9aae989d882b9282"} Oct 30 17:25:39.955285 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:39.955197 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-57f1de-dc7474p26p" event={"ID":"f9770be6-158c-4dc1-ab63-d06993212c6c","Type":"ContainerDied","Data":"797a19d145723ac4f3bd167c600025a6e0c1d99b5f57f23d3b3080e690114608"} Oct 30 17:25:39.955285 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:39.955263 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-57f1de-dc7474p26p" Oct 30 17:25:39.966060 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:39.966041 5052 scope.go:122] "RemoveContainer" containerID="a6bbfbfe1f2704f466a530199ae96cf5761943e2384e59d01830a72dc9f5f6f2" Oct 30 17:25:39.966346 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:25:39.966329 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6bbfbfe1f2704f466a530199ae96cf5761943e2384e59d01830a72dc9f5f6f2\": container with ID starting with a6bbfbfe1f2704f466a530199ae96cf5761943e2384e59d01830a72dc9f5f6f2 not found: ID does not exist" containerID="a6bbfbfe1f2704f466a530199ae96cf5761943e2384e59d01830a72dc9f5f6f2" Oct 30 17:25:39.966418 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:39.966354 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6bbfbfe1f2704f466a530199ae96cf5761943e2384e59d01830a72dc9f5f6f2"} err="failed to get container status \"a6bbfbfe1f2704f466a530199ae96cf5761943e2384e59d01830a72dc9f5f6f2\": rpc error: code = NotFound desc = could not find container \"a6bbfbfe1f2704f466a530199ae96cf5761943e2384e59d01830a72dc9f5f6f2\": container with ID starting with a6bbfbfe1f2704f466a530199ae96cf5761943e2384e59d01830a72dc9f5f6f2 not found: ID does not exist" Oct 30 17:25:39.966418 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:39.966370 5052 scope.go:122] "RemoveContainer" containerID="9d2df79ae5eab609a3241ce0ae7c24ba5d4676e4a6f6ec8a9aae989d882b9282" Oct 30 17:25:39.976004 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:39.975975 5052 scope.go:122] "RemoveContainer" containerID="9d2df79ae5eab609a3241ce0ae7c24ba5d4676e4a6f6ec8a9aae989d882b9282" Oct 30 17:25:39.976283 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:25:39.976261 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d2df79ae5eab609a3241ce0ae7c24ba5d4676e4a6f6ec8a9aae989d882b9282\": container with ID starting with 9d2df79ae5eab609a3241ce0ae7c24ba5d4676e4a6f6ec8a9aae989d882b9282 not found: ID does not exist" containerID="9d2df79ae5eab609a3241ce0ae7c24ba5d4676e4a6f6ec8a9aae989d882b9282" Oct 30 17:25:39.976371 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:39.976290 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d2df79ae5eab609a3241ce0ae7c24ba5d4676e4a6f6ec8a9aae989d882b9282"} err="failed to get container status \"9d2df79ae5eab609a3241ce0ae7c24ba5d4676e4a6f6ec8a9aae989d882b9282\": rpc error: code = NotFound desc = could not find container \"9d2df79ae5eab609a3241ce0ae7c24ba5d4676e4a6f6ec8a9aae989d882b9282\": container with ID starting with 9d2df79ae5eab609a3241ce0ae7c24ba5d4676e4a6f6ec8a9aae989d882b9282 not found: ID does not exist" Oct 30 17:25:39.990219 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:39.990200 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9770be6-158c-4dc1-ab63-d06993212c6c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9770be6-158c-4dc1-ab63-d06993212c6c-plugins-config-volume\") pod \"f9770be6-158c-4dc1-ab63-d06993212c6c\" (UID: \"f9770be6-158c-4dc1-ab63-d06993212c6c\") " Oct 30 17:25:39.990313 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:39.990253 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9770be6-158c-4dc1-ab63-d06993212c6c-kube-api-access-6k594\" (UniqueName: \"kubernetes.io/projected/f9770be6-158c-4dc1-ab63-d06993212c6c-kube-api-access-6k594\") pod \"f9770be6-158c-4dc1-ab63-d06993212c6c\" (UID: \"f9770be6-158c-4dc1-ab63-d06993212c6c\") " Oct 30 17:25:39.990526 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:39.990502 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pws5\" (UniqueName: \"kubernetes.io/projected/57cbccbe-ea6d-4699-a560-5d44ee5a9915-kube-api-access-7pws5\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:25:39.990526 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:39.990520 5052 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57cbccbe-ea6d-4699-a560-5d44ee5a9915-plugins-config-volume\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:25:39.990720 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:39.990536 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9770be6-158c-4dc1-ab63-d06993212c6c-plugins-config-volume" pod "f9770be6-158c-4dc1-ab63-d06993212c6c" (UID: "f9770be6-158c-4dc1-ab63-d06993212c6c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:25:39.992061 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:39.992040 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9770be6-158c-4dc1-ab63-d06993212c6c-kube-api-access-6k594" pod "f9770be6-158c-4dc1-ab63-d06993212c6c" (UID: "f9770be6-158c-4dc1-ab63-d06993212c6c"). InnerVolumeSpecName "kube-api-access-6k594". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:25:40.003789 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:40.003760 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-57f1de-66c5bbc45fd7cb"] Oct 30 17:25:40.009017 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:40.008996 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-57f1de-66c5bbc45fd7cb"] Oct 30 17:25:40.092060 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:40.092033 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6k594\" (UniqueName: \"kubernetes.io/projected/f9770be6-158c-4dc1-ab63-d06993212c6c-kube-api-access-6k594\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:25:40.092060 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:40.092058 5052 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9770be6-158c-4dc1-ab63-d06993212c6c-plugins-config-volume\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:25:40.225131 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:40.225107 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-57f1de-openshift-default-77c5d6878fqjd27" Oct 30 17:25:40.294169 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:40.294137 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ed76266-22d9-4ec7-bc9e-8f84355b3485-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ed76266-22d9-4ec7-bc9e-8f84355b3485-istio-data\") pod \"1ed76266-22d9-4ec7-bc9e-8f84355b3485\" (UID: \"1ed76266-22d9-4ec7-bc9e-8f84355b3485\") " Oct 30 17:25:40.294333 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:40.294193 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ed76266-22d9-4ec7-bc9e-8f84355b3485-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ed76266-22d9-4ec7-bc9e-8f84355b3485-istio-envoy\") pod \"1ed76266-22d9-4ec7-bc9e-8f84355b3485\" (UID: \"1ed76266-22d9-4ec7-bc9e-8f84355b3485\") " Oct 30 17:25:40.294333 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:40.294233 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ed76266-22d9-4ec7-bc9e-8f84355b3485-kube-api-access-q9mp6\" (UniqueName: \"kubernetes.io/projected/1ed76266-22d9-4ec7-bc9e-8f84355b3485-kube-api-access-q9mp6\") pod \"1ed76266-22d9-4ec7-bc9e-8f84355b3485\" (UID: \"1ed76266-22d9-4ec7-bc9e-8f84355b3485\") " Oct 30 17:25:40.294333 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:40.294300 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1ed76266-22d9-4ec7-bc9e-8f84355b3485-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ed76266-22d9-4ec7-bc9e-8f84355b3485-istio-podinfo\") pod \"1ed76266-22d9-4ec7-bc9e-8f84355b3485\" (UID: \"1ed76266-22d9-4ec7-bc9e-8f84355b3485\") " Oct 30 17:25:40.294507 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:40.294333 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ed76266-22d9-4ec7-bc9e-8f84355b3485-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ed76266-22d9-4ec7-bc9e-8f84355b3485-istiod-ca-cert\") pod \"1ed76266-22d9-4ec7-bc9e-8f84355b3485\" (UID: \"1ed76266-22d9-4ec7-bc9e-8f84355b3485\") " Oct 30 17:25:40.294507 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:40.294362 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ed76266-22d9-4ec7-bc9e-8f84355b3485-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ed76266-22d9-4ec7-bc9e-8f84355b3485-credential-socket\") pod \"1ed76266-22d9-4ec7-bc9e-8f84355b3485\" (UID: \"1ed76266-22d9-4ec7-bc9e-8f84355b3485\") " Oct 30 17:25:40.294507 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:40.294396 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ed76266-22d9-4ec7-bc9e-8f84355b3485-istio-data" pod "1ed76266-22d9-4ec7-bc9e-8f84355b3485" (UID: "1ed76266-22d9-4ec7-bc9e-8f84355b3485"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:25:40.294507 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:40.294414 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ed76266-22d9-4ec7-bc9e-8f84355b3485-istio-token\" (UniqueName: \"kubernetes.io/projected/1ed76266-22d9-4ec7-bc9e-8f84355b3485-istio-token\") pod \"1ed76266-22d9-4ec7-bc9e-8f84355b3485\" (UID: \"1ed76266-22d9-4ec7-bc9e-8f84355b3485\") " Oct 30 17:25:40.294507 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:40.294497 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ed76266-22d9-4ec7-bc9e-8f84355b3485-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ed76266-22d9-4ec7-bc9e-8f84355b3485-workload-socket\") pod \"1ed76266-22d9-4ec7-bc9e-8f84355b3485\" (UID: \"1ed76266-22d9-4ec7-bc9e-8f84355b3485\") " Oct 30 17:25:40.294797 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:40.294569 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ed76266-22d9-4ec7-bc9e-8f84355b3485-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ed76266-22d9-4ec7-bc9e-8f84355b3485-workload-certs\") pod \"1ed76266-22d9-4ec7-bc9e-8f84355b3485\" (UID: \"1ed76266-22d9-4ec7-bc9e-8f84355b3485\") " Oct 30 17:25:40.294885 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:40.294861 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ed76266-22d9-4ec7-bc9e-8f84355b3485-credential-socket" pod "1ed76266-22d9-4ec7-bc9e-8f84355b3485" (UID: "1ed76266-22d9-4ec7-bc9e-8f84355b3485"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:25:40.294948 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:40.294908 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ed76266-22d9-4ec7-bc9e-8f84355b3485-istiod-ca-cert" pod "1ed76266-22d9-4ec7-bc9e-8f84355b3485" (UID: "1ed76266-22d9-4ec7-bc9e-8f84355b3485"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:25:40.295009 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:40.294953 5052 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ed76266-22d9-4ec7-bc9e-8f84355b3485-istio-data\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:25:40.295009 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:40.294976 5052 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ed76266-22d9-4ec7-bc9e-8f84355b3485-credential-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:25:40.295096 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:40.295063 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ed76266-22d9-4ec7-bc9e-8f84355b3485-workload-socket" pod "1ed76266-22d9-4ec7-bc9e-8f84355b3485" (UID: "1ed76266-22d9-4ec7-bc9e-8f84355b3485"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:25:40.295096 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:40.295078 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ed76266-22d9-4ec7-bc9e-8f84355b3485-workload-certs" pod "1ed76266-22d9-4ec7-bc9e-8f84355b3485" (UID: "1ed76266-22d9-4ec7-bc9e-8f84355b3485"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:25:40.296896 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:40.296872 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ed76266-22d9-4ec7-bc9e-8f84355b3485-istio-token" pod "1ed76266-22d9-4ec7-bc9e-8f84355b3485" (UID: "1ed76266-22d9-4ec7-bc9e-8f84355b3485"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:25:40.297052 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:40.297020 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ed76266-22d9-4ec7-bc9e-8f84355b3485-istio-envoy" pod "1ed76266-22d9-4ec7-bc9e-8f84355b3485" (UID: "1ed76266-22d9-4ec7-bc9e-8f84355b3485"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:25:40.297052 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:40.297048 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ed76266-22d9-4ec7-bc9e-8f84355b3485-kube-api-access-q9mp6" pod "1ed76266-22d9-4ec7-bc9e-8f84355b3485" (UID: "1ed76266-22d9-4ec7-bc9e-8f84355b3485"). InnerVolumeSpecName "kube-api-access-q9mp6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:25:40.297387 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:40.297356 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1ed76266-22d9-4ec7-bc9e-8f84355b3485-istio-podinfo" pod "1ed76266-22d9-4ec7-bc9e-8f84355b3485" (UID: "1ed76266-22d9-4ec7-bc9e-8f84355b3485"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:25:40.304652 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:40.304631 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-57f1de-dc7474p26p"] Oct 30 17:25:40.308777 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:40.308756 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-57f1de-dc7474p26p"] Oct 30 17:25:40.395481 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:40.395424 5052 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ed76266-22d9-4ec7-bc9e-8f84355b3485-istio-podinfo\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:25:40.395481 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:40.395446 5052 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ed76266-22d9-4ec7-bc9e-8f84355b3485-istiod-ca-cert\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:25:40.395481 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:40.395457 5052 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ed76266-22d9-4ec7-bc9e-8f84355b3485-istio-token\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:25:40.395481 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:40.395466 5052 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ed76266-22d9-4ec7-bc9e-8f84355b3485-workload-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:25:40.395481 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:40.395474 5052 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ed76266-22d9-4ec7-bc9e-8f84355b3485-workload-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:25:40.395481 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:40.395482 5052 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ed76266-22d9-4ec7-bc9e-8f84355b3485-istio-envoy\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:25:40.395776 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:40.395490 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9mp6\" (UniqueName: \"kubernetes.io/projected/1ed76266-22d9-4ec7-bc9e-8f84355b3485-kube-api-access-q9mp6\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:25:40.598968 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:40.598936 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57cbccbe-ea6d-4699-a560-5d44ee5a9915" path="/var/lib/kubelet/pods/57cbccbe-ea6d-4699-a560-5d44ee5a9915/volumes" Oct 30 17:25:40.599552 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:40.599312 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9770be6-158c-4dc1-ab63-d06993212c6c" path="/var/lib/kubelet/pods/f9770be6-158c-4dc1-ab63-d06993212c6c/volumes" Oct 30 17:25:40.962958 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:40.962920 5052 generic.go:356] "Generic (PLEG): container finished" podID="1ed76266-22d9-4ec7-bc9e-8f84355b3485" containerID="1b0a95df9a6f1e0aba896205e8a077f6642dae03348f0c0024265b98badf96ef" exitCode=0 Oct 30 17:25:40.963404 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:40.962974 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-57f1de-openshift-default-77c5d6878fqjd27" Oct 30 17:25:40.963404 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:40.962998 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-57f1de-openshift-default-77c5d6878fqjd27" event={"ID":"1ed76266-22d9-4ec7-bc9e-8f84355b3485","Type":"ContainerDied","Data":"1b0a95df9a6f1e0aba896205e8a077f6642dae03348f0c0024265b98badf96ef"} Oct 30 17:25:40.963404 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:40.963080 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-57f1de-openshift-default-77c5d6878fqjd27" event={"ID":"1ed76266-22d9-4ec7-bc9e-8f84355b3485","Type":"ContainerDied","Data":"4879337a7b024adc9c45466210b2425ce5175d1c56caaa16308ecea094d5bf83"} Oct 30 17:25:40.963404 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:40.963110 5052 scope.go:122] "RemoveContainer" containerID="1b0a95df9a6f1e0aba896205e8a077f6642dae03348f0c0024265b98badf96ef" Oct 30 17:25:40.971872 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:40.971858 5052 scope.go:122] "RemoveContainer" containerID="813286bc891d79dfba4eace2f817569d076169866ab98b4543bfc0f57e938d75" Oct 30 17:25:40.979694 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:40.979677 5052 scope.go:122] "RemoveContainer" containerID="1b0a95df9a6f1e0aba896205e8a077f6642dae03348f0c0024265b98badf96ef" Oct 30 17:25:40.979965 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:25:40.979936 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b0a95df9a6f1e0aba896205e8a077f6642dae03348f0c0024265b98badf96ef\": container with ID starting with 1b0a95df9a6f1e0aba896205e8a077f6642dae03348f0c0024265b98badf96ef not found: ID does not exist" containerID="1b0a95df9a6f1e0aba896205e8a077f6642dae03348f0c0024265b98badf96ef" Oct 30 17:25:40.980063 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:40.979976 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b0a95df9a6f1e0aba896205e8a077f6642dae03348f0c0024265b98badf96ef"} err="failed to get container status \"1b0a95df9a6f1e0aba896205e8a077f6642dae03348f0c0024265b98badf96ef\": rpc error: code = NotFound desc = could not find container \"1b0a95df9a6f1e0aba896205e8a077f6642dae03348f0c0024265b98badf96ef\": container with ID starting with 1b0a95df9a6f1e0aba896205e8a077f6642dae03348f0c0024265b98badf96ef not found: ID does not exist" Oct 30 17:25:40.980063 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:40.979998 5052 scope.go:122] "RemoveContainer" containerID="813286bc891d79dfba4eace2f817569d076169866ab98b4543bfc0f57e938d75" Oct 30 17:25:40.980285 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:25:40.980268 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"813286bc891d79dfba4eace2f817569d076169866ab98b4543bfc0f57e938d75\": container with ID starting with 813286bc891d79dfba4eace2f817569d076169866ab98b4543bfc0f57e938d75 not found: ID does not exist" containerID="813286bc891d79dfba4eace2f817569d076169866ab98b4543bfc0f57e938d75" Oct 30 17:25:40.980343 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:40.980302 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"813286bc891d79dfba4eace2f817569d076169866ab98b4543bfc0f57e938d75"} err="failed to get container status \"813286bc891d79dfba4eace2f817569d076169866ab98b4543bfc0f57e938d75\": rpc error: code = NotFound desc = could not find container \"813286bc891d79dfba4eace2f817569d076169866ab98b4543bfc0f57e938d75\": container with ID starting with 813286bc891d79dfba4eace2f817569d076169866ab98b4543bfc0f57e938d75 not found: ID does not exist" Oct 30 17:25:41.025100 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:41.025080 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-57f1de-openshift-default-77c5d6878fqjd27"] Oct 30 17:25:41.028613 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:41.028596 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-57f1de-openshift-default-77c5d6878fqjd27"] Oct 30 17:25:42.596505 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:42.596431 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ed76266-22d9-4ec7-bc9e-8f84355b3485" path="/var/lib/kubelet/pods/1ed76266-22d9-4ec7-bc9e-8f84355b3485/volumes" Oct 30 17:25:45.968309 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:45.968277 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4wrf8_4f0f46f4-5cda-4d67-b6f3-47df06eb844c/console-operator/2.log" Oct 30 17:25:45.975548 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:45.975525 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4wrf8_4f0f46f4-5cda-4d67-b6f3-47df06eb844c/console-operator/2.log" Oct 30 17:25:48.840204 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:25:48.840170 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 17:25:49.702943 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:49.702907 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cdwcj"] Oct 30 17:25:54.875603 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:25:54.875555 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cdwcj"] Oct 30 17:26:18.839568 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:26:18.839531 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 17:26:48.839708 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:26:48.839670 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 17:27:15.012114 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:27:15.012030 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-ct9g7"] Oct 30 17:27:15.012630 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:27:15.012470 5052 cpu_manager.go:414] "containerMap: removing container" podUID="57cbccbe-ea6d-4699-a560-5d44ee5a9915" containerName="payload-processing" Oct 30 17:27:15.012630 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:27:15.012481 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57cbccbe-ea6d-4699-a560-5d44ee5a9915" containerName="payload-processing" Oct 30 17:27:15.012630 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:27:15.012493 5052 cpu_manager.go:414] "containerMap: removing container" podUID="1ed76266-22d9-4ec7-bc9e-8f84355b3485" containerName="istio-proxy" Oct 30 17:27:15.012630 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:27:15.012498 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ed76266-22d9-4ec7-bc9e-8f84355b3485" containerName="istio-proxy" Oct 30 17:27:15.012630 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:27:15.012505 5052 cpu_manager.go:414] "containerMap: removing container" podUID="b27fdf08-0633-4e1b-8e8b-102e27c7cb20" containerName="main" Oct 30 17:27:15.012630 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:27:15.012510 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b27fdf08-0633-4e1b-8e8b-102e27c7cb20" containerName="main" Oct 30 17:27:15.012630 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:27:15.012520 5052 cpu_manager.go:414] "containerMap: removing container" podUID="1ed76266-22d9-4ec7-bc9e-8f84355b3485" containerName="istio-proxy" Oct 30 17:27:15.012630 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:27:15.012525 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ed76266-22d9-4ec7-bc9e-8f84355b3485" containerName="istio-proxy" Oct 30 17:27:15.012630 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:27:15.012544 5052 cpu_manager.go:414] "containerMap: removing container" podUID="d81b687a-7327-4433-975a-a28e5f9c2780" containerName="maas-api" Oct 30 17:27:15.012630 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:27:15.012548 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d81b687a-7327-4433-975a-a28e5f9c2780" containerName="maas-api" Oct 30 17:27:15.012630 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:27:15.012558 5052 cpu_manager.go:414] "containerMap: removing container" podUID="f9770be6-158c-4dc1-ab63-d06993212c6c" containerName="payload-pre-processing" Oct 30 17:27:15.012630 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:27:15.012562 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9770be6-158c-4dc1-ab63-d06993212c6c" containerName="payload-pre-processing" Oct 30 17:27:15.013083 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:27:15.012646 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="d81b687a-7327-4433-975a-a28e5f9c2780" containerName="maas-api" Oct 30 17:27:15.013083 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:27:15.012654 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9770be6-158c-4dc1-ab63-d06993212c6c" containerName="payload-pre-processing" Oct 30 17:27:15.013083 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:27:15.012664 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="57cbccbe-ea6d-4699-a560-5d44ee5a9915" containerName="payload-processing" Oct 30 17:27:15.013083 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:27:15.012673 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ed76266-22d9-4ec7-bc9e-8f84355b3485" containerName="istio-proxy" Oct 30 17:27:15.013083 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:27:15.012678 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="b27fdf08-0633-4e1b-8e8b-102e27c7cb20" containerName="main" Oct 30 17:27:15.013083 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:27:15.012684 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ed76266-22d9-4ec7-bc9e-8f84355b3485" containerName="istio-proxy" Oct 30 17:27:15.015864 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:27:15.015846 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-ct9g7" Oct 30 17:27:15.030837 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:27:15.030812 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-ct9g7"] Oct 30 17:27:15.041798 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:27:15.041773 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-gwddn\"" Oct 30 17:27:15.134739 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:27:15.134707 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3c887582-ac80-4a00-9e7d-87c569e4e425-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-ct9g7\" (UID: \"3c887582-ac80-4a00-9e7d-87c569e4e425\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-ct9g7" Oct 30 17:27:15.134871 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:27:15.134768 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgjjq\" (UniqueName: \"kubernetes.io/projected/3c887582-ac80-4a00-9e7d-87c569e4e425-kube-api-access-mgjjq\") pod \"kuadrant-operator-controller-manager-57bb9bfff-ct9g7\" (UID: \"3c887582-ac80-4a00-9e7d-87c569e4e425\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-ct9g7" Oct 30 17:27:15.235259 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:27:15.235231 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3c887582-ac80-4a00-9e7d-87c569e4e425-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-ct9g7\" (UID: \"3c887582-ac80-4a00-9e7d-87c569e4e425\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-ct9g7" Oct 30 17:27:15.235382 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:27:15.235318 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgjjq\" (UniqueName: \"kubernetes.io/projected/3c887582-ac80-4a00-9e7d-87c569e4e425-kube-api-access-mgjjq\") pod \"kuadrant-operator-controller-manager-57bb9bfff-ct9g7\" (UID: \"3c887582-ac80-4a00-9e7d-87c569e4e425\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-ct9g7" Oct 30 17:27:15.235637 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:27:15.235620 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3c887582-ac80-4a00-9e7d-87c569e4e425-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-ct9g7\" (UID: \"3c887582-ac80-4a00-9e7d-87c569e4e425\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-ct9g7" Oct 30 17:27:15.261312 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:27:15.261288 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgjjq\" (UniqueName: \"kubernetes.io/projected/3c887582-ac80-4a00-9e7d-87c569e4e425-kube-api-access-mgjjq\") pod \"kuadrant-operator-controller-manager-57bb9bfff-ct9g7\" (UID: \"3c887582-ac80-4a00-9e7d-87c569e4e425\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-ct9g7" Oct 30 17:27:15.329377 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:27:15.329297 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-ct9g7" Oct 30 17:27:15.664723 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:27:15.664697 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-ct9g7"] Oct 30 17:27:15.667755 ip-10-0-131-136 kubenswrapper[5052]: W1030 17:27:15.667728 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c887582_ac80_4a00_9e7d_87c569e4e425.slice/crio-6717892e92a6f76799abe0fd76c0584fdc064c57ed9f58d349df66162dce78ef WatchSource:0}: Error finding container 6717892e92a6f76799abe0fd76c0584fdc064c57ed9f58d349df66162dce78ef: Status 404 returned error can't find the container with id 6717892e92a6f76799abe0fd76c0584fdc064c57ed9f58d349df66162dce78ef Oct 30 17:27:16.463205 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:27:16.463169 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-ct9g7" event={"ID":"3c887582-ac80-4a00-9e7d-87c569e4e425","Type":"ContainerStarted","Data":"9e3fa7211bf1edd1e89a3d6d74ffaf2fee9f180a1e4f2e46d28feecf47caafc7"} Oct 30 17:27:16.463636 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:27:16.463219 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-ct9g7" event={"ID":"3c887582-ac80-4a00-9e7d-87c569e4e425","Type":"ContainerStarted","Data":"6717892e92a6f76799abe0fd76c0584fdc064c57ed9f58d349df66162dce78ef"} Oct 30 17:27:16.463636 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:27:16.463266 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-ct9g7" Oct 30 17:27:16.540644 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:27:16.540568 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-ct9g7" podStartSLOduration=2.5405530560000003 podStartE2EDuration="2.540553056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:27:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:27:16.525681672 +0000 UTC m=+3992.053016893" watchObservedRunningTime="2026-10-30 17:27:16.540553056 +0000 UTC m=+3992.067888274" Oct 30 17:27:18.839273 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:27:18.839240 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 17:27:27.470456 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:27:27.470426 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-ct9g7" Oct 30 17:27:33.787698 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:27:33.787660 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cdwcj"] Oct 30 17:27:48.664494 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:27:48.664446 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cdwcj"] Oct 30 17:27:48.844153 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:27:48.844117 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 17:28:18.840560 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:28:18.840511 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 17:28:26.663077 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:26.663031 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-e2e8d2-openshift-default-9c5dbcc7c-q4cft"] Oct 30 17:28:26.668017 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:26.667988 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-e2e8d2-openshift-default-9c5dbcc7c-q4cft" Oct 30 17:28:26.683025 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:26.683004 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-e2e8d2-openshift-default-dockercfg-2qt92\"" Oct 30 17:28:26.696946 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:26.696919 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-e2e8d2-openshift-default-9c5dbcc7c-q4cft"] Oct 30 17:28:26.753113 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:26.753070 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f07afe6-8fe0-4bc1-a6c5-02d62ae41017-istio-envoy\") pod \"e2e-worker-single-main-e2e8d2-openshift-default-9c5dbcc7c-q4cft\" (UID: \"2f07afe6-8fe0-4bc1-a6c5-02d62ae41017\") " pod="openshift-ingress/e2e-worker-single-main-e2e8d2-openshift-default-9c5dbcc7c-q4cft" Oct 30 17:28:26.753317 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:26.753117 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f07afe6-8fe0-4bc1-a6c5-02d62ae41017-istio-token\") pod \"e2e-worker-single-main-e2e8d2-openshift-default-9c5dbcc7c-q4cft\" (UID: \"2f07afe6-8fe0-4bc1-a6c5-02d62ae41017\") " pod="openshift-ingress/e2e-worker-single-main-e2e8d2-openshift-default-9c5dbcc7c-q4cft" Oct 30 17:28:26.753317 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:26.753214 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f07afe6-8fe0-4bc1-a6c5-02d62ae41017-istio-data\") pod \"e2e-worker-single-main-e2e8d2-openshift-default-9c5dbcc7c-q4cft\" (UID: \"2f07afe6-8fe0-4bc1-a6c5-02d62ae41017\") " pod="openshift-ingress/e2e-worker-single-main-e2e8d2-openshift-default-9c5dbcc7c-q4cft" Oct 30 17:28:26.753449 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:26.753344 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f07afe6-8fe0-4bc1-a6c5-02d62ae41017-credential-socket\") pod \"e2e-worker-single-main-e2e8d2-openshift-default-9c5dbcc7c-q4cft\" (UID: \"2f07afe6-8fe0-4bc1-a6c5-02d62ae41017\") " pod="openshift-ingress/e2e-worker-single-main-e2e8d2-openshift-default-9c5dbcc7c-q4cft" Oct 30 17:28:26.753449 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:26.753382 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f07afe6-8fe0-4bc1-a6c5-02d62ae41017-workload-certs\") pod \"e2e-worker-single-main-e2e8d2-openshift-default-9c5dbcc7c-q4cft\" (UID: \"2f07afe6-8fe0-4bc1-a6c5-02d62ae41017\") " pod="openshift-ingress/e2e-worker-single-main-e2e8d2-openshift-default-9c5dbcc7c-q4cft" Oct 30 17:28:26.753560 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:26.753459 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7pw9\" (UniqueName: \"kubernetes.io/projected/2f07afe6-8fe0-4bc1-a6c5-02d62ae41017-kube-api-access-q7pw9\") pod \"e2e-worker-single-main-e2e8d2-openshift-default-9c5dbcc7c-q4cft\" (UID: \"2f07afe6-8fe0-4bc1-a6c5-02d62ae41017\") " pod="openshift-ingress/e2e-worker-single-main-e2e8d2-openshift-default-9c5dbcc7c-q4cft" Oct 30 17:28:26.753560 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:26.753524 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f07afe6-8fe0-4bc1-a6c5-02d62ae41017-istiod-ca-cert\") pod \"e2e-worker-single-main-e2e8d2-openshift-default-9c5dbcc7c-q4cft\" (UID: \"2f07afe6-8fe0-4bc1-a6c5-02d62ae41017\") " pod="openshift-ingress/e2e-worker-single-main-e2e8d2-openshift-default-9c5dbcc7c-q4cft" Oct 30 17:28:26.753697 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:26.753621 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f07afe6-8fe0-4bc1-a6c5-02d62ae41017-istio-podinfo\") pod \"e2e-worker-single-main-e2e8d2-openshift-default-9c5dbcc7c-q4cft\" (UID: \"2f07afe6-8fe0-4bc1-a6c5-02d62ae41017\") " pod="openshift-ingress/e2e-worker-single-main-e2e8d2-openshift-default-9c5dbcc7c-q4cft" Oct 30 17:28:26.753762 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:26.753700 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f07afe6-8fe0-4bc1-a6c5-02d62ae41017-workload-socket\") pod \"e2e-worker-single-main-e2e8d2-openshift-default-9c5dbcc7c-q4cft\" (UID: \"2f07afe6-8fe0-4bc1-a6c5-02d62ae41017\") " pod="openshift-ingress/e2e-worker-single-main-e2e8d2-openshift-default-9c5dbcc7c-q4cft" Oct 30 17:28:26.854672 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:26.854639 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f07afe6-8fe0-4bc1-a6c5-02d62ae41017-credential-socket\") pod \"e2e-worker-single-main-e2e8d2-openshift-default-9c5dbcc7c-q4cft\" (UID: \"2f07afe6-8fe0-4bc1-a6c5-02d62ae41017\") " pod="openshift-ingress/e2e-worker-single-main-e2e8d2-openshift-default-9c5dbcc7c-q4cft" Oct 30 17:28:26.854853 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:26.854682 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f07afe6-8fe0-4bc1-a6c5-02d62ae41017-workload-certs\") pod \"e2e-worker-single-main-e2e8d2-openshift-default-9c5dbcc7c-q4cft\" (UID: \"2f07afe6-8fe0-4bc1-a6c5-02d62ae41017\") " pod="openshift-ingress/e2e-worker-single-main-e2e8d2-openshift-default-9c5dbcc7c-q4cft" Oct 30 17:28:26.854853 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:26.854725 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7pw9\" (UniqueName: \"kubernetes.io/projected/2f07afe6-8fe0-4bc1-a6c5-02d62ae41017-kube-api-access-q7pw9\") pod \"e2e-worker-single-main-e2e8d2-openshift-default-9c5dbcc7c-q4cft\" (UID: \"2f07afe6-8fe0-4bc1-a6c5-02d62ae41017\") " pod="openshift-ingress/e2e-worker-single-main-e2e8d2-openshift-default-9c5dbcc7c-q4cft" Oct 30 17:28:26.854853 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:26.854773 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f07afe6-8fe0-4bc1-a6c5-02d62ae41017-istiod-ca-cert\") pod \"e2e-worker-single-main-e2e8d2-openshift-default-9c5dbcc7c-q4cft\" (UID: \"2f07afe6-8fe0-4bc1-a6c5-02d62ae41017\") " pod="openshift-ingress/e2e-worker-single-main-e2e8d2-openshift-default-9c5dbcc7c-q4cft" Oct 30 17:28:26.854853 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:26.854838 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f07afe6-8fe0-4bc1-a6c5-02d62ae41017-istio-podinfo\") pod \"e2e-worker-single-main-e2e8d2-openshift-default-9c5dbcc7c-q4cft\" (UID: \"2f07afe6-8fe0-4bc1-a6c5-02d62ae41017\") " pod="openshift-ingress/e2e-worker-single-main-e2e8d2-openshift-default-9c5dbcc7c-q4cft" Oct 30 17:28:26.855051 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:26.854887 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f07afe6-8fe0-4bc1-a6c5-02d62ae41017-workload-socket\") pod \"e2e-worker-single-main-e2e8d2-openshift-default-9c5dbcc7c-q4cft\" (UID: \"2f07afe6-8fe0-4bc1-a6c5-02d62ae41017\") " pod="openshift-ingress/e2e-worker-single-main-e2e8d2-openshift-default-9c5dbcc7c-q4cft" Oct 30 17:28:26.855051 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:26.854918 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f07afe6-8fe0-4bc1-a6c5-02d62ae41017-istio-envoy\") pod \"e2e-worker-single-main-e2e8d2-openshift-default-9c5dbcc7c-q4cft\" (UID: \"2f07afe6-8fe0-4bc1-a6c5-02d62ae41017\") " pod="openshift-ingress/e2e-worker-single-main-e2e8d2-openshift-default-9c5dbcc7c-q4cft" Oct 30 17:28:26.855051 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:26.854940 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f07afe6-8fe0-4bc1-a6c5-02d62ae41017-istio-token\") pod \"e2e-worker-single-main-e2e8d2-openshift-default-9c5dbcc7c-q4cft\" (UID: \"2f07afe6-8fe0-4bc1-a6c5-02d62ae41017\") " pod="openshift-ingress/e2e-worker-single-main-e2e8d2-openshift-default-9c5dbcc7c-q4cft" Oct 30 17:28:26.855051 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:26.854968 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f07afe6-8fe0-4bc1-a6c5-02d62ae41017-istio-data\") pod \"e2e-worker-single-main-e2e8d2-openshift-default-9c5dbcc7c-q4cft\" (UID: \"2f07afe6-8fe0-4bc1-a6c5-02d62ae41017\") " pod="openshift-ingress/e2e-worker-single-main-e2e8d2-openshift-default-9c5dbcc7c-q4cft" Oct 30 17:28:26.855251 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:26.855186 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f07afe6-8fe0-4bc1-a6c5-02d62ae41017-credential-socket\") pod \"e2e-worker-single-main-e2e8d2-openshift-default-9c5dbcc7c-q4cft\" (UID: \"2f07afe6-8fe0-4bc1-a6c5-02d62ae41017\") " pod="openshift-ingress/e2e-worker-single-main-e2e8d2-openshift-default-9c5dbcc7c-q4cft" Oct 30 17:28:26.855314 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:26.855258 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f07afe6-8fe0-4bc1-a6c5-02d62ae41017-workload-socket\") pod \"e2e-worker-single-main-e2e8d2-openshift-default-9c5dbcc7c-q4cft\" (UID: \"2f07afe6-8fe0-4bc1-a6c5-02d62ae41017\") " pod="openshift-ingress/e2e-worker-single-main-e2e8d2-openshift-default-9c5dbcc7c-q4cft" Oct 30 17:28:26.855490 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:26.855452 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f07afe6-8fe0-4bc1-a6c5-02d62ae41017-workload-certs\") pod \"e2e-worker-single-main-e2e8d2-openshift-default-9c5dbcc7c-q4cft\" (UID: \"2f07afe6-8fe0-4bc1-a6c5-02d62ae41017\") " pod="openshift-ingress/e2e-worker-single-main-e2e8d2-openshift-default-9c5dbcc7c-q4cft" Oct 30 17:28:26.855629 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:26.855487 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f07afe6-8fe0-4bc1-a6c5-02d62ae41017-istio-data\") pod \"e2e-worker-single-main-e2e8d2-openshift-default-9c5dbcc7c-q4cft\" (UID: \"2f07afe6-8fe0-4bc1-a6c5-02d62ae41017\") " pod="openshift-ingress/e2e-worker-single-main-e2e8d2-openshift-default-9c5dbcc7c-q4cft" Oct 30 17:28:26.855784 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:26.855762 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f07afe6-8fe0-4bc1-a6c5-02d62ae41017-istiod-ca-cert\") pod \"e2e-worker-single-main-e2e8d2-openshift-default-9c5dbcc7c-q4cft\" (UID: \"2f07afe6-8fe0-4bc1-a6c5-02d62ae41017\") " pod="openshift-ingress/e2e-worker-single-main-e2e8d2-openshift-default-9c5dbcc7c-q4cft" Oct 30 17:28:26.858119 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:26.858095 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f07afe6-8fe0-4bc1-a6c5-02d62ae41017-istio-envoy\") pod \"e2e-worker-single-main-e2e8d2-openshift-default-9c5dbcc7c-q4cft\" (UID: \"2f07afe6-8fe0-4bc1-a6c5-02d62ae41017\") " pod="openshift-ingress/e2e-worker-single-main-e2e8d2-openshift-default-9c5dbcc7c-q4cft" Oct 30 17:28:26.858287 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:26.858268 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f07afe6-8fe0-4bc1-a6c5-02d62ae41017-istio-podinfo\") pod \"e2e-worker-single-main-e2e8d2-openshift-default-9c5dbcc7c-q4cft\" (UID: \"2f07afe6-8fe0-4bc1-a6c5-02d62ae41017\") " pod="openshift-ingress/e2e-worker-single-main-e2e8d2-openshift-default-9c5dbcc7c-q4cft" Oct 30 17:28:26.878031 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:26.878005 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f07afe6-8fe0-4bc1-a6c5-02d62ae41017-istio-token\") pod \"e2e-worker-single-main-e2e8d2-openshift-default-9c5dbcc7c-q4cft\" (UID: \"2f07afe6-8fe0-4bc1-a6c5-02d62ae41017\") " pod="openshift-ingress/e2e-worker-single-main-e2e8d2-openshift-default-9c5dbcc7c-q4cft" Oct 30 17:28:26.878175 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:26.878152 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7pw9\" (UniqueName: \"kubernetes.io/projected/2f07afe6-8fe0-4bc1-a6c5-02d62ae41017-kube-api-access-q7pw9\") pod \"e2e-worker-single-main-e2e8d2-openshift-default-9c5dbcc7c-q4cft\" (UID: \"2f07afe6-8fe0-4bc1-a6c5-02d62ae41017\") " pod="openshift-ingress/e2e-worker-single-main-e2e8d2-openshift-default-9c5dbcc7c-q4cft" Oct 30 17:28:26.985457 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:26.985359 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-e2e8d2-openshift-default-9c5dbcc7c-q4cft" Oct 30 17:28:27.305965 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:27.305934 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-e2e8d2-openshift-default-9c5dbcc7c-q4cft"] Oct 30 17:28:27.308046 ip-10-0-131-136 kubenswrapper[5052]: W1030 17:28:27.308018 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f07afe6_8fe0_4bc1_a6c5_02d62ae41017.slice/crio-5bd5a143f70ca983b9ace51064e7597a3a131286616be918fc4e38bb16536618 WatchSource:0}: Error finding container 5bd5a143f70ca983b9ace51064e7597a3a131286616be918fc4e38bb16536618: Status 404 returned error can't find the container with id 5bd5a143f70ca983b9ace51064e7597a3a131286616be918fc4e38bb16536618 Oct 30 17:28:27.310226 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:27.310195 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:28:27.310314 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:27.310260 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:28:27.310314 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:27.310297 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:28:27.977680 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:27.977634 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-e2e8d2-openshift-default-9c5dbcc7c-q4cft" event={"ID":"2f07afe6-8fe0-4bc1-a6c5-02d62ae41017","Type":"ContainerStarted","Data":"cc5813c05bcf9149f71fd35b45d7af332787c7d05937a9ad2586941a0b293010"} Oct 30 17:28:27.978122 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:27.977706 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-e2e8d2-openshift-default-9c5dbcc7c-q4cft" event={"ID":"2f07afe6-8fe0-4bc1-a6c5-02d62ae41017","Type":"ContainerStarted","Data":"5bd5a143f70ca983b9ace51064e7597a3a131286616be918fc4e38bb16536618"} Oct 30 17:28:27.986031 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:27.986009 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-e2e8d2-openshift-default-9c5dbcc7c-q4cft" Oct 30 17:28:28.036225 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:28.036182 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-e2e8d2-openshift-default-9c5dbcc7c-q4cft" podStartSLOduration=2.036161685 podStartE2EDuration="2.036161685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:28:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:28:28.033836019 +0000 UTC m=+4063.561171250" watchObservedRunningTime="2026-10-30 17:28:28.036161685 +0000 UTC m=+4063.563496903" Oct 30 17:28:28.991507 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:28.991474 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-e2e8d2-openshift-default-9c5dbcc7c-q4cft" Oct 30 17:28:29.989230 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:29.989190 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-e2e8d2-openshift-default-9c5dbcc7c-q4cft" Oct 30 17:28:29.990302 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:29.990277 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-e2e8d2-openshift-default-9c5dbcc7c-q4cft" Oct 30 17:28:31.834808 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:31.834773 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e2e8d2-687db759d9-xxv25"] Oct 30 17:28:31.839171 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:31.839140 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e2e8d2-687db759d9-xxv25" Oct 30 17:28:31.857508 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:31.857484 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-e2e8d2-serving-cert\"" Oct 30 17:28:31.868203 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:31.868163 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e2e8d2-687db759d9-xxv25"] Oct 30 17:28:31.907940 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:31.907894 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cbf5c3b-8676-46d3-9fc0-9417d3e984ad-metrics-certs\") pod \"maas-api-e2e-worker-single-main-e2e8d2-687db759d9-xxv25\" (UID: \"8cbf5c3b-8676-46d3-9fc0-9417d3e984ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e2e8d2-687db759d9-xxv25" Oct 30 17:28:31.908150 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:31.907950 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cbf5c3b-8676-46d3-9fc0-9417d3e984ad-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-e2e8d2-687db759d9-xxv25\" (UID: \"8cbf5c3b-8676-46d3-9fc0-9417d3e984ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e2e8d2-687db759d9-xxv25" Oct 30 17:28:31.908150 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:31.908045 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2cjl\" (UniqueName: \"kubernetes.io/projected/8cbf5c3b-8676-46d3-9fc0-9417d3e984ad-kube-api-access-b2cjl\") pod \"maas-api-e2e-worker-single-main-e2e8d2-687db759d9-xxv25\" (UID: \"8cbf5c3b-8676-46d3-9fc0-9417d3e984ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e2e8d2-687db759d9-xxv25" Oct 30 17:28:31.908273 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:31.908212 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8cbf5c3b-8676-46d3-9fc0-9417d3e984ad-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-e2e8d2-687db759d9-xxv25\" (UID: \"8cbf5c3b-8676-46d3-9fc0-9417d3e984ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e2e8d2-687db759d9-xxv25" Oct 30 17:28:32.009820 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:32.009779 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cbf5c3b-8676-46d3-9fc0-9417d3e984ad-metrics-certs\") pod \"maas-api-e2e-worker-single-main-e2e8d2-687db759d9-xxv25\" (UID: \"8cbf5c3b-8676-46d3-9fc0-9417d3e984ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e2e8d2-687db759d9-xxv25" Oct 30 17:28:32.009820 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:32.009824 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cbf5c3b-8676-46d3-9fc0-9417d3e984ad-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-e2e8d2-687db759d9-xxv25\" (UID: \"8cbf5c3b-8676-46d3-9fc0-9417d3e984ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e2e8d2-687db759d9-xxv25" Oct 30 17:28:32.010080 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:32.009864 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2cjl\" (UniqueName: \"kubernetes.io/projected/8cbf5c3b-8676-46d3-9fc0-9417d3e984ad-kube-api-access-b2cjl\") pod \"maas-api-e2e-worker-single-main-e2e8d2-687db759d9-xxv25\" (UID: \"8cbf5c3b-8676-46d3-9fc0-9417d3e984ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e2e8d2-687db759d9-xxv25" Oct 30 17:28:32.010080 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:32.009956 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8cbf5c3b-8676-46d3-9fc0-9417d3e984ad-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-e2e8d2-687db759d9-xxv25\" (UID: \"8cbf5c3b-8676-46d3-9fc0-9417d3e984ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e2e8d2-687db759d9-xxv25" Oct 30 17:28:32.010184 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:28:32.010135 5052 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e2e8d2-serving-cert: secret "maas-api-e2e-worker-single-main-e2e8d2-serving-cert" not found Oct 30 17:28:32.010228 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:28:32.010210 5052 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cbf5c3b-8676-46d3-9fc0-9417d3e984ad-maas-api-tls podName:8cbf5c3b-8676-46d3-9fc0-9417d3e984ad nodeName:}" failed. No retries permitted until 2026-10-30 17:28:32.510192157 +0000 UTC m=+4068.037527371 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8cbf5c3b-8676-46d3-9fc0-9417d3e984ad-maas-api-tls") pod "maas-api-e2e-worker-single-main-e2e8d2-687db759d9-xxv25" (UID: "8cbf5c3b-8676-46d3-9fc0-9417d3e984ad") : secret "maas-api-e2e-worker-single-main-e2e8d2-serving-cert" not found Oct 30 17:28:32.010639 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:32.010619 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cbf5c3b-8676-46d3-9fc0-9417d3e984ad-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-e2e8d2-687db759d9-xxv25\" (UID: \"8cbf5c3b-8676-46d3-9fc0-9417d3e984ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e2e8d2-687db759d9-xxv25" Oct 30 17:28:32.012556 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:32.012535 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cbf5c3b-8676-46d3-9fc0-9417d3e984ad-metrics-certs\") pod \"maas-api-e2e-worker-single-main-e2e8d2-687db759d9-xxv25\" (UID: \"8cbf5c3b-8676-46d3-9fc0-9417d3e984ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e2e8d2-687db759d9-xxv25" Oct 30 17:28:32.038276 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:32.038237 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2cjl\" (UniqueName: \"kubernetes.io/projected/8cbf5c3b-8676-46d3-9fc0-9417d3e984ad-kube-api-access-b2cjl\") pod \"maas-api-e2e-worker-single-main-e2e8d2-687db759d9-xxv25\" (UID: \"8cbf5c3b-8676-46d3-9fc0-9417d3e984ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e2e8d2-687db759d9-xxv25" Oct 30 17:28:32.259931 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:32.259896 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-e2e8d2-68b65wztdl"] Oct 30 17:28:32.263932 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:32.263914 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e2e8d2-68b65wztdl" Oct 30 17:28:32.281966 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:32.281921 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-083ab955-dockercfg-689cf\"" Oct 30 17:28:32.282061 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:32.282023 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-e2e8d2\"" Oct 30 17:28:32.283789 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:32.283769 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-e2e8d2-68b65wztdl"] Oct 30 17:28:32.340402 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:32.340362 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-e2e8d2-6fbfc78d9nhqt2"] Oct 30 17:28:32.346107 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:32.346082 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e2e8d2-6fbfc78d9nhqt2" Oct 30 17:28:32.363096 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:32.363069 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-e2e8d2-6fbfc78d9nhqt2"] Oct 30 17:28:32.415822 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:32.415791 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2stsg\" (UniqueName: \"kubernetes.io/projected/313490c0-56de-49f8-918c-6a9169c79f3f-kube-api-access-2stsg\") pod \"payload-processing-e2e-worker-single-main-e2e8d2-6fbfc78d9nhqt2\" (UID: \"313490c0-56de-49f8-918c-6a9169c79f3f\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-e2e8d2-6fbfc78d9nhqt2" Oct 30 17:28:32.415980 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:32.415855 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6f6c200-4666-456f-913b-e98f3fa15fa1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-e2e8d2-68b65wztdl\" (UID: \"b6f6c200-4666-456f-913b-e98f3fa15fa1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e2e8d2-68b65wztdl" Oct 30 17:28:32.416032 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:32.415975 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/313490c0-56de-49f8-918c-6a9169c79f3f-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-e2e8d2-6fbfc78d9nhqt2\" (UID: \"313490c0-56de-49f8-918c-6a9169c79f3f\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-e2e8d2-6fbfc78d9nhqt2" Oct 30 17:28:32.416032 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:32.416015 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqf2l\" (UniqueName: \"kubernetes.io/projected/b6f6c200-4666-456f-913b-e98f3fa15fa1-kube-api-access-wqf2l\") pod \"payload-pre-processing-e2e-worker-single-main-e2e8d2-68b65wztdl\" (UID: \"b6f6c200-4666-456f-913b-e98f3fa15fa1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e2e8d2-68b65wztdl" Oct 30 17:28:32.517327 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:32.517234 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2stsg\" (UniqueName: \"kubernetes.io/projected/313490c0-56de-49f8-918c-6a9169c79f3f-kube-api-access-2stsg\") pod \"payload-processing-e2e-worker-single-main-e2e8d2-6fbfc78d9nhqt2\" (UID: \"313490c0-56de-49f8-918c-6a9169c79f3f\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-e2e8d2-6fbfc78d9nhqt2" Oct 30 17:28:32.517327 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:32.517297 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6f6c200-4666-456f-913b-e98f3fa15fa1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-e2e8d2-68b65wztdl\" (UID: \"b6f6c200-4666-456f-913b-e98f3fa15fa1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e2e8d2-68b65wztdl" Oct 30 17:28:32.517635 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:32.517344 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/313490c0-56de-49f8-918c-6a9169c79f3f-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-e2e8d2-6fbfc78d9nhqt2\" (UID: \"313490c0-56de-49f8-918c-6a9169c79f3f\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-e2e8d2-6fbfc78d9nhqt2" Oct 30 17:28:32.517635 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:32.517365 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqf2l\" (UniqueName: \"kubernetes.io/projected/b6f6c200-4666-456f-913b-e98f3fa15fa1-kube-api-access-wqf2l\") pod \"payload-pre-processing-e2e-worker-single-main-e2e8d2-68b65wztdl\" (UID: \"b6f6c200-4666-456f-913b-e98f3fa15fa1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e2e8d2-68b65wztdl" Oct 30 17:28:32.517635 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:32.517411 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8cbf5c3b-8676-46d3-9fc0-9417d3e984ad-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-e2e8d2-687db759d9-xxv25\" (UID: \"8cbf5c3b-8676-46d3-9fc0-9417d3e984ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e2e8d2-687db759d9-xxv25" Oct 30 17:28:32.518201 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:32.518174 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/313490c0-56de-49f8-918c-6a9169c79f3f-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-e2e8d2-6fbfc78d9nhqt2\" (UID: \"313490c0-56de-49f8-918c-6a9169c79f3f\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-e2e8d2-6fbfc78d9nhqt2" Oct 30 17:28:32.518201 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:32.518184 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6f6c200-4666-456f-913b-e98f3fa15fa1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-e2e8d2-68b65wztdl\" (UID: \"b6f6c200-4666-456f-913b-e98f3fa15fa1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e2e8d2-68b65wztdl" Oct 30 17:28:32.519854 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:32.519829 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8cbf5c3b-8676-46d3-9fc0-9417d3e984ad-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-e2e8d2-687db759d9-xxv25\" (UID: \"8cbf5c3b-8676-46d3-9fc0-9417d3e984ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e2e8d2-687db759d9-xxv25" Oct 30 17:28:32.540282 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:32.540253 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqf2l\" (UniqueName: \"kubernetes.io/projected/b6f6c200-4666-456f-913b-e98f3fa15fa1-kube-api-access-wqf2l\") pod \"payload-pre-processing-e2e-worker-single-main-e2e8d2-68b65wztdl\" (UID: \"b6f6c200-4666-456f-913b-e98f3fa15fa1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e2e8d2-68b65wztdl" Oct 30 17:28:32.540398 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:32.540308 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2stsg\" (UniqueName: \"kubernetes.io/projected/313490c0-56de-49f8-918c-6a9169c79f3f-kube-api-access-2stsg\") pod \"payload-processing-e2e-worker-single-main-e2e8d2-6fbfc78d9nhqt2\" (UID: \"313490c0-56de-49f8-918c-6a9169c79f3f\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-e2e8d2-6fbfc78d9nhqt2" Oct 30 17:28:32.577467 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:32.577428 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e2e8d2-68b65wztdl" Oct 30 17:28:32.671132 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:32.670852 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e2e8d2-6fbfc78d9nhqt2" Oct 30 17:28:32.753140 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:32.752937 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e2e8d2-687db759d9-xxv25" Oct 30 17:28:32.878653 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:32.878617 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-e2e8d2-68b65wztdl"] Oct 30 17:28:32.882358 ip-10-0-131-136 kubenswrapper[5052]: W1030 17:28:32.882325 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6f6c200_4666_456f_913b_e98f3fa15fa1.slice/crio-5e110aa56eee72a3cb0239ca0d4f154fb7aeb38abc6754da8ef2051a7f001a58 WatchSource:0}: Error finding container 5e110aa56eee72a3cb0239ca0d4f154fb7aeb38abc6754da8ef2051a7f001a58: Status 404 returned error can't find the container with id 5e110aa56eee72a3cb0239ca0d4f154fb7aeb38abc6754da8ef2051a7f001a58 Oct 30 17:28:32.977496 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:32.977467 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-e2e8d2-6fbfc78d9nhqt2"] Oct 30 17:28:32.981148 ip-10-0-131-136 kubenswrapper[5052]: W1030 17:28:32.981120 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod313490c0_56de_49f8_918c_6a9169c79f3f.slice/crio-74872a679cc6e8745e8eb95599281023aebd1cd544d36478ba2a5dce0f609cb2 WatchSource:0}: Error finding container 74872a679cc6e8745e8eb95599281023aebd1cd544d36478ba2a5dce0f609cb2: Status 404 returned error can't find the container with id 74872a679cc6e8745e8eb95599281023aebd1cd544d36478ba2a5dce0f609cb2 Oct 30 17:28:33.005390 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:33.005365 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e2e8d2-68b65wztdl" event={"ID":"b6f6c200-4666-456f-913b-e98f3fa15fa1","Type":"ContainerStarted","Data":"af7ae2e3b1103e0adf175efc80c25a1adfda72686cfd7257aa1210db05a62639"} Oct 30 17:28:33.005501 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:33.005412 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e2e8d2-68b65wztdl" event={"ID":"b6f6c200-4666-456f-913b-e98f3fa15fa1","Type":"ContainerStarted","Data":"5e110aa56eee72a3cb0239ca0d4f154fb7aeb38abc6754da8ef2051a7f001a58"} Oct 30 17:28:33.005501 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:33.005462 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e2e8d2-68b65wztdl" Oct 30 17:28:33.006688 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:33.006668 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e2e8d2-6fbfc78d9nhqt2" event={"ID":"313490c0-56de-49f8-918c-6a9169c79f3f","Type":"ContainerStarted","Data":"74872a679cc6e8745e8eb95599281023aebd1cd544d36478ba2a5dce0f609cb2"} Oct 30 17:28:33.064987 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:33.064931 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e2e8d2-68b65wztdl" podStartSLOduration=1.064915908 podStartE2EDuration="1.064915908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:28:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:28:33.049532404 +0000 UTC m=+4068.576867625" watchObservedRunningTime="2026-10-30 17:28:33.064915908 +0000 UTC m=+4068.592251126" Oct 30 17:28:33.074837 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:33.067771 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e2e8d2-687db759d9-xxv25"] Oct 30 17:28:33.078407 ip-10-0-131-136 kubenswrapper[5052]: W1030 17:28:33.078374 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cbf5c3b_8676_46d3_9fc0_9417d3e984ad.slice/crio-76984b00fb1d6b15c379d4e81d1bef8134d71232c1dbe8a382bd532222e57cfa WatchSource:0}: Error finding container 76984b00fb1d6b15c379d4e81d1bef8134d71232c1dbe8a382bd532222e57cfa: Status 404 returned error can't find the container with id 76984b00fb1d6b15c379d4e81d1bef8134d71232c1dbe8a382bd532222e57cfa Oct 30 17:28:34.013672 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:34.013629 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e2e8d2-6fbfc78d9nhqt2" event={"ID":"313490c0-56de-49f8-918c-6a9169c79f3f","Type":"ContainerStarted","Data":"fbe3c1bf44f2a6a581772007cabe913eb9f07c77ee6dbf876af098ef73fc498e"} Oct 30 17:28:34.014208 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:34.013717 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e2e8d2-6fbfc78d9nhqt2" Oct 30 17:28:34.015423 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:34.015398 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e2e8d2-687db759d9-xxv25" event={"ID":"8cbf5c3b-8676-46d3-9fc0-9417d3e984ad","Type":"ContainerStarted","Data":"e794854985d722ed7344cd8d1aa1190aee7b136cfa462776a9ea47e95b81827c"} Oct 30 17:28:34.015552 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:34.015441 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e2e8d2-687db759d9-xxv25" event={"ID":"8cbf5c3b-8676-46d3-9fc0-9417d3e984ad","Type":"ContainerStarted","Data":"76984b00fb1d6b15c379d4e81d1bef8134d71232c1dbe8a382bd532222e57cfa"} Oct 30 17:28:34.054328 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:34.054251 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e2e8d2-6fbfc78d9nhqt2" podStartSLOduration=2.054230681 podStartE2EDuration="2.054230681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:28:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:28:34.052986092 +0000 UTC m=+4069.580321314" watchObservedRunningTime="2026-10-30 17:28:34.054230681 +0000 UTC m=+4069.581565902" Oct 30 17:28:34.094501 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:34.094443 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e2e8d2-687db759d9-xxv25" podStartSLOduration=2.6432424389999998 podStartE2EDuration="3.09442517s" totalImagesPullingTime="451.182731ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:28:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:28:34.094064874 +0000 UTC m=+4069.621400093" watchObservedRunningTime="2026-10-30 17:28:34.09442517 +0000 UTC m=+4069.621760370" Oct 30 17:28:35.021562 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:35.021520 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e2e8d2-687db759d9-xxv25" Oct 30 17:28:35.434437 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:35.434399 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cdwcj"] Oct 30 17:28:41.054287 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:41.054254 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e2e8d2-687db759d9-xxv25" Oct 30 17:28:44.018960 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:44.018931 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e2e8d2-68b65wztdl" Oct 30 17:28:45.025362 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:45.025333 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e2e8d2-6fbfc78d9nhqt2" Oct 30 17:28:45.268905 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:45.267936 5052 patch_prober.go:29] interesting pod/e2e-worker-single-main-e2e8d2-openshift-default-9c5dbcc7c-q4cft container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": read tcp 10.132.0.2:34192->10.132.0.244:15021: read: connection reset by peer" start-of-body= Oct 30 17:28:45.268905 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:45.268001 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-e2e8d2-openshift-default-9c5dbcc7c-q4cft" podUID="2f07afe6-8fe0-4bc1-a6c5-02d62ae41017" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": read tcp 10.132.0.2:34192->10.132.0.244:15021: read: connection reset by peer" Oct 30 17:28:46.120468 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:46.120436 5052 generic.go:356] "Generic (PLEG): container finished" podID="2f07afe6-8fe0-4bc1-a6c5-02d62ae41017" containerID="cc5813c05bcf9149f71fd35b45d7af332787c7d05937a9ad2586941a0b293010" exitCode=0 Oct 30 17:28:46.120888 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:46.120504 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-e2e8d2-openshift-default-9c5dbcc7c-q4cft" event={"ID":"2f07afe6-8fe0-4bc1-a6c5-02d62ae41017","Type":"ContainerDied","Data":"cc5813c05bcf9149f71fd35b45d7af332787c7d05937a9ad2586941a0b293010"} Oct 30 17:28:46.120940 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:46.120887 5052 scope.go:122] "RemoveContainer" containerID="cc5813c05bcf9149f71fd35b45d7af332787c7d05937a9ad2586941a0b293010" Oct 30 17:28:46.121995 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:46.121963 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:28:46.122087 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:46.122034 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:28:46.122087 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:46.122068 5052 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:28:47.141661 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:47.141620 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-e2e8d2-openshift-default-9c5dbcc7c-q4cft" event={"ID":"2f07afe6-8fe0-4bc1-a6c5-02d62ae41017","Type":"ContainerStarted","Data":"4ffd20bffa387b72eeae02fdca4c73a59589663f8f0534c53f0bff01e325dc20"} Oct 30 17:28:47.986433 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:47.986395 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-e2e8d2-openshift-default-9c5dbcc7c-q4cft" Oct 30 17:28:47.986655 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:47.986570 5052 patch_prober.go:29] interesting pod/e2e-worker-single-main-e2e8d2-openshift-default-9c5dbcc7c-q4cft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" start-of-body= Oct 30 17:28:47.986655 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:47.986621 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-e2e8d2-openshift-default-9c5dbcc7c-q4cft" podUID="2f07afe6-8fe0-4bc1-a6c5-02d62ae41017" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" Oct 30 17:28:48.862499 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:28:48.862451 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 17:28:48.986708 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:48.986670 5052 patch_prober.go:29] interesting pod/e2e-worker-single-main-e2e8d2-openshift-default-9c5dbcc7c-q4cft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" start-of-body= Oct 30 17:28:48.986917 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:48.986735 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-e2e8d2-openshift-default-9c5dbcc7c-q4cft" podUID="2f07afe6-8fe0-4bc1-a6c5-02d62ae41017" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" Oct 30 17:28:49.986083 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:49.986040 5052 patch_prober.go:29] interesting pod/e2e-worker-single-main-e2e8d2-openshift-default-9c5dbcc7c-q4cft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" start-of-body= Oct 30 17:28:49.986488 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:49.986113 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-e2e8d2-openshift-default-9c5dbcc7c-q4cft" podUID="2f07afe6-8fe0-4bc1-a6c5-02d62ae41017" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" Oct 30 17:28:50.115744 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:50.115700 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cdwcj"] Oct 30 17:28:50.986553 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:50.986506 5052 patch_prober.go:29] interesting pod/e2e-worker-single-main-e2e8d2-openshift-default-9c5dbcc7c-q4cft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" start-of-body= Oct 30 17:28:50.987097 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:50.986571 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-e2e8d2-openshift-default-9c5dbcc7c-q4cft" podUID="2f07afe6-8fe0-4bc1-a6c5-02d62ae41017" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" Oct 30 17:28:51.986088 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:51.986049 5052 patch_prober.go:29] interesting pod/e2e-worker-single-main-e2e8d2-openshift-default-9c5dbcc7c-q4cft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" start-of-body= Oct 30 17:28:51.986289 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:51.986109 5052 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-e2e8d2-openshift-default-9c5dbcc7c-q4cft" podUID="2f07afe6-8fe0-4bc1-a6c5-02d62ae41017" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" Oct 30 17:28:52.989372 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:52.989338 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-e2e8d2-openshift-default-9c5dbcc7c-q4cft" Oct 30 17:28:52.989903 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:52.989735 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-e2e8d2-openshift-default-9c5dbcc7c-q4cft" Oct 30 17:28:52.990336 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:28:52.990315 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-e2e8d2-openshift-default-9c5dbcc7c-q4cft" Oct 30 17:29:05.173325 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:05.173284 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cdwcj"] Oct 30 17:29:18.713857 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:18.713818 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cdwcj"] Oct 30 17:29:18.853001 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:29:18.852956 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 17:29:38.629492 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:38.629460 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-e2e8d2-6fbfc78d9nhqt2"] Oct 30 17:29:38.663634 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:38.663603 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-e2e8d2-68b65wztdl"] Oct 30 17:29:40.196394 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:40.196357 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e2e8d2-687db759d9-xxv25"] Oct 30 17:29:40.196929 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:40.196665 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e2e8d2-687db759d9-xxv25" podUID="8cbf5c3b-8676-46d3-9fc0-9417d3e984ad" containerName="maas-api" containerID="cri-o://e794854985d722ed7344cd8d1aa1190aee7b136cfa462776a9ea47e95b81827c" gracePeriod=30 Oct 30 17:29:40.691083 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:40.691053 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e2e8d2-687db759d9-xxv25" Oct 30 17:29:40.737645 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:40.736623 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8cbf5c3b-8676-46d3-9fc0-9417d3e984ad-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8cbf5c3b-8676-46d3-9fc0-9417d3e984ad-maas-api-tls\") pod \"8cbf5c3b-8676-46d3-9fc0-9417d3e984ad\" (UID: \"8cbf5c3b-8676-46d3-9fc0-9417d3e984ad\") " Oct 30 17:29:40.737645 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:40.736754 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8cbf5c3b-8676-46d3-9fc0-9417d3e984ad-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cbf5c3b-8676-46d3-9fc0-9417d3e984ad-openshift-service-ca\") pod \"8cbf5c3b-8676-46d3-9fc0-9417d3e984ad\" (UID: \"8cbf5c3b-8676-46d3-9fc0-9417d3e984ad\") " Oct 30 17:29:40.737645 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:40.736835 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cbf5c3b-8676-46d3-9fc0-9417d3e984ad-kube-api-access-b2cjl\" (UniqueName: \"kubernetes.io/projected/8cbf5c3b-8676-46d3-9fc0-9417d3e984ad-kube-api-access-b2cjl\") pod \"8cbf5c3b-8676-46d3-9fc0-9417d3e984ad\" (UID: \"8cbf5c3b-8676-46d3-9fc0-9417d3e984ad\") " Oct 30 17:29:40.737645 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:40.736912 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8cbf5c3b-8676-46d3-9fc0-9417d3e984ad-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cbf5c3b-8676-46d3-9fc0-9417d3e984ad-metrics-certs\") pod \"8cbf5c3b-8676-46d3-9fc0-9417d3e984ad\" (UID: \"8cbf5c3b-8676-46d3-9fc0-9417d3e984ad\") " Oct 30 17:29:40.737645 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:40.737513 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cbf5c3b-8676-46d3-9fc0-9417d3e984ad-openshift-service-ca" pod "8cbf5c3b-8676-46d3-9fc0-9417d3e984ad" (UID: "8cbf5c3b-8676-46d3-9fc0-9417d3e984ad"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:29:40.739022 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:40.738995 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cbf5c3b-8676-46d3-9fc0-9417d3e984ad-maas-api-tls" pod "8cbf5c3b-8676-46d3-9fc0-9417d3e984ad" (UID: "8cbf5c3b-8676-46d3-9fc0-9417d3e984ad"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:29:40.739986 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:40.739957 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cbf5c3b-8676-46d3-9fc0-9417d3e984ad-metrics-certs" pod "8cbf5c3b-8676-46d3-9fc0-9417d3e984ad" (UID: "8cbf5c3b-8676-46d3-9fc0-9417d3e984ad"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:29:40.740638 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:40.740569 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cbf5c3b-8676-46d3-9fc0-9417d3e984ad-kube-api-access-b2cjl" pod "8cbf5c3b-8676-46d3-9fc0-9417d3e984ad" (UID: "8cbf5c3b-8676-46d3-9fc0-9417d3e984ad"). InnerVolumeSpecName "kube-api-access-b2cjl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:29:40.838290 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:40.838255 5052 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cbf5c3b-8676-46d3-9fc0-9417d3e984ad-openshift-service-ca\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:29:40.838290 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:40.838287 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2cjl\" (UniqueName: \"kubernetes.io/projected/8cbf5c3b-8676-46d3-9fc0-9417d3e984ad-kube-api-access-b2cjl\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:29:40.838488 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:40.838300 5052 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cbf5c3b-8676-46d3-9fc0-9417d3e984ad-metrics-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:29:40.838488 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:40.838309 5052 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8cbf5c3b-8676-46d3-9fc0-9417d3e984ad-maas-api-tls\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:29:41.457107 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:41.457065 5052 generic.go:356] "Generic (PLEG): container finished" podID="8cbf5c3b-8676-46d3-9fc0-9417d3e984ad" containerID="e794854985d722ed7344cd8d1aa1190aee7b136cfa462776a9ea47e95b81827c" exitCode=0 Oct 30 17:29:41.457629 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:41.457118 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e2e8d2-687db759d9-xxv25" Oct 30 17:29:41.457629 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:41.457113 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e2e8d2-687db759d9-xxv25" event={"ID":"8cbf5c3b-8676-46d3-9fc0-9417d3e984ad","Type":"ContainerDied","Data":"e794854985d722ed7344cd8d1aa1190aee7b136cfa462776a9ea47e95b81827c"} Oct 30 17:29:41.457629 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:41.457204 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e2e8d2-687db759d9-xxv25" event={"ID":"8cbf5c3b-8676-46d3-9fc0-9417d3e984ad","Type":"ContainerDied","Data":"76984b00fb1d6b15c379d4e81d1bef8134d71232c1dbe8a382bd532222e57cfa"} Oct 30 17:29:41.457629 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:41.457296 5052 scope.go:122] "RemoveContainer" containerID="e794854985d722ed7344cd8d1aa1190aee7b136cfa462776a9ea47e95b81827c" Oct 30 17:29:41.468079 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:41.468062 5052 scope.go:122] "RemoveContainer" containerID="e794854985d722ed7344cd8d1aa1190aee7b136cfa462776a9ea47e95b81827c" Oct 30 17:29:41.468365 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:29:41.468343 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e794854985d722ed7344cd8d1aa1190aee7b136cfa462776a9ea47e95b81827c\": container with ID starting with e794854985d722ed7344cd8d1aa1190aee7b136cfa462776a9ea47e95b81827c not found: ID does not exist" containerID="e794854985d722ed7344cd8d1aa1190aee7b136cfa462776a9ea47e95b81827c" Oct 30 17:29:41.468419 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:41.468373 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e794854985d722ed7344cd8d1aa1190aee7b136cfa462776a9ea47e95b81827c"} err="failed to get container status \"e794854985d722ed7344cd8d1aa1190aee7b136cfa462776a9ea47e95b81827c\": rpc error: code = NotFound desc = could not find container \"e794854985d722ed7344cd8d1aa1190aee7b136cfa462776a9ea47e95b81827c\": container with ID starting with e794854985d722ed7344cd8d1aa1190aee7b136cfa462776a9ea47e95b81827c not found: ID does not exist" Oct 30 17:29:41.509638 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:41.509618 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e2e8d2-687db759d9-xxv25"] Oct 30 17:29:41.515468 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:41.515449 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e2e8d2-687db759d9-xxv25"] Oct 30 17:29:42.516068 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:42.516027 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cdwcj"] Oct 30 17:29:42.597978 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:42.597947 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8cbf5c3b-8676-46d3-9fc0-9417d3e984ad" path="/var/lib/kubelet/pods/8cbf5c3b-8676-46d3-9fc0-9417d3e984ad/volumes" Oct 30 17:29:43.629897 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:43.629855 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e2e8d2-6fbfc78d9nhqt2" podUID="313490c0-56de-49f8-918c-6a9169c79f3f" containerName="payload-processing" containerID="cri-o://fbe3c1bf44f2a6a581772007cabe913eb9f07c77ee6dbf876af098ef73fc498e" gracePeriod=25 Oct 30 17:29:43.664588 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:43.664541 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e2e8d2-68b65wztdl" podUID="b6f6c200-4666-456f-913b-e98f3fa15fa1" containerName="payload-pre-processing" containerID="cri-o://af7ae2e3b1103e0adf175efc80c25a1adfda72686cfd7257aa1210db05a62639" gracePeriod=25 Oct 30 17:29:44.016836 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:44.016792 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e2e8d2-68b65wztdl" podUID="b6f6c200-4666-456f-913b-e98f3fa15fa1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:29:45.022935 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:45.022890 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e2e8d2-6fbfc78d9nhqt2" podUID="313490c0-56de-49f8-918c-6a9169c79f3f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:29:47.951372 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:47.951336 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-e2e8d2-openshift-default-9c5dbcc7c-q4cft"] Oct 30 17:29:47.951904 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:47.951618 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-e2e8d2-openshift-default-9c5dbcc7c-q4cft" podUID="2f07afe6-8fe0-4bc1-a6c5-02d62ae41017" containerName="istio-proxy" containerID="cri-o://4ffd20bffa387b72eeae02fdca4c73a59589663f8f0534c53f0bff01e325dc20" gracePeriod=30 Oct 30 17:29:48.841071 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:29:48.841034 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 17:29:52.991270 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:52.991239 5052 patch_prober.go:29] interesting pod/e2e-worker-single-main-e2e8d2-openshift-default-9c5dbcc7c-q4cft container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" start-of-body= Oct 30 17:29:52.991643 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:52.991290 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-e2e8d2-openshift-default-9c5dbcc7c-q4cft" podUID="2f07afe6-8fe0-4bc1-a6c5-02d62ae41017" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" Oct 30 17:29:53.229734 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:53.229679 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-e2e8d2-openshift-default-9c5dbcc7c-q4cft" Oct 30 17:29:53.368009 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:53.367977 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f07afe6-8fe0-4bc1-a6c5-02d62ae41017-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f07afe6-8fe0-4bc1-a6c5-02d62ae41017-istio-envoy\") pod \"2f07afe6-8fe0-4bc1-a6c5-02d62ae41017\" (UID: \"2f07afe6-8fe0-4bc1-a6c5-02d62ae41017\") " Oct 30 17:29:53.368009 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:53.368013 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f07afe6-8fe0-4bc1-a6c5-02d62ae41017-kube-api-access-q7pw9\" (UniqueName: \"kubernetes.io/projected/2f07afe6-8fe0-4bc1-a6c5-02d62ae41017-kube-api-access-q7pw9\") pod \"2f07afe6-8fe0-4bc1-a6c5-02d62ae41017\" (UID: \"2f07afe6-8fe0-4bc1-a6c5-02d62ae41017\") " Oct 30 17:29:53.368265 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:53.368036 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f07afe6-8fe0-4bc1-a6c5-02d62ae41017-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f07afe6-8fe0-4bc1-a6c5-02d62ae41017-istio-data\") pod \"2f07afe6-8fe0-4bc1-a6c5-02d62ae41017\" (UID: \"2f07afe6-8fe0-4bc1-a6c5-02d62ae41017\") " Oct 30 17:29:53.368265 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:53.368128 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f07afe6-8fe0-4bc1-a6c5-02d62ae41017-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f07afe6-8fe0-4bc1-a6c5-02d62ae41017-workload-socket\") pod \"2f07afe6-8fe0-4bc1-a6c5-02d62ae41017\" (UID: \"2f07afe6-8fe0-4bc1-a6c5-02d62ae41017\") " Oct 30 17:29:53.368265 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:53.368171 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f07afe6-8fe0-4bc1-a6c5-02d62ae41017-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f07afe6-8fe0-4bc1-a6c5-02d62ae41017-istiod-ca-cert\") pod \"2f07afe6-8fe0-4bc1-a6c5-02d62ae41017\" (UID: \"2f07afe6-8fe0-4bc1-a6c5-02d62ae41017\") " Oct 30 17:29:53.368265 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:53.368222 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f07afe6-8fe0-4bc1-a6c5-02d62ae41017-istio-token\" (UniqueName: \"kubernetes.io/projected/2f07afe6-8fe0-4bc1-a6c5-02d62ae41017-istio-token\") pod \"2f07afe6-8fe0-4bc1-a6c5-02d62ae41017\" (UID: \"2f07afe6-8fe0-4bc1-a6c5-02d62ae41017\") " Oct 30 17:29:53.368505 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:53.368265 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f07afe6-8fe0-4bc1-a6c5-02d62ae41017-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f07afe6-8fe0-4bc1-a6c5-02d62ae41017-credential-socket\") pod \"2f07afe6-8fe0-4bc1-a6c5-02d62ae41017\" (UID: \"2f07afe6-8fe0-4bc1-a6c5-02d62ae41017\") " Oct 30 17:29:53.368505 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:53.368286 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f07afe6-8fe0-4bc1-a6c5-02d62ae41017-istio-data" pod "2f07afe6-8fe0-4bc1-a6c5-02d62ae41017" (UID: "2f07afe6-8fe0-4bc1-a6c5-02d62ae41017"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:29:53.368505 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:53.368385 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2f07afe6-8fe0-4bc1-a6c5-02d62ae41017-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f07afe6-8fe0-4bc1-a6c5-02d62ae41017-istio-podinfo\") pod \"2f07afe6-8fe0-4bc1-a6c5-02d62ae41017\" (UID: \"2f07afe6-8fe0-4bc1-a6c5-02d62ae41017\") " Oct 30 17:29:53.368505 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:53.368452 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f07afe6-8fe0-4bc1-a6c5-02d62ae41017-workload-socket" pod "2f07afe6-8fe0-4bc1-a6c5-02d62ae41017" (UID: "2f07afe6-8fe0-4bc1-a6c5-02d62ae41017"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:29:53.368772 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:53.368511 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f07afe6-8fe0-4bc1-a6c5-02d62ae41017-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f07afe6-8fe0-4bc1-a6c5-02d62ae41017-workload-certs\") pod \"2f07afe6-8fe0-4bc1-a6c5-02d62ae41017\" (UID: \"2f07afe6-8fe0-4bc1-a6c5-02d62ae41017\") " Oct 30 17:29:53.368772 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:53.368576 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f07afe6-8fe0-4bc1-a6c5-02d62ae41017-credential-socket" pod "2f07afe6-8fe0-4bc1-a6c5-02d62ae41017" (UID: "2f07afe6-8fe0-4bc1-a6c5-02d62ae41017"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:29:53.368772 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:53.368624 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f07afe6-8fe0-4bc1-a6c5-02d62ae41017-istiod-ca-cert" pod "2f07afe6-8fe0-4bc1-a6c5-02d62ae41017" (UID: "2f07afe6-8fe0-4bc1-a6c5-02d62ae41017"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:29:53.368950 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:53.368888 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f07afe6-8fe0-4bc1-a6c5-02d62ae41017-workload-certs" pod "2f07afe6-8fe0-4bc1-a6c5-02d62ae41017" (UID: "2f07afe6-8fe0-4bc1-a6c5-02d62ae41017"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:29:53.368950 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:53.368909 5052 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f07afe6-8fe0-4bc1-a6c5-02d62ae41017-istio-data\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:29:53.368950 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:53.368933 5052 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f07afe6-8fe0-4bc1-a6c5-02d62ae41017-workload-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:29:53.368950 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:53.368945 5052 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f07afe6-8fe0-4bc1-a6c5-02d62ae41017-istiod-ca-cert\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:29:53.369172 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:53.368956 5052 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f07afe6-8fe0-4bc1-a6c5-02d62ae41017-credential-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:29:53.370513 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:53.370482 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f07afe6-8fe0-4bc1-a6c5-02d62ae41017-istio-token" pod "2f07afe6-8fe0-4bc1-a6c5-02d62ae41017" (UID: "2f07afe6-8fe0-4bc1-a6c5-02d62ae41017"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:29:53.370653 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:53.370541 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2f07afe6-8fe0-4bc1-a6c5-02d62ae41017-istio-podinfo" pod "2f07afe6-8fe0-4bc1-a6c5-02d62ae41017" (UID: "2f07afe6-8fe0-4bc1-a6c5-02d62ae41017"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:29:53.370801 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:53.370780 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f07afe6-8fe0-4bc1-a6c5-02d62ae41017-istio-envoy" pod "2f07afe6-8fe0-4bc1-a6c5-02d62ae41017" (UID: "2f07afe6-8fe0-4bc1-a6c5-02d62ae41017"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:29:53.370897 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:53.370797 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f07afe6-8fe0-4bc1-a6c5-02d62ae41017-kube-api-access-q7pw9" pod "2f07afe6-8fe0-4bc1-a6c5-02d62ae41017" (UID: "2f07afe6-8fe0-4bc1-a6c5-02d62ae41017"). InnerVolumeSpecName "kube-api-access-q7pw9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:29:53.469470 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:53.469439 5052 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f07afe6-8fe0-4bc1-a6c5-02d62ae41017-workload-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:29:53.469470 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:53.469465 5052 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f07afe6-8fe0-4bc1-a6c5-02d62ae41017-istio-envoy\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:29:53.469470 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:53.469475 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7pw9\" (UniqueName: \"kubernetes.io/projected/2f07afe6-8fe0-4bc1-a6c5-02d62ae41017-kube-api-access-q7pw9\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:29:53.469716 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:53.469484 5052 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f07afe6-8fe0-4bc1-a6c5-02d62ae41017-istio-token\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:29:53.469716 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:53.469493 5052 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f07afe6-8fe0-4bc1-a6c5-02d62ae41017-istio-podinfo\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:29:53.526032 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:53.525951 5052 generic.go:356] "Generic (PLEG): container finished" podID="2f07afe6-8fe0-4bc1-a6c5-02d62ae41017" containerID="4ffd20bffa387b72eeae02fdca4c73a59589663f8f0534c53f0bff01e325dc20" exitCode=0 Oct 30 17:29:53.526032 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:53.526001 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-e2e8d2-openshift-default-9c5dbcc7c-q4cft" Oct 30 17:29:53.526224 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:53.525999 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-e2e8d2-openshift-default-9c5dbcc7c-q4cft" event={"ID":"2f07afe6-8fe0-4bc1-a6c5-02d62ae41017","Type":"ContainerDied","Data":"4ffd20bffa387b72eeae02fdca4c73a59589663f8f0534c53f0bff01e325dc20"} Oct 30 17:29:53.526224 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:53.526130 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-e2e8d2-openshift-default-9c5dbcc7c-q4cft" event={"ID":"2f07afe6-8fe0-4bc1-a6c5-02d62ae41017","Type":"ContainerDied","Data":"5bd5a143f70ca983b9ace51064e7597a3a131286616be918fc4e38bb16536618"} Oct 30 17:29:53.526224 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:53.526173 5052 scope.go:122] "RemoveContainer" containerID="4ffd20bffa387b72eeae02fdca4c73a59589663f8f0534c53f0bff01e325dc20" Oct 30 17:29:53.535538 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:53.535517 5052 scope.go:122] "RemoveContainer" containerID="cc5813c05bcf9149f71fd35b45d7af332787c7d05937a9ad2586941a0b293010" Oct 30 17:29:53.544328 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:53.544310 5052 scope.go:122] "RemoveContainer" containerID="4ffd20bffa387b72eeae02fdca4c73a59589663f8f0534c53f0bff01e325dc20" Oct 30 17:29:53.544595 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:29:53.544559 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ffd20bffa387b72eeae02fdca4c73a59589663f8f0534c53f0bff01e325dc20\": container with ID starting with 4ffd20bffa387b72eeae02fdca4c73a59589663f8f0534c53f0bff01e325dc20 not found: ID does not exist" containerID="4ffd20bffa387b72eeae02fdca4c73a59589663f8f0534c53f0bff01e325dc20" Oct 30 17:29:53.544683 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:53.544617 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ffd20bffa387b72eeae02fdca4c73a59589663f8f0534c53f0bff01e325dc20"} err="failed to get container status \"4ffd20bffa387b72eeae02fdca4c73a59589663f8f0534c53f0bff01e325dc20\": rpc error: code = NotFound desc = could not find container \"4ffd20bffa387b72eeae02fdca4c73a59589663f8f0534c53f0bff01e325dc20\": container with ID starting with 4ffd20bffa387b72eeae02fdca4c73a59589663f8f0534c53f0bff01e325dc20 not found: ID does not exist" Oct 30 17:29:53.544683 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:53.544641 5052 scope.go:122] "RemoveContainer" containerID="cc5813c05bcf9149f71fd35b45d7af332787c7d05937a9ad2586941a0b293010" Oct 30 17:29:53.544873 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:29:53.544856 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc5813c05bcf9149f71fd35b45d7af332787c7d05937a9ad2586941a0b293010\": container with ID starting with cc5813c05bcf9149f71fd35b45d7af332787c7d05937a9ad2586941a0b293010 not found: ID does not exist" containerID="cc5813c05bcf9149f71fd35b45d7af332787c7d05937a9ad2586941a0b293010" Oct 30 17:29:53.544935 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:53.544882 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc5813c05bcf9149f71fd35b45d7af332787c7d05937a9ad2586941a0b293010"} err="failed to get container status \"cc5813c05bcf9149f71fd35b45d7af332787c7d05937a9ad2586941a0b293010\": rpc error: code = NotFound desc = could not find container \"cc5813c05bcf9149f71fd35b45d7af332787c7d05937a9ad2586941a0b293010\": container with ID starting with cc5813c05bcf9149f71fd35b45d7af332787c7d05937a9ad2586941a0b293010 not found: ID does not exist" Oct 30 17:29:53.576738 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:53.576717 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-e2e8d2-openshift-default-9c5dbcc7c-q4cft"] Oct 30 17:29:53.588518 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:53.588498 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-e2e8d2-openshift-default-9c5dbcc7c-q4cft"] Oct 30 17:29:53.687345 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:53.687318 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-35a4d69a-kserve-797db9f4fc-5zw5g"] Oct 30 17:29:53.688086 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:53.688053 5052 cpu_manager.go:414] "containerMap: removing container" podUID="8cbf5c3b-8676-46d3-9fc0-9417d3e984ad" containerName="maas-api" Oct 30 17:29:53.688086 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:53.688075 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cbf5c3b-8676-46d3-9fc0-9417d3e984ad" containerName="maas-api" Oct 30 17:29:53.688243 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:53.688092 5052 cpu_manager.go:414] "containerMap: removing container" podUID="2f07afe6-8fe0-4bc1-a6c5-02d62ae41017" containerName="istio-proxy" Oct 30 17:29:53.688243 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:53.688098 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f07afe6-8fe0-4bc1-a6c5-02d62ae41017" containerName="istio-proxy" Oct 30 17:29:53.688243 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:53.688109 5052 cpu_manager.go:414] "containerMap: removing container" podUID="2f07afe6-8fe0-4bc1-a6c5-02d62ae41017" containerName="istio-proxy" Oct 30 17:29:53.688243 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:53.688114 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f07afe6-8fe0-4bc1-a6c5-02d62ae41017" containerName="istio-proxy" Oct 30 17:29:53.688243 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:53.688197 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cbf5c3b-8676-46d3-9fc0-9417d3e984ad" containerName="maas-api" Oct 30 17:29:53.688243 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:53.688208 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f07afe6-8fe0-4bc1-a6c5-02d62ae41017" containerName="istio-proxy" Oct 30 17:29:53.688243 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:53.688214 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f07afe6-8fe0-4bc1-a6c5-02d62ae41017" containerName="istio-proxy" Oct 30 17:29:53.692773 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:53.692756 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-35a4d69a-kserve-797db9f4fc-5zw5g" Oct 30 17:29:53.709925 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:53.709892 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-35a4d69a-kserve-self-signed-certs\"" Oct 30 17:29:53.712244 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:53.712204 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-35a4d69a-kserve-797db9f4fc-5zw5g"] Oct 30 17:29:53.773088 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:53.773060 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7a90c84-d82e-4cc2-be0c-34944caf6ae4-model-cache\") pod \"e2e-same-id-a-35a4d69a-kserve-797db9f4fc-5zw5g\" (UID: \"f7a90c84-d82e-4cc2-be0c-34944caf6ae4\") " pod="llm/e2e-same-id-a-35a4d69a-kserve-797db9f4fc-5zw5g" Oct 30 17:29:53.773286 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:53.773100 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7a90c84-d82e-4cc2-be0c-34944caf6ae4-tls-certs\") pod \"e2e-same-id-a-35a4d69a-kserve-797db9f4fc-5zw5g\" (UID: \"f7a90c84-d82e-4cc2-be0c-34944caf6ae4\") " pod="llm/e2e-same-id-a-35a4d69a-kserve-797db9f4fc-5zw5g" Oct 30 17:29:53.773286 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:53.773183 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7a90c84-d82e-4cc2-be0c-34944caf6ae4-tmp-dir\") pod \"e2e-same-id-a-35a4d69a-kserve-797db9f4fc-5zw5g\" (UID: \"f7a90c84-d82e-4cc2-be0c-34944caf6ae4\") " pod="llm/e2e-same-id-a-35a4d69a-kserve-797db9f4fc-5zw5g" Oct 30 17:29:53.773286 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:53.773236 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7a90c84-d82e-4cc2-be0c-34944caf6ae4-dshm\") pod \"e2e-same-id-a-35a4d69a-kserve-797db9f4fc-5zw5g\" (UID: \"f7a90c84-d82e-4cc2-be0c-34944caf6ae4\") " pod="llm/e2e-same-id-a-35a4d69a-kserve-797db9f4fc-5zw5g" Oct 30 17:29:53.773286 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:53.773268 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6mlj\" (UniqueName: \"kubernetes.io/projected/f7a90c84-d82e-4cc2-be0c-34944caf6ae4-kube-api-access-p6mlj\") pod \"e2e-same-id-a-35a4d69a-kserve-797db9f4fc-5zw5g\" (UID: \"f7a90c84-d82e-4cc2-be0c-34944caf6ae4\") " pod="llm/e2e-same-id-a-35a4d69a-kserve-797db9f4fc-5zw5g" Oct 30 17:29:53.773469 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:53.773310 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7a90c84-d82e-4cc2-be0c-34944caf6ae4-home\") pod \"e2e-same-id-a-35a4d69a-kserve-797db9f4fc-5zw5g\" (UID: \"f7a90c84-d82e-4cc2-be0c-34944caf6ae4\") " pod="llm/e2e-same-id-a-35a4d69a-kserve-797db9f4fc-5zw5g" Oct 30 17:29:53.875689 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:53.875568 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7a90c84-d82e-4cc2-be0c-34944caf6ae4-tmp-dir\") pod \"e2e-same-id-a-35a4d69a-kserve-797db9f4fc-5zw5g\" (UID: \"f7a90c84-d82e-4cc2-be0c-34944caf6ae4\") " pod="llm/e2e-same-id-a-35a4d69a-kserve-797db9f4fc-5zw5g" Oct 30 17:29:53.875952 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:53.875931 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7a90c84-d82e-4cc2-be0c-34944caf6ae4-dshm\") pod \"e2e-same-id-a-35a4d69a-kserve-797db9f4fc-5zw5g\" (UID: \"f7a90c84-d82e-4cc2-be0c-34944caf6ae4\") " pod="llm/e2e-same-id-a-35a4d69a-kserve-797db9f4fc-5zw5g" Oct 30 17:29:53.876063 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:53.876033 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7a90c84-d82e-4cc2-be0c-34944caf6ae4-tmp-dir\") pod \"e2e-same-id-a-35a4d69a-kserve-797db9f4fc-5zw5g\" (UID: \"f7a90c84-d82e-4cc2-be0c-34944caf6ae4\") " pod="llm/e2e-same-id-a-35a4d69a-kserve-797db9f4fc-5zw5g" Oct 30 17:29:53.876217 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:53.876092 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6mlj\" (UniqueName: \"kubernetes.io/projected/f7a90c84-d82e-4cc2-be0c-34944caf6ae4-kube-api-access-p6mlj\") pod \"e2e-same-id-a-35a4d69a-kserve-797db9f4fc-5zw5g\" (UID: \"f7a90c84-d82e-4cc2-be0c-34944caf6ae4\") " pod="llm/e2e-same-id-a-35a4d69a-kserve-797db9f4fc-5zw5g" Oct 30 17:29:53.876217 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:53.876159 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7a90c84-d82e-4cc2-be0c-34944caf6ae4-home\") pod \"e2e-same-id-a-35a4d69a-kserve-797db9f4fc-5zw5g\" (UID: \"f7a90c84-d82e-4cc2-be0c-34944caf6ae4\") " pod="llm/e2e-same-id-a-35a4d69a-kserve-797db9f4fc-5zw5g" Oct 30 17:29:53.876350 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:53.876295 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7a90c84-d82e-4cc2-be0c-34944caf6ae4-model-cache\") pod \"e2e-same-id-a-35a4d69a-kserve-797db9f4fc-5zw5g\" (UID: \"f7a90c84-d82e-4cc2-be0c-34944caf6ae4\") " pod="llm/e2e-same-id-a-35a4d69a-kserve-797db9f4fc-5zw5g" Oct 30 17:29:53.876411 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:53.876347 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7a90c84-d82e-4cc2-be0c-34944caf6ae4-tls-certs\") pod \"e2e-same-id-a-35a4d69a-kserve-797db9f4fc-5zw5g\" (UID: \"f7a90c84-d82e-4cc2-be0c-34944caf6ae4\") " pod="llm/e2e-same-id-a-35a4d69a-kserve-797db9f4fc-5zw5g" Oct 30 17:29:53.876808 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:53.876784 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7a90c84-d82e-4cc2-be0c-34944caf6ae4-home\") pod \"e2e-same-id-a-35a4d69a-kserve-797db9f4fc-5zw5g\" (UID: \"f7a90c84-d82e-4cc2-be0c-34944caf6ae4\") " pod="llm/e2e-same-id-a-35a4d69a-kserve-797db9f4fc-5zw5g" Oct 30 17:29:53.877032 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:53.876816 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7a90c84-d82e-4cc2-be0c-34944caf6ae4-model-cache\") pod \"e2e-same-id-a-35a4d69a-kserve-797db9f4fc-5zw5g\" (UID: \"f7a90c84-d82e-4cc2-be0c-34944caf6ae4\") " pod="llm/e2e-same-id-a-35a4d69a-kserve-797db9f4fc-5zw5g" Oct 30 17:29:53.879214 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:53.879190 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7a90c84-d82e-4cc2-be0c-34944caf6ae4-dshm\") pod \"e2e-same-id-a-35a4d69a-kserve-797db9f4fc-5zw5g\" (UID: \"f7a90c84-d82e-4cc2-be0c-34944caf6ae4\") " pod="llm/e2e-same-id-a-35a4d69a-kserve-797db9f4fc-5zw5g" Oct 30 17:29:53.879691 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:53.879669 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7a90c84-d82e-4cc2-be0c-34944caf6ae4-tls-certs\") pod \"e2e-same-id-a-35a4d69a-kserve-797db9f4fc-5zw5g\" (UID: \"f7a90c84-d82e-4cc2-be0c-34944caf6ae4\") " pod="llm/e2e-same-id-a-35a4d69a-kserve-797db9f4fc-5zw5g" Oct 30 17:29:53.899080 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:53.899049 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6mlj\" (UniqueName: \"kubernetes.io/projected/f7a90c84-d82e-4cc2-be0c-34944caf6ae4-kube-api-access-p6mlj\") pod \"e2e-same-id-a-35a4d69a-kserve-797db9f4fc-5zw5g\" (UID: \"f7a90c84-d82e-4cc2-be0c-34944caf6ae4\") " pod="llm/e2e-same-id-a-35a4d69a-kserve-797db9f4fc-5zw5g" Oct 30 17:29:54.008168 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:54.008132 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-35a4d69a-kserve-797db9f4fc-5zw5g" Oct 30 17:29:54.017687 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:54.017656 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e2e8d2-68b65wztdl" podUID="b6f6c200-4666-456f-913b-e98f3fa15fa1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:29:54.310740 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:54.310704 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-35a4d69a-kserve-797db9f4fc-5zw5g"] Oct 30 17:29:54.314524 ip-10-0-131-136 kubenswrapper[5052]: W1030 17:29:54.314494 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf7a90c84_d82e_4cc2_be0c_34944caf6ae4.slice/crio-15a01c7dd250f75a676b4c02ad0af9a18188f6af4d026189dee69c03ea594cb0 WatchSource:0}: Error finding container 15a01c7dd250f75a676b4c02ad0af9a18188f6af4d026189dee69c03ea594cb0: Status 404 returned error can't find the container with id 15a01c7dd250f75a676b4c02ad0af9a18188f6af4d026189dee69c03ea594cb0 Oct 30 17:29:54.534394 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:54.534350 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-35a4d69a-kserve-797db9f4fc-5zw5g" event={"ID":"f7a90c84-d82e-4cc2-be0c-34944caf6ae4","Type":"ContainerStarted","Data":"82bbdfe5b14a6cdc3860c878eecade2a11f0e46a03437ad2d839f4d3c3523038"} Oct 30 17:29:54.534569 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:54.534402 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-35a4d69a-kserve-797db9f4fc-5zw5g" event={"ID":"f7a90c84-d82e-4cc2-be0c-34944caf6ae4","Type":"ContainerStarted","Data":"15a01c7dd250f75a676b4c02ad0af9a18188f6af4d026189dee69c03ea594cb0"} Oct 30 17:29:54.575857 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:54.575770 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-35a4d69a-kserve-797db9f4fc-5zw5g" podStartSLOduration=1.575757035 podStartE2EDuration="1.575757035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:29:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:29:54.575164263 +0000 UTC m=+4150.102499482" watchObservedRunningTime="2026-10-30 17:29:54.575757035 +0000 UTC m=+4150.103092254" Oct 30 17:29:54.598845 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:54.598813 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f07afe6-8fe0-4bc1-a6c5-02d62ae41017" path="/var/lib/kubelet/pods/2f07afe6-8fe0-4bc1-a6c5-02d62ae41017/volumes" Oct 30 17:29:55.008977 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:55.008936 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-35a4d69a-kserve-797db9f4fc-5zw5g" Oct 30 17:29:55.031164 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:55.031122 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e2e8d2-6fbfc78d9nhqt2" podUID="313490c0-56de-49f8-918c-6a9169c79f3f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:29:55.888899 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:29:55.888855 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cdwcj"] Oct 30 17:30:03.925060 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:30:03.925030 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e2e8d2-6fbfc78d9nhqt2" Oct 30 17:30:03.958627 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:30:03.958605 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e2e8d2-68b65wztdl" Oct 30 17:30:04.008671 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:30:04.008638 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-35a4d69a-kserve-797db9f4fc-5zw5g" Oct 30 17:30:04.022868 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:30:04.022844 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-35a4d69a-kserve-797db9f4fc-5zw5g" Oct 30 17:30:04.054642 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:30:04.054617 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6f6c200-4666-456f-913b-e98f3fa15fa1-kube-api-access-wqf2l\" (UniqueName: \"kubernetes.io/projected/b6f6c200-4666-456f-913b-e98f3fa15fa1-kube-api-access-wqf2l\") pod \"b6f6c200-4666-456f-913b-e98f3fa15fa1\" (UID: \"b6f6c200-4666-456f-913b-e98f3fa15fa1\") " Oct 30 17:30:04.054754 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:30:04.054659 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/313490c0-56de-49f8-918c-6a9169c79f3f-kube-api-access-2stsg\" (UniqueName: \"kubernetes.io/projected/313490c0-56de-49f8-918c-6a9169c79f3f-kube-api-access-2stsg\") pod \"313490c0-56de-49f8-918c-6a9169c79f3f\" (UID: \"313490c0-56de-49f8-918c-6a9169c79f3f\") " Oct 30 17:30:04.054807 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:30:04.054756 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/313490c0-56de-49f8-918c-6a9169c79f3f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/313490c0-56de-49f8-918c-6a9169c79f3f-plugins-config-volume\") pod \"313490c0-56de-49f8-918c-6a9169c79f3f\" (UID: \"313490c0-56de-49f8-918c-6a9169c79f3f\") " Oct 30 17:30:04.054807 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:30:04.054789 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6f6c200-4666-456f-913b-e98f3fa15fa1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6f6c200-4666-456f-913b-e98f3fa15fa1-plugins-config-volume\") pod \"b6f6c200-4666-456f-913b-e98f3fa15fa1\" (UID: \"b6f6c200-4666-456f-913b-e98f3fa15fa1\") " Oct 30 17:30:04.055152 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:30:04.055129 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/313490c0-56de-49f8-918c-6a9169c79f3f-plugins-config-volume" pod "313490c0-56de-49f8-918c-6a9169c79f3f" (UID: "313490c0-56de-49f8-918c-6a9169c79f3f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:30:04.055246 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:30:04.055168 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6f6c200-4666-456f-913b-e98f3fa15fa1-plugins-config-volume" pod "b6f6c200-4666-456f-913b-e98f3fa15fa1" (UID: "b6f6c200-4666-456f-913b-e98f3fa15fa1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:30:04.056838 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:30:04.056808 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6f6c200-4666-456f-913b-e98f3fa15fa1-kube-api-access-wqf2l" pod "b6f6c200-4666-456f-913b-e98f3fa15fa1" (UID: "b6f6c200-4666-456f-913b-e98f3fa15fa1"). InnerVolumeSpecName "kube-api-access-wqf2l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:30:04.056931 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:30:04.056844 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/313490c0-56de-49f8-918c-6a9169c79f3f-kube-api-access-2stsg" pod "313490c0-56de-49f8-918c-6a9169c79f3f" (UID: "313490c0-56de-49f8-918c-6a9169c79f3f"). InnerVolumeSpecName "kube-api-access-2stsg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:30:04.155883 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:30:04.155804 5052 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6f6c200-4666-456f-913b-e98f3fa15fa1-plugins-config-volume\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:30:04.155883 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:30:04.155827 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqf2l\" (UniqueName: \"kubernetes.io/projected/b6f6c200-4666-456f-913b-e98f3fa15fa1-kube-api-access-wqf2l\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:30:04.155883 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:30:04.155837 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2stsg\" (UniqueName: \"kubernetes.io/projected/313490c0-56de-49f8-918c-6a9169c79f3f-kube-api-access-2stsg\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:30:04.155883 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:30:04.155845 5052 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/313490c0-56de-49f8-918c-6a9169c79f3f-plugins-config-volume\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:30:04.688241 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:30:04.688205 5052 generic.go:356] "Generic (PLEG): container finished" podID="b6f6c200-4666-456f-913b-e98f3fa15fa1" containerID="af7ae2e3b1103e0adf175efc80c25a1adfda72686cfd7257aa1210db05a62639" exitCode=0 Oct 30 17:30:04.688438 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:30:04.688253 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e2e8d2-68b65wztdl" Oct 30 17:30:04.688438 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:30:04.688296 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e2e8d2-68b65wztdl" event={"ID":"b6f6c200-4666-456f-913b-e98f3fa15fa1","Type":"ContainerDied","Data":"af7ae2e3b1103e0adf175efc80c25a1adfda72686cfd7257aa1210db05a62639"} Oct 30 17:30:04.688438 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:30:04.688355 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e2e8d2-68b65wztdl" event={"ID":"b6f6c200-4666-456f-913b-e98f3fa15fa1","Type":"ContainerDied","Data":"5e110aa56eee72a3cb0239ca0d4f154fb7aeb38abc6754da8ef2051a7f001a58"} Oct 30 17:30:04.688438 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:30:04.688382 5052 scope.go:122] "RemoveContainer" containerID="af7ae2e3b1103e0adf175efc80c25a1adfda72686cfd7257aa1210db05a62639" Oct 30 17:30:04.689802 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:30:04.689781 5052 generic.go:356] "Generic (PLEG): container finished" podID="313490c0-56de-49f8-918c-6a9169c79f3f" containerID="fbe3c1bf44f2a6a581772007cabe913eb9f07c77ee6dbf876af098ef73fc498e" exitCode=0 Oct 30 17:30:04.689929 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:30:04.689833 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e2e8d2-6fbfc78d9nhqt2" Oct 30 17:30:04.689929 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:30:04.689870 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e2e8d2-6fbfc78d9nhqt2" event={"ID":"313490c0-56de-49f8-918c-6a9169c79f3f","Type":"ContainerDied","Data":"fbe3c1bf44f2a6a581772007cabe913eb9f07c77ee6dbf876af098ef73fc498e"} Oct 30 17:30:04.690017 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:30:04.689929 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e2e8d2-6fbfc78d9nhqt2" event={"ID":"313490c0-56de-49f8-918c-6a9169c79f3f","Type":"ContainerDied","Data":"74872a679cc6e8745e8eb95599281023aebd1cd544d36478ba2a5dce0f609cb2"} Oct 30 17:30:04.697801 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:30:04.697779 5052 scope.go:122] "RemoveContainer" containerID="af7ae2e3b1103e0adf175efc80c25a1adfda72686cfd7257aa1210db05a62639" Oct 30 17:30:04.698088 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:30:04.698070 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af7ae2e3b1103e0adf175efc80c25a1adfda72686cfd7257aa1210db05a62639\": container with ID starting with af7ae2e3b1103e0adf175efc80c25a1adfda72686cfd7257aa1210db05a62639 not found: ID does not exist" containerID="af7ae2e3b1103e0adf175efc80c25a1adfda72686cfd7257aa1210db05a62639" Oct 30 17:30:04.698135 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:30:04.698098 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af7ae2e3b1103e0adf175efc80c25a1adfda72686cfd7257aa1210db05a62639"} err="failed to get container status \"af7ae2e3b1103e0adf175efc80c25a1adfda72686cfd7257aa1210db05a62639\": rpc error: code = NotFound desc = could not find container \"af7ae2e3b1103e0adf175efc80c25a1adfda72686cfd7257aa1210db05a62639\": container with ID starting with af7ae2e3b1103e0adf175efc80c25a1adfda72686cfd7257aa1210db05a62639 not found: ID does not exist" Oct 30 17:30:04.698135 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:30:04.698113 5052 scope.go:122] "RemoveContainer" containerID="fbe3c1bf44f2a6a581772007cabe913eb9f07c77ee6dbf876af098ef73fc498e" Oct 30 17:30:04.702562 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:30:04.702545 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-35a4d69a-kserve-797db9f4fc-5zw5g" Oct 30 17:30:04.706377 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:30:04.706362 5052 scope.go:122] "RemoveContainer" containerID="fbe3c1bf44f2a6a581772007cabe913eb9f07c77ee6dbf876af098ef73fc498e" Oct 30 17:30:04.706650 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:30:04.706633 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbe3c1bf44f2a6a581772007cabe913eb9f07c77ee6dbf876af098ef73fc498e\": container with ID starting with fbe3c1bf44f2a6a581772007cabe913eb9f07c77ee6dbf876af098ef73fc498e not found: ID does not exist" containerID="fbe3c1bf44f2a6a581772007cabe913eb9f07c77ee6dbf876af098ef73fc498e" Oct 30 17:30:04.706705 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:30:04.706657 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbe3c1bf44f2a6a581772007cabe913eb9f07c77ee6dbf876af098ef73fc498e"} err="failed to get container status \"fbe3c1bf44f2a6a581772007cabe913eb9f07c77ee6dbf876af098ef73fc498e\": rpc error: code = NotFound desc = could not find container \"fbe3c1bf44f2a6a581772007cabe913eb9f07c77ee6dbf876af098ef73fc498e\": container with ID starting with fbe3c1bf44f2a6a581772007cabe913eb9f07c77ee6dbf876af098ef73fc498e not found: ID does not exist" Oct 30 17:30:04.781191 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:30:04.781160 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-e2e8d2-6fbfc78d9nhqt2"] Oct 30 17:30:04.785973 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:30:04.785944 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-e2e8d2-6fbfc78d9nhqt2"] Oct 30 17:30:04.822345 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:30:04.822214 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-e2e8d2-68b65wztdl"] Oct 30 17:30:04.827687 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:30:04.827661 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-e2e8d2-68b65wztdl"] Oct 30 17:30:06.596970 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:30:06.596930 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="313490c0-56de-49f8-918c-6a9169c79f3f" path="/var/lib/kubelet/pods/313490c0-56de-49f8-918c-6a9169c79f3f/volumes" Oct 30 17:30:06.597376 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:30:06.597361 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6f6c200-4666-456f-913b-e98f3fa15fa1" path="/var/lib/kubelet/pods/b6f6c200-4666-456f-913b-e98f3fa15fa1/volumes" Oct 30 17:30:14.039316 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:30:14.039223 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-35a4d69a-kserve-8457c9478f-gknt6"] Oct 30 17:30:14.039997 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:30:14.039974 5052 cpu_manager.go:414] "containerMap: removing container" podUID="b6f6c200-4666-456f-913b-e98f3fa15fa1" containerName="payload-pre-processing" Oct 30 17:30:14.040103 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:30:14.040001 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6f6c200-4666-456f-913b-e98f3fa15fa1" containerName="payload-pre-processing" Oct 30 17:30:14.040103 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:30:14.040018 5052 cpu_manager.go:414] "containerMap: removing container" podUID="313490c0-56de-49f8-918c-6a9169c79f3f" containerName="payload-processing" Oct 30 17:30:14.040103 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:30:14.040028 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="313490c0-56de-49f8-918c-6a9169c79f3f" containerName="payload-processing" Oct 30 17:30:14.040215 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:30:14.040157 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6f6c200-4666-456f-913b-e98f3fa15fa1" containerName="payload-pre-processing" Oct 30 17:30:14.040215 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:30:14.040169 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="313490c0-56de-49f8-918c-6a9169c79f3f" containerName="payload-processing" Oct 30 17:30:14.043883 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:30:14.043860 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-35a4d69a-kserve-8457c9478f-gknt6" Oct 30 17:30:14.071086 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:30:14.071060 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-35a4d69a-kserve-8457c9478f-gknt6"] Oct 30 17:30:14.072618 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:30:14.072593 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-35a4d69a-kserve-self-signed-certs\"" Oct 30 17:30:14.160233 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:30:14.160189 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6f3c3b8-f40b-46ce-948d-252a84c90eb0-home\") pod \"e2e-same-id-b-35a4d69a-kserve-8457c9478f-gknt6\" (UID: \"f6f3c3b8-f40b-46ce-948d-252a84c90eb0\") " pod="llm/e2e-same-id-b-35a4d69a-kserve-8457c9478f-gknt6" Oct 30 17:30:14.160419 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:30:14.160242 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6f3c3b8-f40b-46ce-948d-252a84c90eb0-model-cache\") pod \"e2e-same-id-b-35a4d69a-kserve-8457c9478f-gknt6\" (UID: \"f6f3c3b8-f40b-46ce-948d-252a84c90eb0\") " pod="llm/e2e-same-id-b-35a4d69a-kserve-8457c9478f-gknt6" Oct 30 17:30:14.160419 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:30:14.160287 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6f3c3b8-f40b-46ce-948d-252a84c90eb0-tmp-dir\") pod \"e2e-same-id-b-35a4d69a-kserve-8457c9478f-gknt6\" (UID: \"f6f3c3b8-f40b-46ce-948d-252a84c90eb0\") " pod="llm/e2e-same-id-b-35a4d69a-kserve-8457c9478f-gknt6" Oct 30 17:30:14.160419 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:30:14.160306 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6f3c3b8-f40b-46ce-948d-252a84c90eb0-dshm\") pod \"e2e-same-id-b-35a4d69a-kserve-8457c9478f-gknt6\" (UID: \"f6f3c3b8-f40b-46ce-948d-252a84c90eb0\") " pod="llm/e2e-same-id-b-35a4d69a-kserve-8457c9478f-gknt6" Oct 30 17:30:14.160550 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:30:14.160435 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6f3c3b8-f40b-46ce-948d-252a84c90eb0-tls-certs\") pod \"e2e-same-id-b-35a4d69a-kserve-8457c9478f-gknt6\" (UID: \"f6f3c3b8-f40b-46ce-948d-252a84c90eb0\") " pod="llm/e2e-same-id-b-35a4d69a-kserve-8457c9478f-gknt6" Oct 30 17:30:14.160550 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:30:14.160532 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96nfw\" (UniqueName: \"kubernetes.io/projected/f6f3c3b8-f40b-46ce-948d-252a84c90eb0-kube-api-access-96nfw\") pod \"e2e-same-id-b-35a4d69a-kserve-8457c9478f-gknt6\" (UID: \"f6f3c3b8-f40b-46ce-948d-252a84c90eb0\") " pod="llm/e2e-same-id-b-35a4d69a-kserve-8457c9478f-gknt6" Oct 30 17:30:14.261229 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:30:14.261188 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96nfw\" (UniqueName: \"kubernetes.io/projected/f6f3c3b8-f40b-46ce-948d-252a84c90eb0-kube-api-access-96nfw\") pod \"e2e-same-id-b-35a4d69a-kserve-8457c9478f-gknt6\" (UID: \"f6f3c3b8-f40b-46ce-948d-252a84c90eb0\") " pod="llm/e2e-same-id-b-35a4d69a-kserve-8457c9478f-gknt6" Oct 30 17:30:14.261452 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:30:14.261281 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6f3c3b8-f40b-46ce-948d-252a84c90eb0-home\") pod \"e2e-same-id-b-35a4d69a-kserve-8457c9478f-gknt6\" (UID: \"f6f3c3b8-f40b-46ce-948d-252a84c90eb0\") " pod="llm/e2e-same-id-b-35a4d69a-kserve-8457c9478f-gknt6" Oct 30 17:30:14.261452 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:30:14.261310 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6f3c3b8-f40b-46ce-948d-252a84c90eb0-model-cache\") pod \"e2e-same-id-b-35a4d69a-kserve-8457c9478f-gknt6\" (UID: \"f6f3c3b8-f40b-46ce-948d-252a84c90eb0\") " pod="llm/e2e-same-id-b-35a4d69a-kserve-8457c9478f-gknt6" Oct 30 17:30:14.261452 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:30:14.261343 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6f3c3b8-f40b-46ce-948d-252a84c90eb0-tmp-dir\") pod \"e2e-same-id-b-35a4d69a-kserve-8457c9478f-gknt6\" (UID: \"f6f3c3b8-f40b-46ce-948d-252a84c90eb0\") " pod="llm/e2e-same-id-b-35a4d69a-kserve-8457c9478f-gknt6" Oct 30 17:30:14.261452 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:30:14.261369 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6f3c3b8-f40b-46ce-948d-252a84c90eb0-dshm\") pod \"e2e-same-id-b-35a4d69a-kserve-8457c9478f-gknt6\" (UID: \"f6f3c3b8-f40b-46ce-948d-252a84c90eb0\") " pod="llm/e2e-same-id-b-35a4d69a-kserve-8457c9478f-gknt6" Oct 30 17:30:14.261452 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:30:14.261416 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6f3c3b8-f40b-46ce-948d-252a84c90eb0-tls-certs\") pod \"e2e-same-id-b-35a4d69a-kserve-8457c9478f-gknt6\" (UID: \"f6f3c3b8-f40b-46ce-948d-252a84c90eb0\") " pod="llm/e2e-same-id-b-35a4d69a-kserve-8457c9478f-gknt6" Oct 30 17:30:14.261787 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:30:14.261746 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6f3c3b8-f40b-46ce-948d-252a84c90eb0-home\") pod \"e2e-same-id-b-35a4d69a-kserve-8457c9478f-gknt6\" (UID: \"f6f3c3b8-f40b-46ce-948d-252a84c90eb0\") " pod="llm/e2e-same-id-b-35a4d69a-kserve-8457c9478f-gknt6" Oct 30 17:30:14.261787 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:30:14.261780 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6f3c3b8-f40b-46ce-948d-252a84c90eb0-tmp-dir\") pod \"e2e-same-id-b-35a4d69a-kserve-8457c9478f-gknt6\" (UID: \"f6f3c3b8-f40b-46ce-948d-252a84c90eb0\") " pod="llm/e2e-same-id-b-35a4d69a-kserve-8457c9478f-gknt6" Oct 30 17:30:14.261891 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:30:14.261817 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6f3c3b8-f40b-46ce-948d-252a84c90eb0-model-cache\") pod \"e2e-same-id-b-35a4d69a-kserve-8457c9478f-gknt6\" (UID: \"f6f3c3b8-f40b-46ce-948d-252a84c90eb0\") " pod="llm/e2e-same-id-b-35a4d69a-kserve-8457c9478f-gknt6" Oct 30 17:30:14.263924 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:30:14.263897 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6f3c3b8-f40b-46ce-948d-252a84c90eb0-dshm\") pod \"e2e-same-id-b-35a4d69a-kserve-8457c9478f-gknt6\" (UID: \"f6f3c3b8-f40b-46ce-948d-252a84c90eb0\") " pod="llm/e2e-same-id-b-35a4d69a-kserve-8457c9478f-gknt6" Oct 30 17:30:14.264086 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:30:14.264063 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6f3c3b8-f40b-46ce-948d-252a84c90eb0-tls-certs\") pod \"e2e-same-id-b-35a4d69a-kserve-8457c9478f-gknt6\" (UID: \"f6f3c3b8-f40b-46ce-948d-252a84c90eb0\") " pod="llm/e2e-same-id-b-35a4d69a-kserve-8457c9478f-gknt6" Oct 30 17:30:14.284787 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:30:14.284758 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96nfw\" (UniqueName: \"kubernetes.io/projected/f6f3c3b8-f40b-46ce-948d-252a84c90eb0-kube-api-access-96nfw\") pod \"e2e-same-id-b-35a4d69a-kserve-8457c9478f-gknt6\" (UID: \"f6f3c3b8-f40b-46ce-948d-252a84c90eb0\") " pod="llm/e2e-same-id-b-35a4d69a-kserve-8457c9478f-gknt6" Oct 30 17:30:14.359174 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:30:14.359101 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-35a4d69a-kserve-8457c9478f-gknt6" Oct 30 17:30:14.668883 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:30:14.668841 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-35a4d69a-kserve-8457c9478f-gknt6"] Oct 30 17:30:14.672489 ip-10-0-131-136 kubenswrapper[5052]: W1030 17:30:14.672448 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf6f3c3b8_f40b_46ce_948d_252a84c90eb0.slice/crio-05d858092a2172d4cdc8d77fe8b0cefccaca14c58a14f36e416d55f15d6ee89e WatchSource:0}: Error finding container 05d858092a2172d4cdc8d77fe8b0cefccaca14c58a14f36e416d55f15d6ee89e: Status 404 returned error can't find the container with id 05d858092a2172d4cdc8d77fe8b0cefccaca14c58a14f36e416d55f15d6ee89e Oct 30 17:30:14.747019 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:30:14.746983 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-35a4d69a-kserve-8457c9478f-gknt6" event={"ID":"f6f3c3b8-f40b-46ce-948d-252a84c90eb0","Type":"ContainerStarted","Data":"836cdaced307026d75229f98a7c6802cb599cf1c59cfeccba0df6b4df34e294f"} Oct 30 17:30:14.747190 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:30:14.747032 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-35a4d69a-kserve-8457c9478f-gknt6" event={"ID":"f6f3c3b8-f40b-46ce-948d-252a84c90eb0","Type":"ContainerStarted","Data":"05d858092a2172d4cdc8d77fe8b0cefccaca14c58a14f36e416d55f15d6ee89e"} Oct 30 17:30:14.790432 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:30:14.790154 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-35a4d69a-kserve-8457c9478f-gknt6" podStartSLOduration=0.790132085 podStartE2EDuration="790.132085ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:30:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:30:14.787905083 +0000 UTC m=+4170.315240294" watchObservedRunningTime="2026-10-30 17:30:14.790132085 +0000 UTC m=+4170.317467303" Oct 30 17:30:15.359666 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:30:15.359633 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-35a4d69a-kserve-8457c9478f-gknt6" Oct 30 17:30:16.312842 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:30:16.312807 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cdwcj"] Oct 30 17:30:19.126829 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:30:19.126780 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 17:30:19.194089 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:30:19.193693 5052 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-flvdc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:30:19.194089 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:30:19.193777 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-flvdc" podUID="42edda16-6400-45ef-bce3-a5ce362045ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:30:19.627616 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:30:19.624802 5052 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-9kbt9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.236:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:30:19.627616 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:30:19.624887 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9kbt9" podUID="384a9c57-053d-477e-badb-355261588881" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.236:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:30:19.669004 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:30:19.668679 5052 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-kpphn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:30:19.669004 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:30:19.668758 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-kpphn" podUID="c4c977d2-7480-4d5c-87d6-743c519e2efc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:30:24.359755 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:30:24.359716 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-35a4d69a-kserve-8457c9478f-gknt6" Oct 30 17:30:24.372270 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:30:24.372242 5052 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-35a4d69a-kserve-8457c9478f-gknt6" Oct 30 17:30:24.906735 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:30:24.906705 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-35a4d69a-kserve-8457c9478f-gknt6" Oct 30 17:30:31.416228 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:30:31.416185 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cdwcj"] Oct 30 17:30:34.203744 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:30:34.203697 5052 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-flvdc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:30:34.204403 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:30:34.204362 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-flvdc" podUID="42edda16-6400-45ef-bce3-a5ce362045ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:30:34.649812 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:30:34.649780 5052 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-9kbt9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.236:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:30:34.650127 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:30:34.650078 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9kbt9" podUID="384a9c57-053d-477e-badb-355261588881" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.236:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:30:34.666821 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:30:34.666702 5052 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-kpphn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:30:34.666821 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:30:34.666761 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-kpphn" podUID="c4c977d2-7480-4d5c-87d6-743c519e2efc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:30:46.000641 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:30:46.000606 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4wrf8_4f0f46f4-5cda-4d67-b6f3-47df06eb844c/console-operator/2.log" Oct 30 17:30:46.009828 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:30:46.009803 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4wrf8_4f0f46f4-5cda-4d67-b6f3-47df06eb844c/console-operator/2.log" Oct 30 17:30:47.238808 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:30:47.238768 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-35a4d69a-kserve-797db9f4fc-5zw5g"] Oct 30 17:30:48.843739 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:30:48.843679 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 17:30:49.195865 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:30:49.195819 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cdwcj"] Oct 30 17:30:50.813200 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:30:50.793269 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-35a4d69a-kserve-8457c9478f-gknt6"] Oct 30 17:30:58.069010 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:30:58.068970 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cdwcj"] Oct 30 17:31:02.260469 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:02.260408 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-35a4d69a-kserve-797db9f4fc-5zw5g" podUID="f7a90c84-d82e-4cc2-be0c-34944caf6ae4" containerName="main" containerID="cri-o://82bbdfe5b14a6cdc3860c878eecade2a11f0e46a03437ad2d839f4d3c3523038" gracePeriod=45 Oct 30 17:31:02.526483 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:02.526427 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-35a4d69a-kserve-797db9f4fc-5zw5g" Oct 30 17:31:02.654109 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:02.654078 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7a90c84-d82e-4cc2-be0c-34944caf6ae4-home\" (UniqueName: \"kubernetes.io/empty-dir/f7a90c84-d82e-4cc2-be0c-34944caf6ae4-home\") pod \"f7a90c84-d82e-4cc2-be0c-34944caf6ae4\" (UID: \"f7a90c84-d82e-4cc2-be0c-34944caf6ae4\") " Oct 30 17:31:02.654301 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:02.654129 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7a90c84-d82e-4cc2-be0c-34944caf6ae4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7a90c84-d82e-4cc2-be0c-34944caf6ae4-tmp-dir\") pod \"f7a90c84-d82e-4cc2-be0c-34944caf6ae4\" (UID: \"f7a90c84-d82e-4cc2-be0c-34944caf6ae4\") " Oct 30 17:31:02.654301 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:02.654184 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7a90c84-d82e-4cc2-be0c-34944caf6ae4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7a90c84-d82e-4cc2-be0c-34944caf6ae4-dshm\") pod \"f7a90c84-d82e-4cc2-be0c-34944caf6ae4\" (UID: \"f7a90c84-d82e-4cc2-be0c-34944caf6ae4\") " Oct 30 17:31:02.654301 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:02.654216 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7a90c84-d82e-4cc2-be0c-34944caf6ae4-tls-certs\" (UniqueName: \"kubernetes.io/secret/f7a90c84-d82e-4cc2-be0c-34944caf6ae4-tls-certs\") pod \"f7a90c84-d82e-4cc2-be0c-34944caf6ae4\" (UID: \"f7a90c84-d82e-4cc2-be0c-34944caf6ae4\") " Oct 30 17:31:02.654494 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:02.654314 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7a90c84-d82e-4cc2-be0c-34944caf6ae4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7a90c84-d82e-4cc2-be0c-34944caf6ae4-model-cache\") pod \"f7a90c84-d82e-4cc2-be0c-34944caf6ae4\" (UID: \"f7a90c84-d82e-4cc2-be0c-34944caf6ae4\") " Oct 30 17:31:02.654494 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:02.654336 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7a90c84-d82e-4cc2-be0c-34944caf6ae4-home" pod "f7a90c84-d82e-4cc2-be0c-34944caf6ae4" (UID: "f7a90c84-d82e-4cc2-be0c-34944caf6ae4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:31:02.654494 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:02.654377 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7a90c84-d82e-4cc2-be0c-34944caf6ae4-kube-api-access-p6mlj\" (UniqueName: \"kubernetes.io/projected/f7a90c84-d82e-4cc2-be0c-34944caf6ae4-kube-api-access-p6mlj\") pod \"f7a90c84-d82e-4cc2-be0c-34944caf6ae4\" (UID: \"f7a90c84-d82e-4cc2-be0c-34944caf6ae4\") " Oct 30 17:31:02.654494 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:02.654458 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7a90c84-d82e-4cc2-be0c-34944caf6ae4-tmp-dir" pod "f7a90c84-d82e-4cc2-be0c-34944caf6ae4" (UID: "f7a90c84-d82e-4cc2-be0c-34944caf6ae4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:31:02.654763 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:02.654633 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7a90c84-d82e-4cc2-be0c-34944caf6ae4-model-cache" pod "f7a90c84-d82e-4cc2-be0c-34944caf6ae4" (UID: "f7a90c84-d82e-4cc2-be0c-34944caf6ae4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:31:02.654763 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:02.654735 5052 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7a90c84-d82e-4cc2-be0c-34944caf6ae4-model-cache\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:31:02.654763 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:02.654749 5052 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7a90c84-d82e-4cc2-be0c-34944caf6ae4-home\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:31:02.654935 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:02.654776 5052 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7a90c84-d82e-4cc2-be0c-34944caf6ae4-tmp-dir\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:31:02.656634 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:02.656605 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7a90c84-d82e-4cc2-be0c-34944caf6ae4-dshm" pod "f7a90c84-d82e-4cc2-be0c-34944caf6ae4" (UID: "f7a90c84-d82e-4cc2-be0c-34944caf6ae4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:31:02.656740 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:02.656677 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7a90c84-d82e-4cc2-be0c-34944caf6ae4-tls-certs" pod "f7a90c84-d82e-4cc2-be0c-34944caf6ae4" (UID: "f7a90c84-d82e-4cc2-be0c-34944caf6ae4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:31:02.656740 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:02.656675 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7a90c84-d82e-4cc2-be0c-34944caf6ae4-kube-api-access-p6mlj" pod "f7a90c84-d82e-4cc2-be0c-34944caf6ae4" (UID: "f7a90c84-d82e-4cc2-be0c-34944caf6ae4"). InnerVolumeSpecName "kube-api-access-p6mlj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:31:02.756144 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:02.756117 5052 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7a90c84-d82e-4cc2-be0c-34944caf6ae4-dshm\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:31:02.756266 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:02.756146 5052 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7a90c84-d82e-4cc2-be0c-34944caf6ae4-tls-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:31:02.756266 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:02.756162 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6mlj\" (UniqueName: \"kubernetes.io/projected/f7a90c84-d82e-4cc2-be0c-34944caf6ae4-kube-api-access-p6mlj\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:31:03.236637 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:03.236601 5052 generic.go:356] "Generic (PLEG): container finished" podID="f7a90c84-d82e-4cc2-be0c-34944caf6ae4" containerID="82bbdfe5b14a6cdc3860c878eecade2a11f0e46a03437ad2d839f4d3c3523038" exitCode=0 Oct 30 17:31:03.236828 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:03.236647 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-35a4d69a-kserve-797db9f4fc-5zw5g" Oct 30 17:31:03.236828 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:03.236680 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-35a4d69a-kserve-797db9f4fc-5zw5g" event={"ID":"f7a90c84-d82e-4cc2-be0c-34944caf6ae4","Type":"ContainerDied","Data":"82bbdfe5b14a6cdc3860c878eecade2a11f0e46a03437ad2d839f4d3c3523038"} Oct 30 17:31:03.236828 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:03.236725 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-35a4d69a-kserve-797db9f4fc-5zw5g" event={"ID":"f7a90c84-d82e-4cc2-be0c-34944caf6ae4","Type":"ContainerDied","Data":"15a01c7dd250f75a676b4c02ad0af9a18188f6af4d026189dee69c03ea594cb0"} Oct 30 17:31:03.236828 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:03.236744 5052 scope.go:122] "RemoveContainer" containerID="82bbdfe5b14a6cdc3860c878eecade2a11f0e46a03437ad2d839f4d3c3523038" Oct 30 17:31:03.246285 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:03.246210 5052 scope.go:122] "RemoveContainer" containerID="82bbdfe5b14a6cdc3860c878eecade2a11f0e46a03437ad2d839f4d3c3523038" Oct 30 17:31:03.246728 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:31:03.246704 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82bbdfe5b14a6cdc3860c878eecade2a11f0e46a03437ad2d839f4d3c3523038\": container with ID starting with 82bbdfe5b14a6cdc3860c878eecade2a11f0e46a03437ad2d839f4d3c3523038 not found: ID does not exist" containerID="82bbdfe5b14a6cdc3860c878eecade2a11f0e46a03437ad2d839f4d3c3523038" Oct 30 17:31:03.246856 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:03.246739 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82bbdfe5b14a6cdc3860c878eecade2a11f0e46a03437ad2d839f4d3c3523038"} err="failed to get container status \"82bbdfe5b14a6cdc3860c878eecade2a11f0e46a03437ad2d839f4d3c3523038\": rpc error: code = NotFound desc = could not find container \"82bbdfe5b14a6cdc3860c878eecade2a11f0e46a03437ad2d839f4d3c3523038\": container with ID starting with 82bbdfe5b14a6cdc3860c878eecade2a11f0e46a03437ad2d839f4d3c3523038 not found: ID does not exist" Oct 30 17:31:03.286661 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:03.286631 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-35a4d69a-kserve-797db9f4fc-5zw5g"] Oct 30 17:31:03.298306 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:03.298284 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-35a4d69a-kserve-797db9f4fc-5zw5g"] Oct 30 17:31:04.596752 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:04.596718 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7a90c84-d82e-4cc2-be0c-34944caf6ae4" path="/var/lib/kubelet/pods/f7a90c84-d82e-4cc2-be0c-34944caf6ae4/volumes" Oct 30 17:31:05.716367 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:05.716323 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cdwcj"] Oct 30 17:31:06.129444 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:06.129364 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-35a4d69a-kserve-8457c9478f-gknt6" podUID="f6f3c3b8-f40b-46ce-948d-252a84c90eb0" containerName="main" containerID="cri-o://836cdaced307026d75229f98a7c6802cb599cf1c59cfeccba0df6b4df34e294f" gracePeriod=45 Oct 30 17:31:07.056604 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:07.055764 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-35a4d69a-kserve-8457c9478f-gknt6" Oct 30 17:31:07.117138 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:07.117103 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6f3c3b8-f40b-46ce-948d-252a84c90eb0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6f3c3b8-f40b-46ce-948d-252a84c90eb0-tmp-dir\") pod \"f6f3c3b8-f40b-46ce-948d-252a84c90eb0\" (UID: \"f6f3c3b8-f40b-46ce-948d-252a84c90eb0\") " Oct 30 17:31:07.118220 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:07.118193 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6f3c3b8-f40b-46ce-948d-252a84c90eb0-tls-certs\" (UniqueName: \"kubernetes.io/secret/f6f3c3b8-f40b-46ce-948d-252a84c90eb0-tls-certs\") pod \"f6f3c3b8-f40b-46ce-948d-252a84c90eb0\" (UID: \"f6f3c3b8-f40b-46ce-948d-252a84c90eb0\") " Oct 30 17:31:07.118452 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:07.118433 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6f3c3b8-f40b-46ce-948d-252a84c90eb0-kube-api-access-96nfw\" (UniqueName: \"kubernetes.io/projected/f6f3c3b8-f40b-46ce-948d-252a84c90eb0-kube-api-access-96nfw\") pod \"f6f3c3b8-f40b-46ce-948d-252a84c90eb0\" (UID: \"f6f3c3b8-f40b-46ce-948d-252a84c90eb0\") " Oct 30 17:31:07.118682 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:07.118651 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6f3c3b8-f40b-46ce-948d-252a84c90eb0-home\" (UniqueName: \"kubernetes.io/empty-dir/f6f3c3b8-f40b-46ce-948d-252a84c90eb0-home\") pod \"f6f3c3b8-f40b-46ce-948d-252a84c90eb0\" (UID: \"f6f3c3b8-f40b-46ce-948d-252a84c90eb0\") " Oct 30 17:31:07.141600 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:07.118836 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6f3c3b8-f40b-46ce-948d-252a84c90eb0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6f3c3b8-f40b-46ce-948d-252a84c90eb0-dshm\") pod \"f6f3c3b8-f40b-46ce-948d-252a84c90eb0\" (UID: \"f6f3c3b8-f40b-46ce-948d-252a84c90eb0\") " Oct 30 17:31:07.142299 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:07.142275 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6f3c3b8-f40b-46ce-948d-252a84c90eb0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6f3c3b8-f40b-46ce-948d-252a84c90eb0-model-cache\") pod \"f6f3c3b8-f40b-46ce-948d-252a84c90eb0\" (UID: \"f6f3c3b8-f40b-46ce-948d-252a84c90eb0\") " Oct 30 17:31:07.150759 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:07.123039 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6f3c3b8-f40b-46ce-948d-252a84c90eb0-tmp-dir" pod "f6f3c3b8-f40b-46ce-948d-252a84c90eb0" (UID: "f6f3c3b8-f40b-46ce-948d-252a84c90eb0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:31:07.150759 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:07.123533 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6f3c3b8-f40b-46ce-948d-252a84c90eb0-home" pod "f6f3c3b8-f40b-46ce-948d-252a84c90eb0" (UID: "f6f3c3b8-f40b-46ce-948d-252a84c90eb0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:31:07.170284 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:07.170251 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6f3c3b8-f40b-46ce-948d-252a84c90eb0-model-cache" pod "f6f3c3b8-f40b-46ce-948d-252a84c90eb0" (UID: "f6f3c3b8-f40b-46ce-948d-252a84c90eb0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:31:07.184613 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:07.179728 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6f3c3b8-f40b-46ce-948d-252a84c90eb0-kube-api-access-96nfw" pod "f6f3c3b8-f40b-46ce-948d-252a84c90eb0" (UID: "f6f3c3b8-f40b-46ce-948d-252a84c90eb0"). InnerVolumeSpecName "kube-api-access-96nfw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:31:07.251452 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:07.251297 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96nfw\" (UniqueName: \"kubernetes.io/projected/f6f3c3b8-f40b-46ce-948d-252a84c90eb0-kube-api-access-96nfw\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:31:07.251452 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:07.251340 5052 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6f3c3b8-f40b-46ce-948d-252a84c90eb0-home\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:31:07.251452 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:07.251356 5052 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6f3c3b8-f40b-46ce-948d-252a84c90eb0-model-cache\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:31:07.251452 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:07.251370 5052 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6f3c3b8-f40b-46ce-948d-252a84c90eb0-tmp-dir\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:31:07.271084 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:07.271006 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6f3c3b8-f40b-46ce-948d-252a84c90eb0-tls-certs" pod "f6f3c3b8-f40b-46ce-948d-252a84c90eb0" (UID: "f6f3c3b8-f40b-46ce-948d-252a84c90eb0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:31:07.362644 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:07.354523 5052 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6f3c3b8-f40b-46ce-948d-252a84c90eb0-tls-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:31:07.416604 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:07.407974 5052 generic.go:356] "Generic (PLEG): container finished" podID="f6f3c3b8-f40b-46ce-948d-252a84c90eb0" containerID="836cdaced307026d75229f98a7c6802cb599cf1c59cfeccba0df6b4df34e294f" exitCode=0 Oct 30 17:31:07.416604 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:07.413662 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-35a4d69a-kserve-8457c9478f-gknt6" event={"ID":"f6f3c3b8-f40b-46ce-948d-252a84c90eb0","Type":"ContainerDied","Data":"836cdaced307026d75229f98a7c6802cb599cf1c59cfeccba0df6b4df34e294f"} Oct 30 17:31:07.416604 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:07.414065 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-35a4d69a-kserve-8457c9478f-gknt6" event={"ID":"f6f3c3b8-f40b-46ce-948d-252a84c90eb0","Type":"ContainerDied","Data":"05d858092a2172d4cdc8d77fe8b0cefccaca14c58a14f36e416d55f15d6ee89e"} Oct 30 17:31:07.416604 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:07.414095 5052 scope.go:122] "RemoveContainer" containerID="836cdaced307026d75229f98a7c6802cb599cf1c59cfeccba0df6b4df34e294f" Oct 30 17:31:07.416604 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:07.413749 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-35a4d69a-kserve-8457c9478f-gknt6" Oct 30 17:31:07.450735 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:07.450687 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6f3c3b8-f40b-46ce-948d-252a84c90eb0-dshm" pod "f6f3c3b8-f40b-46ce-948d-252a84c90eb0" (UID: "f6f3c3b8-f40b-46ce-948d-252a84c90eb0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:31:07.492615 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:07.492350 5052 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6f3c3b8-f40b-46ce-948d-252a84c90eb0-dshm\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:31:07.524897 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:07.524338 5052 scope.go:122] "RemoveContainer" containerID="836cdaced307026d75229f98a7c6802cb599cf1c59cfeccba0df6b4df34e294f" Oct 30 17:31:07.524897 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:31:07.524786 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"836cdaced307026d75229f98a7c6802cb599cf1c59cfeccba0df6b4df34e294f\": container with ID starting with 836cdaced307026d75229f98a7c6802cb599cf1c59cfeccba0df6b4df34e294f not found: ID does not exist" containerID="836cdaced307026d75229f98a7c6802cb599cf1c59cfeccba0df6b4df34e294f" Oct 30 17:31:07.524897 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:07.524820 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"836cdaced307026d75229f98a7c6802cb599cf1c59cfeccba0df6b4df34e294f"} err="failed to get container status \"836cdaced307026d75229f98a7c6802cb599cf1c59cfeccba0df6b4df34e294f\": rpc error: code = NotFound desc = could not find container \"836cdaced307026d75229f98a7c6802cb599cf1c59cfeccba0df6b4df34e294f\": container with ID starting with 836cdaced307026d75229f98a7c6802cb599cf1c59cfeccba0df6b4df34e294f not found: ID does not exist" Oct 30 17:31:07.919942 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:07.919893 5052 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-same-id-b-35a4d69a-kserve-8457c9478f-gknt6" podUID="f6f3c3b8-f40b-46ce-948d-252a84c90eb0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.249:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:31:08.049607 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:08.027260 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-35a4d69a-kserve-8457c9478f-gknt6"] Oct 30 17:31:08.049607 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:08.027328 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-35a4d69a-kserve-8457c9478f-gknt6"] Oct 30 17:31:08.596764 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:08.596731 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6f3c3b8-f40b-46ce-948d-252a84c90eb0" path="/var/lib/kubelet/pods/f6f3c3b8-f40b-46ce-948d-252a84c90eb0/volumes" Oct 30 17:31:13.098411 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:13.098362 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cdwcj"] Oct 30 17:31:18.839598 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:31:18.839537 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 17:31:30.798140 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:30.798103 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cdwcj"] Oct 30 17:31:32.055545 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:32.055509 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-rmtpq"] Oct 30 17:31:32.055996 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:32.055980 5052 cpu_manager.go:414] "containerMap: removing container" podUID="f7a90c84-d82e-4cc2-be0c-34944caf6ae4" containerName="main" Oct 30 17:31:32.056042 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:32.055997 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7a90c84-d82e-4cc2-be0c-34944caf6ae4" containerName="main" Oct 30 17:31:32.056042 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:32.056025 5052 cpu_manager.go:414] "containerMap: removing container" podUID="f6f3c3b8-f40b-46ce-948d-252a84c90eb0" containerName="main" Oct 30 17:31:32.056042 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:32.056030 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6f3c3b8-f40b-46ce-948d-252a84c90eb0" containerName="main" Oct 30 17:31:32.056148 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:32.056100 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6f3c3b8-f40b-46ce-948d-252a84c90eb0" containerName="main" Oct 30 17:31:32.056148 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:32.056114 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7a90c84-d82e-4cc2-be0c-34944caf6ae4" containerName="main" Oct 30 17:31:32.059091 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:32.059073 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-595cc648c-rmtpq" Oct 30 17:31:32.089842 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:32.089809 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-rmtpq"] Oct 30 17:31:32.135985 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:32.135953 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9421326-17de-4a12-831f-edfd135b94c1-openshift-service-ca\") pod \"maas-api-595cc648c-rmtpq\" (UID: \"e9421326-17de-4a12-831f-edfd135b94c1\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-rmtpq" Oct 30 17:31:32.135985 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:32.135988 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9421326-17de-4a12-831f-edfd135b94c1-metrics-certs\") pod \"maas-api-595cc648c-rmtpq\" (UID: \"e9421326-17de-4a12-831f-edfd135b94c1\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-rmtpq" Oct 30 17:31:32.136197 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:32.136096 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9421326-17de-4a12-831f-edfd135b94c1-maas-api-tls\") pod \"maas-api-595cc648c-rmtpq\" (UID: \"e9421326-17de-4a12-831f-edfd135b94c1\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-rmtpq" Oct 30 17:31:32.136197 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:32.136116 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cm6tw\" (UniqueName: \"kubernetes.io/projected/e9421326-17de-4a12-831f-edfd135b94c1-kube-api-access-cm6tw\") pod \"maas-api-595cc648c-rmtpq\" (UID: \"e9421326-17de-4a12-831f-edfd135b94c1\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-rmtpq" Oct 30 17:31:32.237565 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:32.237531 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9421326-17de-4a12-831f-edfd135b94c1-maas-api-tls\") pod \"maas-api-595cc648c-rmtpq\" (UID: \"e9421326-17de-4a12-831f-edfd135b94c1\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-rmtpq" Oct 30 17:31:32.237565 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:32.237569 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cm6tw\" (UniqueName: \"kubernetes.io/projected/e9421326-17de-4a12-831f-edfd135b94c1-kube-api-access-cm6tw\") pod \"maas-api-595cc648c-rmtpq\" (UID: \"e9421326-17de-4a12-831f-edfd135b94c1\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-rmtpq" Oct 30 17:31:32.237826 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:32.237707 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9421326-17de-4a12-831f-edfd135b94c1-openshift-service-ca\") pod \"maas-api-595cc648c-rmtpq\" (UID: \"e9421326-17de-4a12-831f-edfd135b94c1\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-rmtpq" Oct 30 17:31:32.237826 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:32.237732 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9421326-17de-4a12-831f-edfd135b94c1-metrics-certs\") pod \"maas-api-595cc648c-rmtpq\" (UID: \"e9421326-17de-4a12-831f-edfd135b94c1\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-rmtpq" Oct 30 17:31:32.238443 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:32.238418 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9421326-17de-4a12-831f-edfd135b94c1-openshift-service-ca\") pod \"maas-api-595cc648c-rmtpq\" (UID: \"e9421326-17de-4a12-831f-edfd135b94c1\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-rmtpq" Oct 30 17:31:32.240095 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:32.240069 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9421326-17de-4a12-831f-edfd135b94c1-maas-api-tls\") pod \"maas-api-595cc648c-rmtpq\" (UID: \"e9421326-17de-4a12-831f-edfd135b94c1\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-rmtpq" Oct 30 17:31:32.240203 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:32.240127 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9421326-17de-4a12-831f-edfd135b94c1-metrics-certs\") pod \"maas-api-595cc648c-rmtpq\" (UID: \"e9421326-17de-4a12-831f-edfd135b94c1\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-rmtpq" Oct 30 17:31:32.260827 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:32.260797 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cm6tw\" (UniqueName: \"kubernetes.io/projected/e9421326-17de-4a12-831f-edfd135b94c1-kube-api-access-cm6tw\") pod \"maas-api-595cc648c-rmtpq\" (UID: \"e9421326-17de-4a12-831f-edfd135b94c1\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-rmtpq" Oct 30 17:31:32.373605 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:32.373511 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-595cc648c-rmtpq" Oct 30 17:31:32.675401 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:32.675360 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-rmtpq"] Oct 30 17:31:32.679305 ip-10-0-131-136 kubenswrapper[5052]: W1030 17:31:32.679275 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9421326_17de_4a12_831f_edfd135b94c1.slice/crio-67dc66d5e74f66733bb413a7e2eb7d443830e6dc6cac504134047ad111247192 WatchSource:0}: Error finding container 67dc66d5e74f66733bb413a7e2eb7d443830e6dc6cac504134047ad111247192: Status 404 returned error can't find the container with id 67dc66d5e74f66733bb413a7e2eb7d443830e6dc6cac504134047ad111247192 Oct 30 17:31:32.680877 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:32.680854 5052 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 17:31:33.572940 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:33.572886 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-595cc648c-rmtpq" event={"ID":"e9421326-17de-4a12-831f-edfd135b94c1","Type":"ContainerStarted","Data":"5ccdd10549464243280e9bb105fcd0e766b80abf22d572b5fbcef18181225b2c"} Oct 30 17:31:33.573440 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:33.572960 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-595cc648c-rmtpq" Oct 30 17:31:33.573440 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:33.572977 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-595cc648c-rmtpq" event={"ID":"e9421326-17de-4a12-831f-edfd135b94c1","Type":"ContainerStarted","Data":"67dc66d5e74f66733bb413a7e2eb7d443830e6dc6cac504134047ad111247192"} Oct 30 17:31:33.614050 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:33.614004 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-595cc648c-rmtpq" podStartSLOduration=1.233216534 podStartE2EDuration="1.613992303s" totalImagesPullingTime="380.775769ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:31:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:31:33.612520929 +0000 UTC m=+4249.139856209" watchObservedRunningTime="2026-10-30 17:31:33.613992303 +0000 UTC m=+4249.141327521" Oct 30 17:31:39.584418 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:39.584383 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-595cc648c-rmtpq" Oct 30 17:31:39.664258 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:39.664205 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-6xr5r"] Oct 30 17:31:39.664456 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:39.664433 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5c48877c75-6xr5r" podUID="c03b7d08-8589-49f4-a662-c7bf1d9c8caf" containerName="maas-api" containerID="cri-o://ac804409ebd126acbe5d4205c1d4973451a0f045cba8a290ff9dbc671a0bb7ac" gracePeriod=30 Oct 30 17:31:40.810046 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:40.810006 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cdwcj"] Oct 30 17:31:40.934114 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:40.934089 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-6xr5r" Oct 30 17:31:41.026102 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:41.026016 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c03b7d08-8589-49f4-a662-c7bf1d9c8caf-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c03b7d08-8589-49f4-a662-c7bf1d9c8caf-metrics-certs\") pod \"c03b7d08-8589-49f4-a662-c7bf1d9c8caf\" (UID: \"c03b7d08-8589-49f4-a662-c7bf1d9c8caf\") " Oct 30 17:31:41.026102 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:41.026079 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c03b7d08-8589-49f4-a662-c7bf1d9c8caf-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c03b7d08-8589-49f4-a662-c7bf1d9c8caf-openshift-service-ca\") pod \"c03b7d08-8589-49f4-a662-c7bf1d9c8caf\" (UID: \"c03b7d08-8589-49f4-a662-c7bf1d9c8caf\") " Oct 30 17:31:41.026346 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:41.026114 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c03b7d08-8589-49f4-a662-c7bf1d9c8caf-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c03b7d08-8589-49f4-a662-c7bf1d9c8caf-maas-api-tls\") pod \"c03b7d08-8589-49f4-a662-c7bf1d9c8caf\" (UID: \"c03b7d08-8589-49f4-a662-c7bf1d9c8caf\") " Oct 30 17:31:41.026346 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:41.026213 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c03b7d08-8589-49f4-a662-c7bf1d9c8caf-kube-api-access-fr2gv\" (UniqueName: \"kubernetes.io/projected/c03b7d08-8589-49f4-a662-c7bf1d9c8caf-kube-api-access-fr2gv\") pod \"c03b7d08-8589-49f4-a662-c7bf1d9c8caf\" (UID: \"c03b7d08-8589-49f4-a662-c7bf1d9c8caf\") " Oct 30 17:31:41.026597 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:41.026557 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c03b7d08-8589-49f4-a662-c7bf1d9c8caf-openshift-service-ca" pod "c03b7d08-8589-49f4-a662-c7bf1d9c8caf" (UID: "c03b7d08-8589-49f4-a662-c7bf1d9c8caf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:31:41.028393 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:41.028366 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c03b7d08-8589-49f4-a662-c7bf1d9c8caf-maas-api-tls" pod "c03b7d08-8589-49f4-a662-c7bf1d9c8caf" (UID: "c03b7d08-8589-49f4-a662-c7bf1d9c8caf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:31:41.028487 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:41.028410 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c03b7d08-8589-49f4-a662-c7bf1d9c8caf-kube-api-access-fr2gv" pod "c03b7d08-8589-49f4-a662-c7bf1d9c8caf" (UID: "c03b7d08-8589-49f4-a662-c7bf1d9c8caf"). InnerVolumeSpecName "kube-api-access-fr2gv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:31:41.028571 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:41.028547 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c03b7d08-8589-49f4-a662-c7bf1d9c8caf-metrics-certs" pod "c03b7d08-8589-49f4-a662-c7bf1d9c8caf" (UID: "c03b7d08-8589-49f4-a662-c7bf1d9c8caf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:31:41.127300 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:41.127260 5052 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c03b7d08-8589-49f4-a662-c7bf1d9c8caf-metrics-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:31:41.127300 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:41.127298 5052 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c03b7d08-8589-49f4-a662-c7bf1d9c8caf-openshift-service-ca\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:31:41.127300 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:41.127310 5052 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c03b7d08-8589-49f4-a662-c7bf1d9c8caf-maas-api-tls\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:31:41.127556 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:41.127322 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fr2gv\" (UniqueName: \"kubernetes.io/projected/c03b7d08-8589-49f4-a662-c7bf1d9c8caf-kube-api-access-fr2gv\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:31:41.617192 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:41.617143 5052 generic.go:356] "Generic (PLEG): container finished" podID="c03b7d08-8589-49f4-a662-c7bf1d9c8caf" containerID="ac804409ebd126acbe5d4205c1d4973451a0f045cba8a290ff9dbc671a0bb7ac" exitCode=0 Oct 30 17:31:41.617410 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:41.617298 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-6xr5r" Oct 30 17:31:41.618882 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:41.618651 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-6xr5r" event={"ID":"c03b7d08-8589-49f4-a662-c7bf1d9c8caf","Type":"ContainerDied","Data":"ac804409ebd126acbe5d4205c1d4973451a0f045cba8a290ff9dbc671a0bb7ac"} Oct 30 17:31:41.618882 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:41.618725 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-6xr5r" event={"ID":"c03b7d08-8589-49f4-a662-c7bf1d9c8caf","Type":"ContainerDied","Data":"1e8edf057a96b786e204885cc7258951310f8063b0131fcc42201807812841c1"} Oct 30 17:31:41.618882 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:41.618760 5052 scope.go:122] "RemoveContainer" containerID="ac804409ebd126acbe5d4205c1d4973451a0f045cba8a290ff9dbc671a0bb7ac" Oct 30 17:31:41.633742 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:41.633716 5052 scope.go:122] "RemoveContainer" containerID="ac804409ebd126acbe5d4205c1d4973451a0f045cba8a290ff9dbc671a0bb7ac" Oct 30 17:31:41.634032 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:31:41.634006 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac804409ebd126acbe5d4205c1d4973451a0f045cba8a290ff9dbc671a0bb7ac\": container with ID starting with ac804409ebd126acbe5d4205c1d4973451a0f045cba8a290ff9dbc671a0bb7ac not found: ID does not exist" containerID="ac804409ebd126acbe5d4205c1d4973451a0f045cba8a290ff9dbc671a0bb7ac" Oct 30 17:31:41.634120 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:41.634046 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac804409ebd126acbe5d4205c1d4973451a0f045cba8a290ff9dbc671a0bb7ac"} err="failed to get container status \"ac804409ebd126acbe5d4205c1d4973451a0f045cba8a290ff9dbc671a0bb7ac\": rpc error: code = NotFound desc = could not find container \"ac804409ebd126acbe5d4205c1d4973451a0f045cba8a290ff9dbc671a0bb7ac\": container with ID starting with ac804409ebd126acbe5d4205c1d4973451a0f045cba8a290ff9dbc671a0bb7ac not found: ID does not exist" Oct 30 17:31:41.670990 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:41.670951 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-6xr5r"] Oct 30 17:31:41.682064 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:41.682037 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-6xr5r"] Oct 30 17:31:42.615160 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:42.615123 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c03b7d08-8589-49f4-a662-c7bf1d9c8caf" path="/var/lib/kubelet/pods/c03b7d08-8589-49f4-a662-c7bf1d9c8caf/volumes" Oct 30 17:31:48.843865 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:31:48.843820 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 17:31:48.983174 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:31:48.983143 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cdwcj"] Oct 30 17:32:02.393202 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:32:02.393159 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cdwcj"] Oct 30 17:32:18.841397 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:32:18.841357 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 17:32:35.716221 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:32:35.716179 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cdwcj"] Oct 30 17:32:46.000272 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:32:46.000242 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5ddb48fb-kmgjr_25d85829-1725-4150-9c59-09c80da4dcab/authorino/0.log" Oct 30 17:32:48.842440 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:32:48.842401 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 17:33:18.839990 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:33:18.839954 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 17:33:27.182715 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:33:27.182682 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-5cfb77bf8c-hv6k2_dc986c17-d850-4f2b-8888-139ae5d8385c/manager/0.log" Oct 30 17:33:27.710515 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:33:27.710490 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-575c7b5c5b-p8bxg_44aa1bfa-6067-45b4-ada0-6ea89af3a587/manager/0.log" Oct 30 17:33:28.613935 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:33:28.613910 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-cbcd98d6d-l9znm_99a70d63-02f8-4891-be23-4d8b6891af04/copy-manifests/0.log" Oct 30 17:33:28.642749 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:33:28.642717 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-cbcd98d6d-l9znm_99a70d63-02f8-4891-be23-4d8b6891af04/manager/0.log" Oct 30 17:33:29.318822 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:33:29.318789 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-7d94f55f4b-4f74l_f3dcbff0-3582-4f17-8781-6aedfe47d607/proxy/0.log" Oct 30 17:33:30.353339 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:33:30.353309 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-595cc648c-rmtpq_e9421326-17de-4a12-831f-edfd135b94c1/maas-api/0.log" Oct 30 17:33:33.495201 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:33:33.495166 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5ddb48fb-kmgjr_25d85829-1725-4150-9c59-09c80da4dcab/authorino/0.log" Oct 30 17:33:34.327036 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:33:34.327002 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-57bb9bfff-ct9g7_3c887582-ac80-4a00-9e7d-87c569e4e425/manager/0.log" Oct 30 17:33:34.582291 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:33:34.582182 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-cdwcj_c1731802-685a-44c6-9700-efc3784bbbeb/limitador/0.log" Oct 30 17:33:34.771507 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:33:34.771478 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-856dd584c5-bh7c6_8c01f5e2-a20b-45c3-a6c6-4b62b02e88cd/manager/0.log" Oct 30 17:33:36.148552 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:33:36.148522 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-54d485b749-wgfjt_8cae735b-c0f8-4066-9ad3-2a688079eecf/kube-auth-proxy/0.log" Oct 30 17:33:36.746154 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:33:36.746117 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-9kbt9_384a9c57-053d-477e-badb-355261588881/istio-proxy/0.log" Oct 30 17:33:37.389787 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:33:37.389761 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-flvdc_42edda16-6400-45ef-bce3-a5ce362045ed/istio-proxy/0.log" Oct 30 17:33:37.583013 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:33:37.582980 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-kpphn_c4c977d2-7480-4d5c-87d6-743c519e2efc/istio-proxy/0.log" Oct 30 17:33:38.183711 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:33:38.183684 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-t8b5p_fcc7c9e1-99ea-4b97-9e95-2e23a00ac096/istio-proxy/0.log" Oct 30 17:33:38.577285 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:33:38.577204 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-4ptgl_973d1ea4-3981-4350-b719-edb7911c23ab/payload-processing/0.log" Oct 30 17:33:38.772660 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:33:38.772627 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-df85cd6ff-6pd8x_b99d8a1f-000b-48ff-b4b0-7a62bc79c032/router/0.log" Oct 30 17:33:48.840499 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:33:48.840460 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 17:33:49.572523 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:33:49.572493 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-62cvz_730923b8-cd00-4831-91d2-b4b0071ae57b/main/0.log" Oct 30 17:33:50.155597 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:33:50.155552 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-lhs5r_84448e7a-d354-4820-b6a9-df2183c1b9c2/main/0.log" Oct 30 17:33:50.355270 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:33:50.355229 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89845t_c2e67399-be2a-4721-a2ca-2128d7dfa09a/main/0.log" Oct 30 17:33:50.537972 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:33:50.537900 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-4p2lm_6ba3d48f-8737-4e17-a90b-89c3f7b41c59/main/0.log" Oct 30 17:33:51.675307 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:33:51.675275 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5ddb48fb-kmgjr_25d85829-1725-4150-9c59-09c80da4dcab/authorino/0.log" Oct 30 17:34:18.839096 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:34:18.839061 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 17:34:21.207799 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:34:21.207770 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-5cfb77bf8c-hv6k2_dc986c17-d850-4f2b-8888-139ae5d8385c/manager/0.log" Oct 30 17:34:21.661491 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:34:21.661460 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-575c7b5c5b-p8bxg_44aa1bfa-6067-45b4-ada0-6ea89af3a587/manager/0.log" Oct 30 17:34:22.502594 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:34:22.502562 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-cbcd98d6d-l9znm_99a70d63-02f8-4891-be23-4d8b6891af04/copy-manifests/0.log" Oct 30 17:34:22.529419 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:34:22.529393 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-cbcd98d6d-l9znm_99a70d63-02f8-4891-be23-4d8b6891af04/manager/0.log" Oct 30 17:34:23.219479 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:34:23.219441 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-7d94f55f4b-4f74l_f3dcbff0-3582-4f17-8781-6aedfe47d607/proxy/0.log" Oct 30 17:34:33.985685 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:34:33.985650 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-595cc648c-rmtpq_e9421326-17de-4a12-831f-edfd135b94c1/maas-api/0.log" Oct 30 17:34:48.839404 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:34:48.839309 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 17:34:56.492765 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:34:56.492717 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5ddb48fb-kmgjr_25d85829-1725-4150-9c59-09c80da4dcab/authorino/0.log" Oct 30 17:34:57.326954 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:34:57.326921 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-57bb9bfff-ct9g7_3c887582-ac80-4a00-9e7d-87c569e4e425/manager/0.log" Oct 30 17:34:57.559508 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:34:57.559476 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-cdwcj_c1731802-685a-44c6-9700-efc3784bbbeb/limitador/0.log" Oct 30 17:34:57.758554 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:34:57.758506 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-856dd584c5-bh7c6_8c01f5e2-a20b-45c3-a6c6-4b62b02e88cd/manager/0.log" Oct 30 17:34:58.920057 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:34:58.920027 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-54d485b749-wgfjt_8cae735b-c0f8-4066-9ad3-2a688079eecf/kube-auth-proxy/0.log" Oct 30 17:34:59.512918 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:34:59.512888 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-9kbt9_384a9c57-053d-477e-badb-355261588881/istio-proxy/0.log" Oct 30 17:35:00.117532 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:35:00.117497 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-flvdc_42edda16-6400-45ef-bce3-a5ce362045ed/istio-proxy/0.log" Oct 30 17:35:00.315937 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:35:00.315904 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-kpphn_c4c977d2-7480-4d5c-87d6-743c519e2efc/istio-proxy/0.log" Oct 30 17:35:00.922280 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:35:00.922252 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-t8b5p_fcc7c9e1-99ea-4b97-9e95-2e23a00ac096/istio-proxy/0.log" Oct 30 17:35:01.298029 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:35:01.297947 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-4ptgl_973d1ea4-3981-4350-b719-edb7911c23ab/payload-processing/0.log" Oct 30 17:35:01.479106 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:35:01.479073 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-df85cd6ff-6pd8x_b99d8a1f-000b-48ff-b4b0-7a62bc79c032/router/0.log" Oct 30 17:35:02.027433 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:35:02.027403 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-62cvz_730923b8-cd00-4831-91d2-b4b0071ae57b/main/0.log" Oct 30 17:35:02.609875 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:35:02.609842 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-lhs5r_84448e7a-d354-4820-b6a9-df2183c1b9c2/main/0.log" Oct 30 17:35:02.806044 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:35:02.806016 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89845t_c2e67399-be2a-4721-a2ca-2128d7dfa09a/main/0.log" Oct 30 17:35:03.004501 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:35:03.004469 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-4p2lm_6ba3d48f-8737-4e17-a90b-89c3f7b41c59/main/0.log" Oct 30 17:35:18.840255 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:35:18.840215 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 17:35:46.034113 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:35:46.034079 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4wrf8_4f0f46f4-5cda-4d67-b6f3-47df06eb844c/console-operator/2.log" Oct 30 17:35:46.042024 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:35:46.041993 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4wrf8_4f0f46f4-5cda-4d67-b6f3-47df06eb844c/console-operator/2.log" Oct 30 17:35:48.839948 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:35:48.839912 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 17:36:18.841008 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:36:18.840961 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 17:36:48.838949 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:36:48.838912 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 17:36:58.912305 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:36:58.912268 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-575c7b5c5b-p8bxg_44aa1bfa-6067-45b4-ada0-6ea89af3a587/manager/0.log" Oct 30 17:36:59.184169 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:36:59.184066 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-575c7b5c5b-p8bxg_44aa1bfa-6067-45b4-ada0-6ea89af3a587/manager/0.log" Oct 30 17:36:59.926112 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:36:59.926082 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-4ptgl_973d1ea4-3981-4350-b719-edb7911c23ab/payload-processing/0.log" Oct 30 17:37:00.111996 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:37:00.111966 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-4ptgl_973d1ea4-3981-4350-b719-edb7911c23ab/payload-processing/0.log" Oct 30 17:37:18.840367 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:37:18.840320 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 17:37:48.839553 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:37:48.839471 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 17:38:11.911470 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:11.911429 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-kpphn"] Oct 30 17:38:11.912103 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:11.911705 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-kpphn" podUID="c4c977d2-7480-4d5c-87d6-743c519e2efc" containerName="istio-proxy" containerID="cri-o://9105500bee23a6f4283835e83c6a352c25fd5a56b8e185c02dad5ca0abd07cb5" gracePeriod=30 Oct 30 17:38:11.922643 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:11.922619 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9kbt9"] Oct 30 17:38:11.922849 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:11.922812 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9kbt9" podUID="384a9c57-053d-477e-badb-355261588881" containerName="istio-proxy" containerID="cri-o://412866de6c1f173ecd61049e96aab8dd01501c4baf9a0d3b2cc9a4d11e88ab49" gracePeriod=30 Oct 30 17:38:11.924487 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:11.924462 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-t8b5p"] Oct 30 17:38:11.924683 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:11.924659 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-t8b5p" podUID="fcc7c9e1-99ea-4b97-9e95-2e23a00ac096" containerName="istio-proxy" containerID="cri-o://9e7b48641705d27c2cb8662a4c19de28ca2c5163584c0c95c4f608e52147f282" gracePeriod=30 Oct 30 17:38:11.933843 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:11.933819 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-flvdc"] Oct 30 17:38:11.934003 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:11.933983 5052 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-flvdc" podUID="42edda16-6400-45ef-bce3-a5ce362045ed" containerName="istio-proxy" containerID="cri-o://be1a58141492dfed2b72217076bc6e1c6fe7165d79543555ea496726d3ebf6f8" gracePeriod=30 Oct 30 17:38:17.239696 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.239659 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9kbt9" Oct 30 17:38:17.276700 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.276665 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-t8b5p" Oct 30 17:38:17.281527 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.281501 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-kpphn" Oct 30 17:38:17.285493 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.285457 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-flvdc" Oct 30 17:38:17.316825 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.316794 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/384a9c57-053d-477e-badb-355261588881-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/384a9c57-053d-477e-badb-355261588881-istio-data\") pod \"384a9c57-053d-477e-badb-355261588881\" (UID: \"384a9c57-053d-477e-badb-355261588881\") " Oct 30 17:38:17.316975 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.316832 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcc7c9e1-99ea-4b97-9e95-2e23a00ac096-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcc7c9e1-99ea-4b97-9e95-2e23a00ac096-workload-socket\") pod \"fcc7c9e1-99ea-4b97-9e95-2e23a00ac096\" (UID: \"fcc7c9e1-99ea-4b97-9e95-2e23a00ac096\") " Oct 30 17:38:17.316975 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.316862 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fcc7c9e1-99ea-4b97-9e95-2e23a00ac096-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcc7c9e1-99ea-4b97-9e95-2e23a00ac096-istio-podinfo\") pod \"fcc7c9e1-99ea-4b97-9e95-2e23a00ac096\" (UID: \"fcc7c9e1-99ea-4b97-9e95-2e23a00ac096\") " Oct 30 17:38:17.316975 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.316892 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42edda16-6400-45ef-bce3-a5ce362045ed-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42edda16-6400-45ef-bce3-a5ce362045ed-istio-envoy\") pod \"42edda16-6400-45ef-bce3-a5ce362045ed\" (UID: \"42edda16-6400-45ef-bce3-a5ce362045ed\") " Oct 30 17:38:17.316975 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.316922 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4c977d2-7480-4d5c-87d6-743c519e2efc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4c977d2-7480-4d5c-87d6-743c519e2efc-workload-socket\") pod \"c4c977d2-7480-4d5c-87d6-743c519e2efc\" (UID: \"c4c977d2-7480-4d5c-87d6-743c519e2efc\") " Oct 30 17:38:17.316975 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.316951 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/42edda16-6400-45ef-bce3-a5ce362045ed-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42edda16-6400-45ef-bce3-a5ce362045ed-istio-podinfo\") pod \"42edda16-6400-45ef-bce3-a5ce362045ed\" (UID: \"42edda16-6400-45ef-bce3-a5ce362045ed\") " Oct 30 17:38:17.317159 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.316995 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcc7c9e1-99ea-4b97-9e95-2e23a00ac096-istio-token\" (UniqueName: \"kubernetes.io/projected/fcc7c9e1-99ea-4b97-9e95-2e23a00ac096-istio-token\") pod \"fcc7c9e1-99ea-4b97-9e95-2e23a00ac096\" (UID: \"fcc7c9e1-99ea-4b97-9e95-2e23a00ac096\") " Oct 30 17:38:17.317159 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.317020 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/384a9c57-053d-477e-badb-355261588881-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/384a9c57-053d-477e-badb-355261588881-workload-certs\") pod \"384a9c57-053d-477e-badb-355261588881\" (UID: \"384a9c57-053d-477e-badb-355261588881\") " Oct 30 17:38:17.317159 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.317052 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c4c977d2-7480-4d5c-87d6-743c519e2efc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4c977d2-7480-4d5c-87d6-743c519e2efc-istio-podinfo\") pod \"c4c977d2-7480-4d5c-87d6-743c519e2efc\" (UID: \"c4c977d2-7480-4d5c-87d6-743c519e2efc\") " Oct 30 17:38:17.317159 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.317084 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/384a9c57-053d-477e-badb-355261588881-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/384a9c57-053d-477e-badb-355261588881-istio-podinfo\") pod \"384a9c57-053d-477e-badb-355261588881\" (UID: \"384a9c57-053d-477e-badb-355261588881\") " Oct 30 17:38:17.317159 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.317130 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcc7c9e1-99ea-4b97-9e95-2e23a00ac096-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcc7c9e1-99ea-4b97-9e95-2e23a00ac096-istio-envoy\") pod \"fcc7c9e1-99ea-4b97-9e95-2e23a00ac096\" (UID: \"fcc7c9e1-99ea-4b97-9e95-2e23a00ac096\") " Oct 30 17:38:17.317377 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.317190 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/384a9c57-053d-477e-badb-355261588881-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/384a9c57-053d-477e-badb-355261588881-istiod-ca-cert\") pod \"384a9c57-053d-477e-badb-355261588881\" (UID: \"384a9c57-053d-477e-badb-355261588881\") " Oct 30 17:38:17.317377 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.317233 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4c977d2-7480-4d5c-87d6-743c519e2efc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4c977d2-7480-4d5c-87d6-743c519e2efc-istio-data\") pod \"c4c977d2-7480-4d5c-87d6-743c519e2efc\" (UID: \"c4c977d2-7480-4d5c-87d6-743c519e2efc\") " Oct 30 17:38:17.317377 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.317263 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/384a9c57-053d-477e-badb-355261588881-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/384a9c57-053d-477e-badb-355261588881-credential-socket\") pod \"384a9c57-053d-477e-badb-355261588881\" (UID: \"384a9c57-053d-477e-badb-355261588881\") " Oct 30 17:38:17.317377 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.317290 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4c977d2-7480-4d5c-87d6-743c519e2efc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4c977d2-7480-4d5c-87d6-743c519e2efc-istiod-ca-cert\") pod \"c4c977d2-7480-4d5c-87d6-743c519e2efc\" (UID: \"c4c977d2-7480-4d5c-87d6-743c519e2efc\") " Oct 30 17:38:17.317377 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.317322 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcc7c9e1-99ea-4b97-9e95-2e23a00ac096-kube-api-access-vvwlt\" (UniqueName: \"kubernetes.io/projected/fcc7c9e1-99ea-4b97-9e95-2e23a00ac096-kube-api-access-vvwlt\") pod \"fcc7c9e1-99ea-4b97-9e95-2e23a00ac096\" (UID: \"fcc7c9e1-99ea-4b97-9e95-2e23a00ac096\") " Oct 30 17:38:17.317377 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.317352 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/384a9c57-053d-477e-badb-355261588881-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/384a9c57-053d-477e-badb-355261588881-workload-socket\") pod \"384a9c57-053d-477e-badb-355261588881\" (UID: \"384a9c57-053d-477e-badb-355261588881\") " Oct 30 17:38:17.317821 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.317385 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcc7c9e1-99ea-4b97-9e95-2e23a00ac096-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcc7c9e1-99ea-4b97-9e95-2e23a00ac096-istio-data\") pod \"fcc7c9e1-99ea-4b97-9e95-2e23a00ac096\" (UID: \"fcc7c9e1-99ea-4b97-9e95-2e23a00ac096\") " Oct 30 17:38:17.317821 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.317413 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42edda16-6400-45ef-bce3-a5ce362045ed-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42edda16-6400-45ef-bce3-a5ce362045ed-istiod-ca-cert\") pod \"42edda16-6400-45ef-bce3-a5ce362045ed\" (UID: \"42edda16-6400-45ef-bce3-a5ce362045ed\") " Oct 30 17:38:17.317821 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.317481 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4c977d2-7480-4d5c-87d6-743c519e2efc-kube-api-access-bg6wj\" (UniqueName: \"kubernetes.io/projected/c4c977d2-7480-4d5c-87d6-743c519e2efc-kube-api-access-bg6wj\") pod \"c4c977d2-7480-4d5c-87d6-743c519e2efc\" (UID: \"c4c977d2-7480-4d5c-87d6-743c519e2efc\") " Oct 30 17:38:17.317821 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.317542 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42edda16-6400-45ef-bce3-a5ce362045ed-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42edda16-6400-45ef-bce3-a5ce362045ed-workload-certs\") pod \"42edda16-6400-45ef-bce3-a5ce362045ed\" (UID: \"42edda16-6400-45ef-bce3-a5ce362045ed\") " Oct 30 17:38:17.317821 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.317572 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4c977d2-7480-4d5c-87d6-743c519e2efc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4c977d2-7480-4d5c-87d6-743c519e2efc-istio-envoy\") pod \"c4c977d2-7480-4d5c-87d6-743c519e2efc\" (UID: \"c4c977d2-7480-4d5c-87d6-743c519e2efc\") " Oct 30 17:38:17.317821 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.317644 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcc7c9e1-99ea-4b97-9e95-2e23a00ac096-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcc7c9e1-99ea-4b97-9e95-2e23a00ac096-istiod-ca-cert\") pod \"fcc7c9e1-99ea-4b97-9e95-2e23a00ac096\" (UID: \"fcc7c9e1-99ea-4b97-9e95-2e23a00ac096\") " Oct 30 17:38:17.317821 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.317682 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcc7c9e1-99ea-4b97-9e95-2e23a00ac096-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcc7c9e1-99ea-4b97-9e95-2e23a00ac096-credential-socket\") pod \"fcc7c9e1-99ea-4b97-9e95-2e23a00ac096\" (UID: \"fcc7c9e1-99ea-4b97-9e95-2e23a00ac096\") " Oct 30 17:38:17.317821 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.317714 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42edda16-6400-45ef-bce3-a5ce362045ed-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42edda16-6400-45ef-bce3-a5ce362045ed-istio-data\") pod \"42edda16-6400-45ef-bce3-a5ce362045ed\" (UID: \"42edda16-6400-45ef-bce3-a5ce362045ed\") " Oct 30 17:38:17.317821 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.317742 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4c977d2-7480-4d5c-87d6-743c519e2efc-istio-token\" (UniqueName: \"kubernetes.io/projected/c4c977d2-7480-4d5c-87d6-743c519e2efc-istio-token\") pod \"c4c977d2-7480-4d5c-87d6-743c519e2efc\" (UID: \"c4c977d2-7480-4d5c-87d6-743c519e2efc\") " Oct 30 17:38:17.317821 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.317790 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4c977d2-7480-4d5c-87d6-743c519e2efc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4c977d2-7480-4d5c-87d6-743c519e2efc-workload-certs\") pod \"c4c977d2-7480-4d5c-87d6-743c519e2efc\" (UID: \"c4c977d2-7480-4d5c-87d6-743c519e2efc\") " Oct 30 17:38:17.320934 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.317084 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcc7c9e1-99ea-4b97-9e95-2e23a00ac096-workload-socket" pod "fcc7c9e1-99ea-4b97-9e95-2e23a00ac096" (UID: "fcc7c9e1-99ea-4b97-9e95-2e23a00ac096"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:38:17.320934 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.317092 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/384a9c57-053d-477e-badb-355261588881-istio-data" pod "384a9c57-053d-477e-badb-355261588881" (UID: "384a9c57-053d-477e-badb-355261588881"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:38:17.320934 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.317803 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/384a9c57-053d-477e-badb-355261588881-istiod-ca-cert" pod "384a9c57-053d-477e-badb-355261588881" (UID: "384a9c57-053d-477e-badb-355261588881"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:38:17.320934 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.317894 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4c977d2-7480-4d5c-87d6-743c519e2efc-istiod-ca-cert" pod "c4c977d2-7480-4d5c-87d6-743c519e2efc" (UID: "c4c977d2-7480-4d5c-87d6-743c519e2efc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:38:17.320934 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.317979 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4c977d2-7480-4d5c-87d6-743c519e2efc-istio-data" pod "c4c977d2-7480-4d5c-87d6-743c519e2efc" (UID: "c4c977d2-7480-4d5c-87d6-743c519e2efc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:38:17.320934 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.318038 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4c977d2-7480-4d5c-87d6-743c519e2efc-workload-certs" pod "c4c977d2-7480-4d5c-87d6-743c519e2efc" (UID: "c4c977d2-7480-4d5c-87d6-743c519e2efc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:38:17.320934 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.318184 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/384a9c57-053d-477e-badb-355261588881-credential-socket" pod "384a9c57-053d-477e-badb-355261588881" (UID: "384a9c57-053d-477e-badb-355261588881"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:38:17.320934 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.318212 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/384a9c57-053d-477e-badb-355261588881-workload-socket" pod "384a9c57-053d-477e-badb-355261588881" (UID: "384a9c57-053d-477e-badb-355261588881"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:38:17.320934 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.318241 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcc7c9e1-99ea-4b97-9e95-2e23a00ac096-istio-data" pod "fcc7c9e1-99ea-4b97-9e95-2e23a00ac096" (UID: "fcc7c9e1-99ea-4b97-9e95-2e23a00ac096"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:38:17.320934 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.318256 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4c977d2-7480-4d5c-87d6-743c519e2efc-workload-socket" pod "c4c977d2-7480-4d5c-87d6-743c519e2efc" (UID: "c4c977d2-7480-4d5c-87d6-743c519e2efc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:38:17.320934 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.318341 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42edda16-6400-45ef-bce3-a5ce362045ed-istiod-ca-cert" pod "42edda16-6400-45ef-bce3-a5ce362045ed" (UID: "42edda16-6400-45ef-bce3-a5ce362045ed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:38:17.320934 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.318445 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42edda16-6400-45ef-bce3-a5ce362045ed-istio-data" pod "42edda16-6400-45ef-bce3-a5ce362045ed" (UID: "42edda16-6400-45ef-bce3-a5ce362045ed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:38:17.320934 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.318530 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcc7c9e1-99ea-4b97-9e95-2e23a00ac096-istiod-ca-cert" pod "fcc7c9e1-99ea-4b97-9e95-2e23a00ac096" (UID: "fcc7c9e1-99ea-4b97-9e95-2e23a00ac096"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:38:17.320934 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.318786 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42edda16-6400-45ef-bce3-a5ce362045ed-workload-certs" pod "42edda16-6400-45ef-bce3-a5ce362045ed" (UID: "42edda16-6400-45ef-bce3-a5ce362045ed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:38:17.320934 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.319055 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/384a9c57-053d-477e-badb-355261588881-workload-certs" pod "384a9c57-053d-477e-badb-355261588881" (UID: "384a9c57-053d-477e-badb-355261588881"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:38:17.320934 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.319064 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42edda16-6400-45ef-bce3-a5ce362045ed-kube-api-access-6r6s4\" (UniqueName: \"kubernetes.io/projected/42edda16-6400-45ef-bce3-a5ce362045ed-kube-api-access-6r6s4\") pod \"42edda16-6400-45ef-bce3-a5ce362045ed\" (UID: \"42edda16-6400-45ef-bce3-a5ce362045ed\") " Oct 30 17:38:17.320934 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.319376 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42edda16-6400-45ef-bce3-a5ce362045ed-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42edda16-6400-45ef-bce3-a5ce362045ed-workload-socket\") pod \"42edda16-6400-45ef-bce3-a5ce362045ed\" (UID: \"42edda16-6400-45ef-bce3-a5ce362045ed\") " Oct 30 17:38:17.320934 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.319417 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/384a9c57-053d-477e-badb-355261588881-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/384a9c57-053d-477e-badb-355261588881-istio-envoy\") pod \"384a9c57-053d-477e-badb-355261588881\" (UID: \"384a9c57-053d-477e-badb-355261588881\") " Oct 30 17:38:17.320934 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.319446 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4c977d2-7480-4d5c-87d6-743c519e2efc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4c977d2-7480-4d5c-87d6-743c519e2efc-credential-socket\") pod \"c4c977d2-7480-4d5c-87d6-743c519e2efc\" (UID: \"c4c977d2-7480-4d5c-87d6-743c519e2efc\") " Oct 30 17:38:17.320934 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.319482 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42edda16-6400-45ef-bce3-a5ce362045ed-istio-token\" (UniqueName: \"kubernetes.io/projected/42edda16-6400-45ef-bce3-a5ce362045ed-istio-token\") pod \"42edda16-6400-45ef-bce3-a5ce362045ed\" (UID: \"42edda16-6400-45ef-bce3-a5ce362045ed\") " Oct 30 17:38:17.320934 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.319519 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcc7c9e1-99ea-4b97-9e95-2e23a00ac096-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcc7c9e1-99ea-4b97-9e95-2e23a00ac096-workload-certs\") pod \"fcc7c9e1-99ea-4b97-9e95-2e23a00ac096\" (UID: \"fcc7c9e1-99ea-4b97-9e95-2e23a00ac096\") " Oct 30 17:38:17.320934 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.319594 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/384a9c57-053d-477e-badb-355261588881-kube-api-access-zwrzs\" (UniqueName: \"kubernetes.io/projected/384a9c57-053d-477e-badb-355261588881-kube-api-access-zwrzs\") pod \"384a9c57-053d-477e-badb-355261588881\" (UID: \"384a9c57-053d-477e-badb-355261588881\") " Oct 30 17:38:17.320934 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.319643 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/384a9c57-053d-477e-badb-355261588881-istio-token\" (UniqueName: \"kubernetes.io/projected/384a9c57-053d-477e-badb-355261588881-istio-token\") pod \"384a9c57-053d-477e-badb-355261588881\" (UID: \"384a9c57-053d-477e-badb-355261588881\") " Oct 30 17:38:17.320934 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.319691 5052 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42edda16-6400-45ef-bce3-a5ce362045ed-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42edda16-6400-45ef-bce3-a5ce362045ed-credential-socket\") pod \"42edda16-6400-45ef-bce3-a5ce362045ed\" (UID: \"42edda16-6400-45ef-bce3-a5ce362045ed\") " Oct 30 17:38:17.320934 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.319827 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4c977d2-7480-4d5c-87d6-743c519e2efc-credential-socket" pod "c4c977d2-7480-4d5c-87d6-743c519e2efc" (UID: "c4c977d2-7480-4d5c-87d6-743c519e2efc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:38:17.320934 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.320039 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42edda16-6400-45ef-bce3-a5ce362045ed-workload-socket" pod "42edda16-6400-45ef-bce3-a5ce362045ed" (UID: "42edda16-6400-45ef-bce3-a5ce362045ed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:38:17.320934 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.320222 5052 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/384a9c57-053d-477e-badb-355261588881-istio-data\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:38:17.320934 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.320238 5052 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcc7c9e1-99ea-4b97-9e95-2e23a00ac096-workload-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:38:17.320934 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.320252 5052 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4c977d2-7480-4d5c-87d6-743c519e2efc-workload-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:38:17.320934 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.320250 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42edda16-6400-45ef-bce3-a5ce362045ed-credential-socket" pod "42edda16-6400-45ef-bce3-a5ce362045ed" (UID: "42edda16-6400-45ef-bce3-a5ce362045ed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:38:17.320934 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.320266 5052 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/384a9c57-053d-477e-badb-355261588881-workload-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:38:17.320934 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.320280 5052 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/384a9c57-053d-477e-badb-355261588881-istiod-ca-cert\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:38:17.320934 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.320295 5052 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4c977d2-7480-4d5c-87d6-743c519e2efc-istio-data\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:38:17.320934 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.320307 5052 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/384a9c57-053d-477e-badb-355261588881-credential-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:38:17.320934 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.320321 5052 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4c977d2-7480-4d5c-87d6-743c519e2efc-istiod-ca-cert\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:38:17.320934 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.320334 5052 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/384a9c57-053d-477e-badb-355261588881-workload-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:38:17.320934 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.320346 5052 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcc7c9e1-99ea-4b97-9e95-2e23a00ac096-istio-data\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:38:17.320934 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.320360 5052 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42edda16-6400-45ef-bce3-a5ce362045ed-istiod-ca-cert\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:38:17.320934 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.320373 5052 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42edda16-6400-45ef-bce3-a5ce362045ed-workload-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:38:17.320934 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.320378 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcc7c9e1-99ea-4b97-9e95-2e23a00ac096-credential-socket" pod "fcc7c9e1-99ea-4b97-9e95-2e23a00ac096" (UID: "fcc7c9e1-99ea-4b97-9e95-2e23a00ac096"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:38:17.320934 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.320385 5052 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcc7c9e1-99ea-4b97-9e95-2e23a00ac096-istiod-ca-cert\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:38:17.320934 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.320434 5052 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42edda16-6400-45ef-bce3-a5ce362045ed-istio-data\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:38:17.320934 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.320451 5052 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4c977d2-7480-4d5c-87d6-743c519e2efc-workload-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:38:17.320934 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.320465 5052 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42edda16-6400-45ef-bce3-a5ce362045ed-workload-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:38:17.320934 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.320483 5052 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4c977d2-7480-4d5c-87d6-743c519e2efc-credential-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:38:17.320934 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.320629 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcc7c9e1-99ea-4b97-9e95-2e23a00ac096-kube-api-access-vvwlt" pod "fcc7c9e1-99ea-4b97-9e95-2e23a00ac096" (UID: "fcc7c9e1-99ea-4b97-9e95-2e23a00ac096"). InnerVolumeSpecName "kube-api-access-vvwlt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:38:17.324149 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.321300 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcc7c9e1-99ea-4b97-9e95-2e23a00ac096-workload-certs" pod "fcc7c9e1-99ea-4b97-9e95-2e23a00ac096" (UID: "fcc7c9e1-99ea-4b97-9e95-2e23a00ac096"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:38:17.324149 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.321176 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42edda16-6400-45ef-bce3-a5ce362045ed-istio-envoy" pod "42edda16-6400-45ef-bce3-a5ce362045ed" (UID: "42edda16-6400-45ef-bce3-a5ce362045ed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:38:17.324149 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.321545 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fcc7c9e1-99ea-4b97-9e95-2e23a00ac096-istio-podinfo" pod "fcc7c9e1-99ea-4b97-9e95-2e23a00ac096" (UID: "fcc7c9e1-99ea-4b97-9e95-2e23a00ac096"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:38:17.324149 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.322248 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4c977d2-7480-4d5c-87d6-743c519e2efc-istio-token" pod "c4c977d2-7480-4d5c-87d6-743c519e2efc" (UID: "c4c977d2-7480-4d5c-87d6-743c519e2efc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:38:17.324149 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.323725 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42edda16-6400-45ef-bce3-a5ce362045ed-kube-api-access-6r6s4" pod "42edda16-6400-45ef-bce3-a5ce362045ed" (UID: "42edda16-6400-45ef-bce3-a5ce362045ed"). InnerVolumeSpecName "kube-api-access-6r6s4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:38:17.324149 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.323767 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4c977d2-7480-4d5c-87d6-743c519e2efc-kube-api-access-bg6wj" pod "c4c977d2-7480-4d5c-87d6-743c519e2efc" (UID: "c4c977d2-7480-4d5c-87d6-743c519e2efc"). InnerVolumeSpecName "kube-api-access-bg6wj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:38:17.324149 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.323784 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c4c977d2-7480-4d5c-87d6-743c519e2efc-istio-podinfo" pod "c4c977d2-7480-4d5c-87d6-743c519e2efc" (UID: "c4c977d2-7480-4d5c-87d6-743c519e2efc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:38:17.324149 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.323791 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcc7c9e1-99ea-4b97-9e95-2e23a00ac096-istio-token" pod "fcc7c9e1-99ea-4b97-9e95-2e23a00ac096" (UID: "fcc7c9e1-99ea-4b97-9e95-2e23a00ac096"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:38:17.324149 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.323823 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42edda16-6400-45ef-bce3-a5ce362045ed-istio-token" pod "42edda16-6400-45ef-bce3-a5ce362045ed" (UID: "42edda16-6400-45ef-bce3-a5ce362045ed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:38:17.324149 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.323852 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcc7c9e1-99ea-4b97-9e95-2e23a00ac096-istio-envoy" pod "fcc7c9e1-99ea-4b97-9e95-2e23a00ac096" (UID: "fcc7c9e1-99ea-4b97-9e95-2e23a00ac096"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:38:17.324149 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.323948 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/42edda16-6400-45ef-bce3-a5ce362045ed-istio-podinfo" pod "42edda16-6400-45ef-bce3-a5ce362045ed" (UID: "42edda16-6400-45ef-bce3-a5ce362045ed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:38:17.324914 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.324214 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/384a9c57-053d-477e-badb-355261588881-kube-api-access-zwrzs" pod "384a9c57-053d-477e-badb-355261588881" (UID: "384a9c57-053d-477e-badb-355261588881"). InnerVolumeSpecName "kube-api-access-zwrzs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:38:17.324914 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.324254 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4c977d2-7480-4d5c-87d6-743c519e2efc-istio-envoy" pod "c4c977d2-7480-4d5c-87d6-743c519e2efc" (UID: "c4c977d2-7480-4d5c-87d6-743c519e2efc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:38:17.324914 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.324501 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/384a9c57-053d-477e-badb-355261588881-istio-token" pod "384a9c57-053d-477e-badb-355261588881" (UID: "384a9c57-053d-477e-badb-355261588881"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:38:17.325087 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.324998 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/384a9c57-053d-477e-badb-355261588881-istio-envoy" pod "384a9c57-053d-477e-badb-355261588881" (UID: "384a9c57-053d-477e-badb-355261588881"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:38:17.326783 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.326760 5052 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/384a9c57-053d-477e-badb-355261588881-istio-podinfo" pod "384a9c57-053d-477e-badb-355261588881" (UID: "384a9c57-053d-477e-badb-355261588881"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:38:17.421947 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.421917 5052 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcc7c9e1-99ea-4b97-9e95-2e23a00ac096-istio-token\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:38:17.421947 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.421946 5052 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4c977d2-7480-4d5c-87d6-743c519e2efc-istio-podinfo\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:38:17.421947 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.421955 5052 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/384a9c57-053d-477e-badb-355261588881-istio-podinfo\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:38:17.422200 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.421964 5052 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcc7c9e1-99ea-4b97-9e95-2e23a00ac096-istio-envoy\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:38:17.422200 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.421974 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvwlt\" (UniqueName: \"kubernetes.io/projected/fcc7c9e1-99ea-4b97-9e95-2e23a00ac096-kube-api-access-vvwlt\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:38:17.422200 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.421984 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bg6wj\" (UniqueName: \"kubernetes.io/projected/c4c977d2-7480-4d5c-87d6-743c519e2efc-kube-api-access-bg6wj\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:38:17.422200 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.421992 5052 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4c977d2-7480-4d5c-87d6-743c519e2efc-istio-envoy\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:38:17.422200 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.422000 5052 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcc7c9e1-99ea-4b97-9e95-2e23a00ac096-credential-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:38:17.422200 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.422007 5052 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4c977d2-7480-4d5c-87d6-743c519e2efc-istio-token\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:38:17.422200 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.422017 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6r6s4\" (UniqueName: \"kubernetes.io/projected/42edda16-6400-45ef-bce3-a5ce362045ed-kube-api-access-6r6s4\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:38:17.422200 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.422026 5052 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/384a9c57-053d-477e-badb-355261588881-istio-envoy\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:38:17.422200 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.422035 5052 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42edda16-6400-45ef-bce3-a5ce362045ed-istio-token\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:38:17.422200 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.422043 5052 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcc7c9e1-99ea-4b97-9e95-2e23a00ac096-workload-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:38:17.422200 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.422051 5052 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwrzs\" (UniqueName: \"kubernetes.io/projected/384a9c57-053d-477e-badb-355261588881-kube-api-access-zwrzs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:38:17.422200 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.422058 5052 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/384a9c57-053d-477e-badb-355261588881-istio-token\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:38:17.422200 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.422066 5052 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42edda16-6400-45ef-bce3-a5ce362045ed-credential-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:38:17.422200 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.422074 5052 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcc7c9e1-99ea-4b97-9e95-2e23a00ac096-istio-podinfo\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:38:17.422200 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.422082 5052 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42edda16-6400-45ef-bce3-a5ce362045ed-istio-envoy\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:38:17.422200 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.422090 5052 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42edda16-6400-45ef-bce3-a5ce362045ed-istio-podinfo\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\"" Oct 30 17:38:17.794305 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.794271 5052 generic.go:356] "Generic (PLEG): container finished" podID="c4c977d2-7480-4d5c-87d6-743c519e2efc" containerID="9105500bee23a6f4283835e83c6a352c25fd5a56b8e185c02dad5ca0abd07cb5" exitCode=0 Oct 30 17:38:17.794649 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.794338 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-kpphn" Oct 30 17:38:17.794649 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.794356 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-kpphn" event={"ID":"c4c977d2-7480-4d5c-87d6-743c519e2efc","Type":"ContainerDied","Data":"9105500bee23a6f4283835e83c6a352c25fd5a56b8e185c02dad5ca0abd07cb5"} Oct 30 17:38:17.794649 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.794438 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-kpphn" event={"ID":"c4c977d2-7480-4d5c-87d6-743c519e2efc","Type":"ContainerDied","Data":"fd8deaf4757c6de6ed7cf6a9bfe249a48466041bd4be8034be05da7a0e240e04"} Oct 30 17:38:17.794649 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.794471 5052 scope.go:122] "RemoveContainer" containerID="9105500bee23a6f4283835e83c6a352c25fd5a56b8e185c02dad5ca0abd07cb5" Oct 30 17:38:17.795780 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.795759 5052 generic.go:356] "Generic (PLEG): container finished" podID="fcc7c9e1-99ea-4b97-9e95-2e23a00ac096" containerID="9e7b48641705d27c2cb8662a4c19de28ca2c5163584c0c95c4f608e52147f282" exitCode=0 Oct 30 17:38:17.795897 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.795826 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-t8b5p" Oct 30 17:38:17.795897 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.795839 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-t8b5p" event={"ID":"fcc7c9e1-99ea-4b97-9e95-2e23a00ac096","Type":"ContainerDied","Data":"9e7b48641705d27c2cb8662a4c19de28ca2c5163584c0c95c4f608e52147f282"} Oct 30 17:38:17.795897 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.795890 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-t8b5p" event={"ID":"fcc7c9e1-99ea-4b97-9e95-2e23a00ac096","Type":"ContainerDied","Data":"655d7856bc618f3258c537f8a41c33477fffaf1929539f36b24b3b3721562c54"} Oct 30 17:38:17.802808 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.802786 5052 generic.go:356] "Generic (PLEG): container finished" podID="42edda16-6400-45ef-bce3-a5ce362045ed" containerID="be1a58141492dfed2b72217076bc6e1c6fe7165d79543555ea496726d3ebf6f8" exitCode=0 Oct 30 17:38:17.802919 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.802829 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-flvdc" Oct 30 17:38:17.802919 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.802857 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-flvdc" event={"ID":"42edda16-6400-45ef-bce3-a5ce362045ed","Type":"ContainerDied","Data":"be1a58141492dfed2b72217076bc6e1c6fe7165d79543555ea496726d3ebf6f8"} Oct 30 17:38:17.802919 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.802898 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-flvdc" event={"ID":"42edda16-6400-45ef-bce3-a5ce362045ed","Type":"ContainerDied","Data":"1f1bdf20ba5370ee95a254fb9ada80b84cfe8cd4c5b96387b84ece3d1c700ac5"} Oct 30 17:38:17.803971 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.803944 5052 scope.go:122] "RemoveContainer" containerID="9105500bee23a6f4283835e83c6a352c25fd5a56b8e185c02dad5ca0abd07cb5" Oct 30 17:38:17.804241 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:38:17.804214 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9105500bee23a6f4283835e83c6a352c25fd5a56b8e185c02dad5ca0abd07cb5\": container with ID starting with 9105500bee23a6f4283835e83c6a352c25fd5a56b8e185c02dad5ca0abd07cb5 not found: ID does not exist" containerID="9105500bee23a6f4283835e83c6a352c25fd5a56b8e185c02dad5ca0abd07cb5" Oct 30 17:38:17.804357 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.804251 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9105500bee23a6f4283835e83c6a352c25fd5a56b8e185c02dad5ca0abd07cb5"} err="failed to get container status \"9105500bee23a6f4283835e83c6a352c25fd5a56b8e185c02dad5ca0abd07cb5\": rpc error: code = NotFound desc = could not find container \"9105500bee23a6f4283835e83c6a352c25fd5a56b8e185c02dad5ca0abd07cb5\": container with ID starting with 9105500bee23a6f4283835e83c6a352c25fd5a56b8e185c02dad5ca0abd07cb5 not found: ID does not exist" Oct 30 17:38:17.804357 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.804274 5052 scope.go:122] "RemoveContainer" containerID="9e7b48641705d27c2cb8662a4c19de28ca2c5163584c0c95c4f608e52147f282" Oct 30 17:38:17.804502 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.804475 5052 generic.go:356] "Generic (PLEG): container finished" podID="384a9c57-053d-477e-badb-355261588881" containerID="412866de6c1f173ecd61049e96aab8dd01501c4baf9a0d3b2cc9a4d11e88ab49" exitCode=0 Oct 30 17:38:17.804502 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.804493 5052 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9kbt9" Oct 30 17:38:17.804639 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.804508 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9kbt9" event={"ID":"384a9c57-053d-477e-badb-355261588881","Type":"ContainerDied","Data":"412866de6c1f173ecd61049e96aab8dd01501c4baf9a0d3b2cc9a4d11e88ab49"} Oct 30 17:38:17.804639 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.804548 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9kbt9" event={"ID":"384a9c57-053d-477e-badb-355261588881","Type":"ContainerDied","Data":"002300998ffbb2b9adac292c5f8f8bafd04b4ea8932b00432630e20ed22c3afc"} Oct 30 17:38:17.816058 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.816035 5052 scope.go:122] "RemoveContainer" containerID="9e7b48641705d27c2cb8662a4c19de28ca2c5163584c0c95c4f608e52147f282" Oct 30 17:38:17.816440 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:38:17.816412 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e7b48641705d27c2cb8662a4c19de28ca2c5163584c0c95c4f608e52147f282\": container with ID starting with 9e7b48641705d27c2cb8662a4c19de28ca2c5163584c0c95c4f608e52147f282 not found: ID does not exist" containerID="9e7b48641705d27c2cb8662a4c19de28ca2c5163584c0c95c4f608e52147f282" Oct 30 17:38:17.816527 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.816449 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e7b48641705d27c2cb8662a4c19de28ca2c5163584c0c95c4f608e52147f282"} err="failed to get container status \"9e7b48641705d27c2cb8662a4c19de28ca2c5163584c0c95c4f608e52147f282\": rpc error: code = NotFound desc = could not find container \"9e7b48641705d27c2cb8662a4c19de28ca2c5163584c0c95c4f608e52147f282\": container with ID starting with 9e7b48641705d27c2cb8662a4c19de28ca2c5163584c0c95c4f608e52147f282 not found: ID does not exist" Oct 30 17:38:17.816527 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.816472 5052 scope.go:122] "RemoveContainer" containerID="be1a58141492dfed2b72217076bc6e1c6fe7165d79543555ea496726d3ebf6f8" Oct 30 17:38:17.824720 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.824699 5052 scope.go:122] "RemoveContainer" containerID="be1a58141492dfed2b72217076bc6e1c6fe7165d79543555ea496726d3ebf6f8" Oct 30 17:38:17.824930 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:38:17.824914 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be1a58141492dfed2b72217076bc6e1c6fe7165d79543555ea496726d3ebf6f8\": container with ID starting with be1a58141492dfed2b72217076bc6e1c6fe7165d79543555ea496726d3ebf6f8 not found: ID does not exist" containerID="be1a58141492dfed2b72217076bc6e1c6fe7165d79543555ea496726d3ebf6f8" Oct 30 17:38:17.824992 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.824935 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be1a58141492dfed2b72217076bc6e1c6fe7165d79543555ea496726d3ebf6f8"} err="failed to get container status \"be1a58141492dfed2b72217076bc6e1c6fe7165d79543555ea496726d3ebf6f8\": rpc error: code = NotFound desc = could not find container \"be1a58141492dfed2b72217076bc6e1c6fe7165d79543555ea496726d3ebf6f8\": container with ID starting with be1a58141492dfed2b72217076bc6e1c6fe7165d79543555ea496726d3ebf6f8 not found: ID does not exist" Oct 30 17:38:17.824992 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.824948 5052 scope.go:122] "RemoveContainer" containerID="412866de6c1f173ecd61049e96aab8dd01501c4baf9a0d3b2cc9a4d11e88ab49" Oct 30 17:38:17.832372 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.832354 5052 scope.go:122] "RemoveContainer" containerID="412866de6c1f173ecd61049e96aab8dd01501c4baf9a0d3b2cc9a4d11e88ab49" Oct 30 17:38:17.832647 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:38:17.832629 5052 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"412866de6c1f173ecd61049e96aab8dd01501c4baf9a0d3b2cc9a4d11e88ab49\": container with ID starting with 412866de6c1f173ecd61049e96aab8dd01501c4baf9a0d3b2cc9a4d11e88ab49 not found: ID does not exist" containerID="412866de6c1f173ecd61049e96aab8dd01501c4baf9a0d3b2cc9a4d11e88ab49" Oct 30 17:38:17.832726 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.832650 5052 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"412866de6c1f173ecd61049e96aab8dd01501c4baf9a0d3b2cc9a4d11e88ab49"} err="failed to get container status \"412866de6c1f173ecd61049e96aab8dd01501c4baf9a0d3b2cc9a4d11e88ab49\": rpc error: code = NotFound desc = could not find container \"412866de6c1f173ecd61049e96aab8dd01501c4baf9a0d3b2cc9a4d11e88ab49\": container with ID starting with 412866de6c1f173ecd61049e96aab8dd01501c4baf9a0d3b2cc9a4d11e88ab49 not found: ID does not exist" Oct 30 17:38:17.845642 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.845621 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-kpphn"] Oct 30 17:38:17.857028 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.857008 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-kpphn"] Oct 30 17:38:17.924485 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.924454 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-t8b5p"] Oct 30 17:38:17.930968 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.930945 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-t8b5p"] Oct 30 17:38:17.964016 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.963990 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-flvdc"] Oct 30 17:38:17.969380 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:17.969356 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-flvdc"] Oct 30 17:38:18.001048 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:18.001028 5052 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9kbt9"] Oct 30 17:38:18.005098 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:18.005079 5052 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9kbt9"] Oct 30 17:38:18.597693 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:18.597658 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="384a9c57-053d-477e-badb-355261588881" path="/var/lib/kubelet/pods/384a9c57-053d-477e-badb-355261588881/volumes" Oct 30 17:38:18.598125 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:18.598111 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42edda16-6400-45ef-bce3-a5ce362045ed" path="/var/lib/kubelet/pods/42edda16-6400-45ef-bce3-a5ce362045ed/volumes" Oct 30 17:38:18.598506 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:18.598495 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4c977d2-7480-4d5c-87d6-743c519e2efc" path="/var/lib/kubelet/pods/c4c977d2-7480-4d5c-87d6-743c519e2efc/volumes" Oct 30 17:38:18.599236 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:18.599221 5052 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcc7c9e1-99ea-4b97-9e95-2e23a00ac096" path="/var/lib/kubelet/pods/fcc7c9e1-99ea-4b97-9e95-2e23a00ac096/volumes" Oct 30 17:38:18.839456 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:38:18.839420 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 17:38:43.029748 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:43.029714 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-575c7b5c5b-p8bxg_44aa1bfa-6067-45b4-ada0-6ea89af3a587/manager/0.log" Oct 30 17:38:43.335393 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:43.335277 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-575c7b5c5b-p8bxg_44aa1bfa-6067-45b4-ada0-6ea89af3a587/manager/0.log" Oct 30 17:38:44.030061 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:44.030030 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-4ptgl_973d1ea4-3981-4350-b719-edb7911c23ab/payload-processing/0.log" Oct 30 17:38:44.203728 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:44.203699 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-4ptgl_973d1ea4-3981-4350-b719-edb7911c23ab/payload-processing/0.log" Oct 30 17:38:44.535858 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:44.535824 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-vstlf/must-gather-s8d4j"] Oct 30 17:38:44.536269 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:44.536256 5052 cpu_manager.go:414] "containerMap: removing container" podUID="c03b7d08-8589-49f4-a662-c7bf1d9c8caf" containerName="maas-api" Oct 30 17:38:44.536367 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:44.536271 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c03b7d08-8589-49f4-a662-c7bf1d9c8caf" containerName="maas-api" Oct 30 17:38:44.536367 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:44.536283 5052 cpu_manager.go:414] "containerMap: removing container" podUID="42edda16-6400-45ef-bce3-a5ce362045ed" containerName="istio-proxy" Oct 30 17:38:44.536367 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:44.536288 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42edda16-6400-45ef-bce3-a5ce362045ed" containerName="istio-proxy" Oct 30 17:38:44.536367 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:44.536301 5052 cpu_manager.go:414] "containerMap: removing container" podUID="384a9c57-053d-477e-badb-355261588881" containerName="istio-proxy" Oct 30 17:38:44.536367 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:44.536307 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="384a9c57-053d-477e-badb-355261588881" containerName="istio-proxy" Oct 30 17:38:44.536367 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:44.536327 5052 cpu_manager.go:414] "containerMap: removing container" podUID="c4c977d2-7480-4d5c-87d6-743c519e2efc" containerName="istio-proxy" Oct 30 17:38:44.536367 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:44.536334 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4c977d2-7480-4d5c-87d6-743c519e2efc" containerName="istio-proxy" Oct 30 17:38:44.536367 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:44.536355 5052 cpu_manager.go:414] "containerMap: removing container" podUID="fcc7c9e1-99ea-4b97-9e95-2e23a00ac096" containerName="istio-proxy" Oct 30 17:38:44.536367 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:44.536360 5052 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcc7c9e1-99ea-4b97-9e95-2e23a00ac096" containerName="istio-proxy" Oct 30 17:38:44.536708 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:44.536432 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcc7c9e1-99ea-4b97-9e95-2e23a00ac096" containerName="istio-proxy" Oct 30 17:38:44.536708 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:44.536440 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="c03b7d08-8589-49f4-a662-c7bf1d9c8caf" containerName="maas-api" Oct 30 17:38:44.536708 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:44.536448 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="42edda16-6400-45ef-bce3-a5ce362045ed" containerName="istio-proxy" Oct 30 17:38:44.536708 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:44.536455 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="384a9c57-053d-477e-badb-355261588881" containerName="istio-proxy" Oct 30 17:38:44.536708 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:44.536462 5052 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4c977d2-7480-4d5c-87d6-743c519e2efc" containerName="istio-proxy" Oct 30 17:38:44.539487 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:44.539466 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vstlf/must-gather-s8d4j" Oct 30 17:38:44.566980 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:44.566955 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vstlf\"/\"kube-root-ca.crt\"" Oct 30 17:38:44.567103 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:44.567003 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vstlf\"/\"openshift-service-ca.crt\"" Oct 30 17:38:44.567103 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:44.567048 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-vstlf\"/\"default-dockercfg-mtr8p\"" Oct 30 17:38:44.568820 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:44.568804 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vstlf/must-gather-s8d4j"] Oct 30 17:38:44.670388 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:44.663860 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t95n9\" (UniqueName: \"kubernetes.io/projected/1e853875-6ec8-4345-b611-f27579e66c91-kube-api-access-t95n9\") pod \"must-gather-s8d4j\" (UID: \"1e853875-6ec8-4345-b611-f27579e66c91\") " pod="openshift-must-gather-vstlf/must-gather-s8d4j" Oct 30 17:38:44.670388 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:44.663919 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/1e853875-6ec8-4345-b611-f27579e66c91-must-gather-output\") pod \"must-gather-s8d4j\" (UID: \"1e853875-6ec8-4345-b611-f27579e66c91\") " pod="openshift-must-gather-vstlf/must-gather-s8d4j" Oct 30 17:38:44.764899 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:44.764856 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t95n9\" (UniqueName: \"kubernetes.io/projected/1e853875-6ec8-4345-b611-f27579e66c91-kube-api-access-t95n9\") pod \"must-gather-s8d4j\" (UID: \"1e853875-6ec8-4345-b611-f27579e66c91\") " pod="openshift-must-gather-vstlf/must-gather-s8d4j" Oct 30 17:38:44.765113 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:44.764912 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/1e853875-6ec8-4345-b611-f27579e66c91-must-gather-output\") pod \"must-gather-s8d4j\" (UID: \"1e853875-6ec8-4345-b611-f27579e66c91\") " pod="openshift-must-gather-vstlf/must-gather-s8d4j" Oct 30 17:38:44.765269 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:44.765247 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/1e853875-6ec8-4345-b611-f27579e66c91-must-gather-output\") pod \"must-gather-s8d4j\" (UID: \"1e853875-6ec8-4345-b611-f27579e66c91\") " pod="openshift-must-gather-vstlf/must-gather-s8d4j" Oct 30 17:38:44.798143 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:44.798086 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vstlf\"/\"kube-root-ca.crt\"" Oct 30 17:38:44.818939 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:44.818919 5052 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vstlf\"/\"openshift-service-ca.crt\"" Oct 30 17:38:44.828646 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:44.828628 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t95n9\" (UniqueName: \"kubernetes.io/projected/1e853875-6ec8-4345-b611-f27579e66c91-kube-api-access-t95n9\") pod \"must-gather-s8d4j\" (UID: \"1e853875-6ec8-4345-b611-f27579e66c91\") " pod="openshift-must-gather-vstlf/must-gather-s8d4j" Oct 30 17:38:44.866052 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:44.866028 5052 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-vstlf\"/\"default-dockercfg-mtr8p\"" Oct 30 17:38:44.873539 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:44.873523 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vstlf/must-gather-s8d4j" Oct 30 17:38:45.175007 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:45.174975 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vstlf/must-gather-s8d4j"] Oct 30 17:38:45.178437 ip-10-0-131-136 kubenswrapper[5052]: W1030 17:38:45.178404 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1e853875_6ec8_4345_b611_f27579e66c91.slice/crio-1a388c2a4acb372f262059a8341a29f0afda24803e606787b3a60cbc920533af WatchSource:0}: Error finding container 1a388c2a4acb372f262059a8341a29f0afda24803e606787b3a60cbc920533af: Status 404 returned error can't find the container with id 1a388c2a4acb372f262059a8341a29f0afda24803e606787b3a60cbc920533af Oct 30 17:38:45.180386 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:45.180363 5052 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 17:38:45.950927 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:45.950889 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vstlf/must-gather-s8d4j" event={"ID":"1e853875-6ec8-4345-b611-f27579e66c91","Type":"ContainerStarted","Data":"1a388c2a4acb372f262059a8341a29f0afda24803e606787b3a60cbc920533af"} Oct 30 17:38:46.959045 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:46.959013 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vstlf/must-gather-s8d4j" event={"ID":"1e853875-6ec8-4345-b611-f27579e66c91","Type":"ContainerStarted","Data":"4d39887cc3922cd9459ddca25425583356f1bf80aba2a8c2d4de4ee91571a0b3"} Oct 30 17:38:46.959476 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:46.959055 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vstlf/must-gather-s8d4j" event={"ID":"1e853875-6ec8-4345-b611-f27579e66c91","Type":"ContainerStarted","Data":"e250d9d6df5dd3fa73d77533eba1804dea8267b2f34312bdc52a3b98131fc54b"} Oct 30 17:38:46.998061 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:46.998000 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-vstlf/must-gather-s8d4j" podStartSLOduration=2.33465672 podStartE2EDuration="2.997986482s" totalImagesPullingTime="663.329762ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:38:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:38:46.997117386 +0000 UTC m=+4682.524452605" watchObservedRunningTime="2026-10-30 17:38:46.997986482 +0000 UTC m=+4682.525321700" Oct 30 17:38:48.840367 ip-10-0-131-136 kubenswrapper[5052]: E1030 17:38:48.840326 5052 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wxvph" containerName="extractor" Oct 30 17:38:49.269259 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:49.269225 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-ncfzj_029f4ecb-eb6b-4693-9e9e-ca61f554dd33/global-pull-secret-syncer/0.log" Oct 30 17:38:49.439926 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:49.439893 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-4dfsk_d719cbcb-9874-4b85-8fe7-e1bf53511235/connection-checker/0.log" Oct 30 17:38:49.558001 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:49.557934 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-cpxbb_90d73ca9-5d88-4467-99fa-d4e1a67f3828/connection-checker/0.log" Oct 30 17:38:49.678692 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:49.678525 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-fhzbv_a6341cee-9ba8-41ce-bb75-29c9cb4e6d29/connection-checker/0.log" Oct 30 17:38:50.115600 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:50.112638 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-sg9tg_4c515c55-0717-4974-9f1a-68e0a25984cd/konnectivity-agent/0.log" Oct 30 17:38:50.489654 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:50.489611 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-136.ec2.internal_3de5cf7923bfc1c5b7b74a0700d7b3fa/haproxy/0.log" Oct 30 17:38:56.741603 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:56.739902 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5ddb48fb-kmgjr_25d85829-1725-4150-9c59-09c80da4dcab/authorino/0.log" Oct 30 17:38:57.253242 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:57.249777 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-57bb9bfff-ct9g7_3c887582-ac80-4a00-9e7d-87c569e4e425/manager/0.log" Oct 30 17:38:57.349404 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:57.349367 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-cdwcj_c1731802-685a-44c6-9700-efc3784bbbeb/limitador/0.log" Oct 30 17:38:57.522154 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:57.522072 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-856dd584c5-bh7c6_8c01f5e2-a20b-45c3-a6c6-4b62b02e88cd/manager/0.log" Oct 30 17:38:58.541605 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:58.540696 5052 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-vstlf/perf-node-gather-daemonset-cbn5t"] Oct 30 17:38:58.551254 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:58.551229 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vstlf/perf-node-gather-daemonset-cbn5t" Oct 30 17:38:58.570124 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:58.563294 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vstlf/perf-node-gather-daemonset-cbn5t"] Oct 30 17:38:58.712807 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:58.712768 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xdlw\" (UniqueName: \"kubernetes.io/projected/d9b8b63b-4ce4-4583-a949-7ceabd81e559-kube-api-access-5xdlw\") pod \"perf-node-gather-daemonset-cbn5t\" (UID: \"d9b8b63b-4ce4-4583-a949-7ceabd81e559\") " pod="openshift-must-gather-vstlf/perf-node-gather-daemonset-cbn5t" Oct 30 17:38:58.712988 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:58.712908 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d9b8b63b-4ce4-4583-a949-7ceabd81e559-lib-modules\") pod \"perf-node-gather-daemonset-cbn5t\" (UID: \"d9b8b63b-4ce4-4583-a949-7ceabd81e559\") " pod="openshift-must-gather-vstlf/perf-node-gather-daemonset-cbn5t" Oct 30 17:38:58.712988 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:58.712959 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d9b8b63b-4ce4-4583-a949-7ceabd81e559-sys\") pod \"perf-node-gather-daemonset-cbn5t\" (UID: \"d9b8b63b-4ce4-4583-a949-7ceabd81e559\") " pod="openshift-must-gather-vstlf/perf-node-gather-daemonset-cbn5t" Oct 30 17:38:58.713074 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:58.712991 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d9b8b63b-4ce4-4583-a949-7ceabd81e559-proc\") pod \"perf-node-gather-daemonset-cbn5t\" (UID: \"d9b8b63b-4ce4-4583-a949-7ceabd81e559\") " pod="openshift-must-gather-vstlf/perf-node-gather-daemonset-cbn5t" Oct 30 17:38:58.713074 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:58.713028 5052 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d9b8b63b-4ce4-4583-a949-7ceabd81e559-podres\") pod \"perf-node-gather-daemonset-cbn5t\" (UID: \"d9b8b63b-4ce4-4583-a949-7ceabd81e559\") " pod="openshift-must-gather-vstlf/perf-node-gather-daemonset-cbn5t" Oct 30 17:38:58.814372 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:58.814293 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d9b8b63b-4ce4-4583-a949-7ceabd81e559-lib-modules\") pod \"perf-node-gather-daemonset-cbn5t\" (UID: \"d9b8b63b-4ce4-4583-a949-7ceabd81e559\") " pod="openshift-must-gather-vstlf/perf-node-gather-daemonset-cbn5t" Oct 30 17:38:58.814372 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:58.814336 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d9b8b63b-4ce4-4583-a949-7ceabd81e559-sys\") pod \"perf-node-gather-daemonset-cbn5t\" (UID: \"d9b8b63b-4ce4-4583-a949-7ceabd81e559\") " pod="openshift-must-gather-vstlf/perf-node-gather-daemonset-cbn5t" Oct 30 17:38:58.814372 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:58.814361 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d9b8b63b-4ce4-4583-a949-7ceabd81e559-proc\") pod \"perf-node-gather-daemonset-cbn5t\" (UID: \"d9b8b63b-4ce4-4583-a949-7ceabd81e559\") " pod="openshift-must-gather-vstlf/perf-node-gather-daemonset-cbn5t" Oct 30 17:38:58.814748 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:58.814394 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d9b8b63b-4ce4-4583-a949-7ceabd81e559-podres\") pod \"perf-node-gather-daemonset-cbn5t\" (UID: \"d9b8b63b-4ce4-4583-a949-7ceabd81e559\") " pod="openshift-must-gather-vstlf/perf-node-gather-daemonset-cbn5t" Oct 30 17:38:58.814748 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:58.814499 5052 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xdlw\" (UniqueName: \"kubernetes.io/projected/d9b8b63b-4ce4-4583-a949-7ceabd81e559-kube-api-access-5xdlw\") pod \"perf-node-gather-daemonset-cbn5t\" (UID: \"d9b8b63b-4ce4-4583-a949-7ceabd81e559\") " pod="openshift-must-gather-vstlf/perf-node-gather-daemonset-cbn5t" Oct 30 17:38:58.814748 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:58.814537 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d9b8b63b-4ce4-4583-a949-7ceabd81e559-lib-modules\") pod \"perf-node-gather-daemonset-cbn5t\" (UID: \"d9b8b63b-4ce4-4583-a949-7ceabd81e559\") " pod="openshift-must-gather-vstlf/perf-node-gather-daemonset-cbn5t" Oct 30 17:38:58.814748 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:58.814575 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d9b8b63b-4ce4-4583-a949-7ceabd81e559-podres\") pod \"perf-node-gather-daemonset-cbn5t\" (UID: \"d9b8b63b-4ce4-4583-a949-7ceabd81e559\") " pod="openshift-must-gather-vstlf/perf-node-gather-daemonset-cbn5t" Oct 30 17:38:58.814748 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:58.814620 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d9b8b63b-4ce4-4583-a949-7ceabd81e559-sys\") pod \"perf-node-gather-daemonset-cbn5t\" (UID: \"d9b8b63b-4ce4-4583-a949-7ceabd81e559\") " pod="openshift-must-gather-vstlf/perf-node-gather-daemonset-cbn5t" Oct 30 17:38:58.814748 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:58.814679 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d9b8b63b-4ce4-4583-a949-7ceabd81e559-proc\") pod \"perf-node-gather-daemonset-cbn5t\" (UID: \"d9b8b63b-4ce4-4583-a949-7ceabd81e559\") " pod="openshift-must-gather-vstlf/perf-node-gather-daemonset-cbn5t" Oct 30 17:38:58.849115 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:58.849078 5052 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xdlw\" (UniqueName: \"kubernetes.io/projected/d9b8b63b-4ce4-4583-a949-7ceabd81e559-kube-api-access-5xdlw\") pod \"perf-node-gather-daemonset-cbn5t\" (UID: \"d9b8b63b-4ce4-4583-a949-7ceabd81e559\") " pod="openshift-must-gather-vstlf/perf-node-gather-daemonset-cbn5t" Oct 30 17:38:58.878978 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:58.878948 5052 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vstlf/perf-node-gather-daemonset-cbn5t" Oct 30 17:38:59.180839 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:59.180801 5052 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vstlf/perf-node-gather-daemonset-cbn5t"] Oct 30 17:38:59.193599 ip-10-0-131-136 kubenswrapper[5052]: W1030 17:38:59.184741 5052 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd9b8b63b_4ce4_4583_a949_7ceabd81e559.slice/crio-b1aa34500d18d38e10349e00c5a53c3e22ff71ab1f1549d8c63697986a0bdac3 WatchSource:0}: Error finding container b1aa34500d18d38e10349e00c5a53c3e22ff71ab1f1549d8c63697986a0bdac3: Status 404 returned error can't find the container with id b1aa34500d18d38e10349e00c5a53c3e22ff71ab1f1549d8c63697986a0bdac3 Oct 30 17:38:59.736900 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:38:59.734419 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-sfq4m_a0cd0943-8d62-41e4-b3e7-29eaabec7277/cluster-monitoring-operator/0.log" Oct 30 17:39:00.100653 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:39:00.100560 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vstlf/perf-node-gather-daemonset-cbn5t" event={"ID":"d9b8b63b-4ce4-4583-a949-7ceabd81e559","Type":"ContainerStarted","Data":"a510a57a17dc0fea0dbece4410e413f45ece11ef8a2475ceafc135be4bd8457e"} Oct 30 17:39:00.100653 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:39:00.100630 5052 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vstlf/perf-node-gather-daemonset-cbn5t" event={"ID":"d9b8b63b-4ce4-4583-a949-7ceabd81e559","Type":"ContainerStarted","Data":"b1aa34500d18d38e10349e00c5a53c3e22ff71ab1f1549d8c63697986a0bdac3"} Oct 30 17:39:00.100855 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:39:00.100695 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-vstlf/perf-node-gather-daemonset-cbn5t" Oct 30 17:39:00.140544 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:39:00.140486 5052 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-vstlf/perf-node-gather-daemonset-cbn5t" podStartSLOduration=2.140472337 podStartE2EDuration="2.140472337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:38:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:39:00.13926698 +0000 UTC m=+4695.666602200" watchObservedRunningTime="2026-10-30 17:39:00.140472337 +0000 UTC m=+4695.667807555" Oct 30 17:39:00.786010 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:39:00.785976 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nwkb6_ed3e3fe0-ec0c-4bce-914f-d1932f552782/node-exporter/0.log" Oct 30 17:39:00.906453 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:39:00.906419 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nwkb6_ed3e3fe0-ec0c-4bce-914f-d1932f552782/kube-rbac-proxy/0.log" Oct 30 17:39:01.021159 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:39:01.021130 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nwkb6_ed3e3fe0-ec0c-4bce-914f-d1932f552782/init-textfile/0.log" Oct 30 17:39:02.910279 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:39:02.910242 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-qmvpb_29041a5c-549c-47ec-ba84-a199ba0931b0/prometheus-operator-admission-webhook/0.log" Oct 30 17:39:05.370831 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:39:05.370803 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-992nb_1042378b-9f46-4e19-88e4-2628e532e8c3/networking-console-plugin/0.log" Oct 30 17:39:06.123159 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:39:06.123127 5052 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-vstlf/perf-node-gather-daemonset-cbn5t" Oct 30 17:39:06.406086 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:39:06.406062 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4wrf8_4f0f46f4-5cda-4d67-b6f3-47df06eb844c/console-operator/2.log" Oct 30 17:39:06.409798 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:39:06.409777 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4wrf8_4f0f46f4-5cda-4d67-b6f3-47df06eb844c/console-operator/3.log" Oct 30 17:39:07.527027 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:39:07.526993 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-gr95f_1cabe56f-84a6-42b2-b4c5-979d56a0d838/download-server/0.log" Oct 30 17:39:08.700050 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:39:08.700017 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-5fw2s_4d9a19d6-94d4-4490-8f93-89ca745b768f/volume-data-source-validator/0.log" Oct 30 17:39:10.991950 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:39:10.991911 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qg6t6_3e90fa4d-66a8-4689-8f41-ca090d6697a9/dns/0.log" Oct 30 17:39:11.112387 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:39:11.112336 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qg6t6_3e90fa4d-66a8-4689-8f41-ca090d6697a9/kube-rbac-proxy/0.log" Oct 30 17:39:11.843607 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:39:11.843500 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-lghnm_9ec1217e-8623-42a6-a052-adee153936a2/dns-node-resolver/0.log" Oct 30 17:39:12.990347 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:39:12.990304 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-db6c9d67-kdw6z_4db30ca2-0f2f-4573-9f1c-ed4306cab9c5/registry/0.log" Oct 30 17:39:13.083421 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:39:13.083392 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2l5ht_d10078a0-c21c-4f18-9c5a-4915fb7c8af3/node-ca/0.log" Oct 30 17:39:13.815069 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:39:13.815039 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-54d485b749-wgfjt_8cae735b-c0f8-4066-9ad3-2a688079eecf/kube-auth-proxy/0.log" Oct 30 17:39:13.997862 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:39:13.997833 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-4ptgl_973d1ea4-3981-4350-b719-edb7911c23ab/payload-processing/0.log" Oct 30 17:39:14.070281 ip-10-0-131-136 kubenswrapper[5052]: I1030 17:39:14.070200 5052 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-df85cd6ff-6pd8x_b99d8a1f-000b-48ff-b4b0-7a62bc79c032/router/0.log"